Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification
Authors: Aojun Zhou
Ke Wang
Zimu Lu
Weikang Shi
Sichun Luo
Zipeng Qin
Shaoqing Lu
Anya Jia
Linqi Song
Mingjie Zhan
Hongsheng Li
Venue: ICLR 2024 poster (2024)
Keywords: mathematical reasoning
large language models
zero-shot learning
code generation
prompting
Primary area: generative models
TL;DR: We propose a zero-shot prompt that elicits effective code-based self-verification for mathematical reasoning.

Abstract: Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has brought significant advancements in addressing math reasoning problems. In particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter, shows remarkable performance on challenging math datasets. In this paper, we explore the effect of code on enhancing LLMs' reasoning capability by introducing different constraints on the Code Usage Frequency of GPT-4 Code Interpreter. We found that its success can be largely attributed to its powerful skills in generating and executing code, evaluating the output of code execution, and rectifying its solution when receiving unreasonable outputs. Based on this insight, we propose a novel and effective prompting method, explicit $\underline{\text{c}}$ode-based $\underline{\text{s}}$elf-$\underline{\text{v}}$erification (CSV), to further boost the mathematical reasoning potential of GPT-4 Code Interpreter. This method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to use code to self-verify its answers. In instances where the verification state registers as "False", the model shall automatically amend its solution, analogous to our approach of rectifying errors during a mathematics examination. Furthermore, we recognize that the states of the verification result indicate the confidence of a solution, which can improve the effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we achieve an impressive zero-shot accuracy on MATH dataset.

Published as a conference paper at ICLR 2024
SOLVING CHALLENGING MATH WORD PROBLEMS USING GPT-4 CODE INTERPRETER WITH CODE- BASED S SILFVVRIFICATION
Aojun Zhou' '
Ke Wang" Zimu Lu"
Weikang Shi" Sichun Luo"
Zipeng Qin'
Shaoqing Lu
Anya Jia
Linqi Song'
Mingjie Zhan 1t Hongsheng Li
MMLab, The Chinese University of Hong Kong
Shanghai Artificial Intelligence Laboratory
3 City University of Hong Kong (aojunzhou, zmjdll) ( gggal.mll..... com
*CSUST
Tufts University
hslißee. cuhk. edu.hk
ABSTRACT
Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has brought significant advancements in solving math problems. In particular, OpenAl's latest version of GPT-4, known GPT-4 Code Interpreter, shows remarkable performance on challenging math datasets. In this paper, we explore the effect of code on enhancing LLMs' reasoning capability by introducing different constraints on the Code Usage Frequency GPT-4 Code Interpreter. We found that its success can be primarily attributed to its powerful skills generating and executing code, evaluating the execution result, and rectifying solution when receiving unreasonable outputs. Based this, propose a novel prompting method, explicit code-based self-verification (CSV). This method employs zero-shot prompt on the GPT-4 Code Interpreter to encourage it to use code to self-verify its answers. instances where the verification state is "False", the model will automatically amend its solution. Furthermore, we recognize that the states of the verification result indicate the confidence of solution, which can improve the effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we achieve an impressive zero-shot accuracy of various mathematical problem-solving benchmarks.
INTRODUCTION
Large language models (LLMs)) (Brown al. 2020 OpenAl 2023 Anil et al. 2023) have shown impressive success in various tasks. However, they still fall short in mathematical reasoning, often producing nonsensical inaccurate content and struggling with complex calculations. Previous works to tackle these challenges include the Chain-of-Thoughee
(CoT) (Wei al. 2022) framework, which enhances LLMs' logical reasoning abilities by generating intermediate reasoning steps. Additionally, PAL. (Gao et al. 2023) the Python interpreter to improve computational accuracy.
Recently, OpenAl has unveiled an improved version of GPT-4, namely the GPT-4 Code Interprete or GPT4-Code, which is good at providing natural language reasoning. alongside step-by-step Python code. Notably, it can generate and execute code incrementally, and subsequently present the execution results back to the LLM. This mechanism has shown promising results in solving mathematical problems. Our initial experiments show that GPT4-Code achieved an impressive zero-shot accuracy of 69.69% on the challenging MATH dataset (Hendrycks et al. 2021). marking significant improvement of 27.5% over GPT-4's performance (42.2%).
While GPT4-Code has demonstrated proficiency in solving math problems, there has been notable absence of systematic analysis focusing on understanding and further enhancing its mathematical problem-solving abilities. A critical distinction between GPT4-Code and its predecessor, GPT- lies in GPT4-Code's ability to automatically generate and execute code. Therefore, this paper
Equual contribution.
I Project leader.
"Corresoonding author.
2 https: https://chat..c openai com/ 2mode1 -G1 -odee ioterppeeeeeeeeeeeeeeeeeeeeeee

---

Published as a conference paper at ICLR 2024
presents pilot experiments investigating GPT4-Code's code generation and execution mechanism using specific code-constrained prompts. The analysis reveals that GPT4-Code's 's strong performance is not solely due to its code generation and execution abilities but also its capacity to adjust its problem-solving strategies based on feedback from code execution process debugging (akin to (Chen et al., 2023b)), examples illustrated in Appendix [E Due to this clever mechanism, there is an increased frequency of code usage. Hence, we introduce Code Usage Frequency differentiate these unique prompting strategies quantitatively analyze the impact of code-constrained prompts on GPT4-Code for mathematical problem-sol
term self-
The step-by-stie code generation and self-debugging mechanisms highlight the critical role of code in mathematical mooble--solving. Nevertheless, the self-debugging mechanism only verifies the correctness of code while lacking the verification of the reasoning steps and the final answer, which has been demonstrated to be of vital importance to solve math problems of LLMs (Cobbe et al., 2021 Lightman et al., 2023).
We therefore ask the question: can we fully exploit the code generation and self-debugging mechanisms in GPT4-code, that it automatically verify and correct its solutions, without extra assistance from other models or users?
To answer this question, we propose a simple yet effective prompting technique termed the explicit code-based self- verffieritnnntion (CSV), which guides GPT4-Code to generate additional code that verifies the answer and adjusts the reasoning steps if there's a flaw reasoning. Unlike previous methods that rely on external language models for verification (Lightman et al. 2023 Cobbe et al., 2021), our approach leverages GPTT4-Code's inherent strengths. This approach offers two key benefits: (I) When the verification indicates an answer is False, GPT4-Code can rectify its prior solution and provide an improved alternative. (2) Solutions verified as True tend to be more reliable, akin to human problem-solving. However, even if a solution self-verified as False, we do not directly abandon it. Instead, we propose a weighted majority voting strategy that incorporates the code-based solution verification results, opposed to relying exclusively the frequency of answers. We assign different weights to the solutions according to their verification states, reflecting the solutions' varying levels of reliability. In alignment with the Code Usage Frequency analysis from our pilot experiments, our explicit code-based self-verification prompt boosts GPT4-Code's accuracy in mathematical problem-solviee with increased code usage.
Empirical study demonstrates the effectiveness of our proposed pipeline on the MATH, GSM8K, and MMLU-Math datasets using GPT4-Code Our method achieves impressive accuracy of 84.3% on the MATH dataset, greatly outperforming the base GPT4-Code and previous SOTA methods.
This paper's 's main contributions can be summarized in three key aspects:
This study provides the first systematic analysis of code generation, execution, and self- debugging's 's role in mathematical gobblem-ollvigg.. Our findings reveal that GPT4-Code's impressive mathematical problem-solving proficiency is primarily attributed to its step-by- step code generation and dynamic solution refinement based on code execution outcomes.
We introduce the innovative explicit code-based self-verification (CSV) prompt, which leverages GPT4-Code's ' advanced code generation mechanism. This prompt guides the model to verify the answer and then reevaluate its solution with code. CSV not only extends the verification to the logic behind problem-soll solving but also improves the efficacy of the majority voting method by integrating the verification states.
We have demonstrated the superior performance of GPT4-Code and the proposed CSV pipeline through extensive quantitative experiments variety of challenging mathematical reasoning datasets.
2 RELATED WORK
Chain-of- Chain-of-Thought Reasoning, The Chain-of-Thouse (CoT) prompting approach proposed by (Wei et al.] (022) shows the multi-step reasoning capabilities of LLMSs. (Kojima et al., 2022) implements Zero-shot-Coll which can serve as a strong zero-shot baseline. Further research extends the reasoning capabilities of CoT by applying majority voting (Wang et al. 2023), choosing complex few-shot examples and outputting results (Fu al. (002)) breaking down the problem (Zhou al.
(Yao et al. 2023). Similar to -ero-shot-CoT. our method applics "step by like prompts to regularize GPT4-Code's use of code. Additionally, We enhance majority voting by leveraging the results of CSV voting weights.
2023), or even using
2

---

Published as a conference paper at ICLR 2024
Question
(c) Model output
Q: The greatest common divisor of positive integers and is 6. The least common multiple of m and 71 is 126. What is the least possible value of n?? A: 60
BasePrompt:: problem: (Q). One importann property reliting GCD and 1CM that
GCD( L.MM(m, vi
In this peoblem, we can get
(a) Model output
(b) Model output
Let's compute the right hand side of the equation first
Prompt You are not allowed use any code. The problem: (Q). For any two posiive imegers mn and the product of their GCD and LCM aumiarrs
Promat can use code only time.
problem: [Q]
god.kmm 6, 126; w> 756
For any two positive integers mn and m. their god and lemn safisy: god (m,, x)) lem( re)
product of the
The product m and mn and both mn and ns must be mulliples of
GC Dim, n) L.C. MC,, m]]
In this problem, we're given that gol((r,, (n) and Icm (rt, 126:
6. So we need find all the factors of 756 that are multiples of
3a 6 12G
Given that GC ((m, n) 6 and L.C (Mm, rx) 126 6 126 m
integers generate possible paiss (m, n). After have pairs (m,, n). we can find the pair that minimizes mn + n.
that
iniim"""*0.51 + L. mf ni. (66. 126). (12, 63), (28)
[G. timex #f B
in mang((
Solving gives add up to smallest sum.
pair 756.
The pains of factors
that multiples
import sympp sp
god.emm 6.126; product
Jom
(6. 126) (12, G3) (18, 42) We want find the puir with the least sum.
paits [dd, product if d) for d in sp.divproduct) if <<m product 1 d]
The puits factors of 756 are (1,756).
(2,577)).., (12,63). (14.54). (18,42). (21 10,,0, (27,28).
min puir min(puins koymsum)
smoss lppais, sumtpaicc) for pair factor_ppiis] minisums, key=lambda x)1I) .> (188 42), 60)
We that the piir (21.36) gives the smallest which
96
x> (127. 28).55) 55)
min pair, sumfmmin poir)
So. The least value on 57. Therefore, the solution 57
The puir (i,, ms) that min m + (27, 28), and minimum possuble value
The puir with smallest sum (18, 42), sum is GD.So, the least possible value is GD
X
Figure 1: Example from the 277th number theory problem in the MATH dataset: (a) Prompt I generates
lext-only reasoning steps. (b) Prompt 2 generates intermediate text reasoning steps and post-hoc code, akin
to PAI. Gao et al. 2023). (c) Base prompt without code usage constrain. Details of the three prompts are
presented in Sec. 33. (For more examples, see Tab. and I
in Appendix HD)
Solving Math Problems with Code. Large language models have been found to be less accurate in performing arithmetic calculations (Cobbe et al., [021] Lewkowycz al. 2022 Gao et al. 2023 Lu et al.] 2022). Consequently, previous works have attempted to solve math problems with the assistance of code. The GSM8K dataset (Cobbe et al. 202T) uses calculation annotations to extract all arithmetic calculations solved by an external calculator: the Python eval function. Program- Aided Language model (PAL) (Gao et al.] 2023) and Program of Thoughts (PoT) (Chen et al.) 2022) obtain the answer by generating and executing Python code. Although they can improve computational accuracy, many generated codes get wrong answers due the lack of verification. Our approach not only utilizes the ability of GPT4-Code to generate codes and refine codes that fail to run, but also uses CSV to enhance the accuracy of the answers.
Saff-veifiiattion Previous studies train additional verifier to verify the correctness of final answers (Cobbe et al.] (02T) or intermediate steps (Lightman et al.] 2023 Li et al.] (023). (Weng et al. 2023) showed the self-verification abilities of LLMs by generating and ranking multiple answers. Furthermore, Self-refine proposed by (Madaan al.) 2023) iteratively refines output through self-generated feedback. Self-debug (Chen et al. 2023b) prompts the LLM to debug its own prediction for code generation. Unlike these methods that require LLMs to give verification feedback in natural language, our method applies generated codes to verify the answers and votes on different answers based on the verification results, thus improving the accuracy of the verification.
3 METHOD
We first conduct a pilot experiment with GPT4-Code on the challenging MATH dataset (Hendryeks) et al.] 2021]. Remarkably, it achieves an accuracy of 69.69%. significantly surpassing the previous -tate-of-the-art performance of 53.9% (Zheng et al. 2023). Encouraged by the compelling performance of GPT4-Code, we strive to systematically explore and analyze its underlying code mechanisms. In Sec. 3.1l we illustrate, via our code-constr prompts design, that GPT4-Code's 's robust performance in solving math problems derives not only from its ability to generate accurate step-by-step code, but also from its self-debugging mechanism. In Sec. 3.2 we aim to leverage GPT4-Code's 's deff-debugging strengths to further improve its mathematical problem-solving ability.
3

---

Published as a conference paper at ICLR 2024
Verifiation Prompt
Base Prompt
(Please veriy your answer using code interpreter by yourselt)
Prompt (You allwed code.)
Prompt
(You can use code only 1 lime)
Code Usage Frequency
(a)
(b)
73.54
100
3.0
67.58
89.89
60
10.50
2.0
Accuracy
Accuracy
60
15
40
40
10
20
0.5
0.1
Level 1
Level 2
Level
Level
Level
Prompts
Levels
Figure 2: Performance on MATH dataset different levels by applying different prompts to adjust
frequency code usage. (a) Comparison overall accuracy between the four prompts. (b) Code Usage Frequency is in proportion accuracy all five levels, and this phenomenon is especially apparent when the problems are relatively complicated (i.e., with higher levels). The red points denoting Prompt show that the model still occasionally uses code, especially when the problem is very difficult. However, even then, the Code Usage Frequency is negligible.
3.1 PILOT EXPERIMENTS ON ANALYZING CODE USAGE OF GPT4-CODE
To explore the impact of code on GPT4-Code's mathematical skills, we adopt a straightforward approach by constraining GPT4-Code's 's uasge of code through thoughtfully constructed prompts. Specifically, we introduce two code-constrained prompts and base prompt for comparison:
Prompt 1. No code usage is allowed: With this prompt, GPT4-Code is prohibited from
using code. This prompts GPT4-Code to rely solely Natural Language (NL) reasoning chain, resembling solutions the CoT framework (eii et al. 2022). The resulting sequence of reasoning steps is depicted as CNL. with an example given in Fig. [
Prompt Code can be used only once: In this prompt setting, GPT4-Code is permitted to employ code within a single code block to generate the solution, mirroring the PAL approach introduced by (Gao et al. 2023). We denote this sequence as CSL. representing series of Symbolic Language (SL), such as Python. An example shown Fig. (b).
Base Prompt. GPT4-Code is prompted to tackle the problem without any restrictions code usage. This prompt leads to GPT4-Code's usual performance, which be denoted as C ((CINL. ((cINL,Cls) clss), (c2NL. c2sr)..... representing list of reasoning steps, each consisted of both natural language and code, with an example shown in Fig. (c).
Apart from the specific example in Fig. I we introduce Code Usage Frequency to record the number of code executions for different prompts. The results of the experiments using these prompts are shown in Fig. (b). This figure illustrates positive correlation between the better performance of GPT4-Code and the higher Code Usage Frequency. More specifically,
Prompt v.s. Prompt 2. Prompt results in almost negligible code usage, while Prompr 2 results in approximately time's 's code usage. Prompt yields an accuracy gain of 6.78 percent over Prompt 1. This suggests that the Python code chains CSL can improve computational capability more than the natural language chains CNT. This observation is consistent with the findings in previous Python- based prompting methods (Gao al., [2033 Chen et al.) (002).. However, employing code only once comes with an inherent trawback - the model lacks the ability to self-debug when the code output triggers eπrоr produces implausible outcome.
Prompt v.s. Base Prompt.
The Base Prompt consistently produces solutions that entail multiple instances of code usage, resulting in a large Code Usage Frequency. Additionally, the Base Prompt's accuracy of 69.69%% represents a 2.1 1% improvement over Prompt 2's 67.58% These improvements two unique advantages: (1)
in Code Usage Frequency and accuracy might be attributable Generating code in brief and frequent segments, divided among natural language reasoning steps, tends to result in higher accuracy. (2) The model possesses the capability to evaluate the results of
4

---

Published as a conference paper at ICLR 2024
Question
(b) Example: w/ self verifications
Q: What is the smallest positive integer solution ( 100, 1?
We linsly change the incquelit
& 100000000.
Then let's evalzute the fourth 100,0000000
A: 101
inport powCL00000,,
smallest integer moth moth crik)(o) a> (1000, 100)
(a) Example: w/o self verification
Thus, the smallest positive integer a 100. Let's verify this solution.
subprobless, stop rearragg imqqulity isclate a.
Bresking
verifcatton leli (satallent intieger we
We can multiplying both sides the insquality 4 and taking
False
This gives / 1000000000
The smallest integer an should be the next integer grester than 100, not 100 iiself.
Let's solve
Let's calculate that next integer.
smallest intggrr 100
root pow: 184)
vesification- le8i (smallesst integer (101,
Since 100 already positive integer, the smallest posdtive integer solurice to the inqquliy 100 pon, 000 is 100.
Upun substituting 101 back. holds which verilies
is
100
solution. final
102
Figure 3: Question from the 712th intermediate algebra problem verification, the model generates wrong answer. (b) With velf-verification, model corrects the error and generates the correct answer. The CSV prompt: Solve the problem using code interpreter step by step, even in every sub-step. And following your answer, please verify using code interpreter by yourself.
MATH dataset. (a) Without self-
code execution and make corrections to solution steps if the outcomes contain bugs or are deemed illogical, as illustrated in Tab, and Tab. [ (Appendix EE.
From these observations, it is plausible to enhance and build upon the favorable attributes of GPT4- Code further improve its precision in tackling math problems.
3.2 EXPLICIT CODE-BASED SELF-VERIFICATION PROMPTING
Inspired by the observations on Code Usage Frequency analysis, we seek to harness the capabilities of GPT4-Code. These capabilities include the model's aptitude for generating accurate code, evaluating the outcomes of code execution, and automatically adjusting reasoning steps of solutions when needed. Our objective is to utilize these strengths to augment solution verification.
To achieve this objective, we propose the technique termed as explicit code-based self-verification (CSV). This method prompts GPT4-Code to explicitly validate its answer through code generation. By implementing this prompt, introduce extra verification stage to the solution C, referred as V. The verification result can be classified as True, False, or Uncertain. An Uncertain classification indicates that GPT4-Code encountered difficulties in identifying an effective method for answer verification, thereby abstaining from delivering definitive verification result. Leveraging GPT4-Code's inherent autonomous capabilities, we can formulate the proposed prompting as:
True
final answer
C-V=
False
Cnew
True final answer
Uncertain
- final answer
An example is presented in Fig. (b). Incorporated with CSV, the model becomes capable of using code to verify answers, then reviewing and adjusting how it arrived at the solution if the verification result is False. aiming obtaining the correct answer. The different types of verification code be seen in Tab Tab Tab[T8 and T9b (Appendix D.. Upon refining and correcting the initial solution, we anticipate notable increase in accuracy. It worth noting that both the verification and rectification stages code-based. This inevitably results in increased Code Usage Frequency, akin to the aforementioned analysis, which will further demonstrated in subsequent experiments.
We perform experiments with CSV, and these results can be found in Fig. 2 The experiment here is conducted with GPT4-Code MATH (Hendrycks et al. 2021 In Fig. (b), the accuracy achieved
5

---

Published as a conference paper at ICLR 2024
Question: Given that the repetend in the decimal representation of contains 18 digits, find the 38th cight in the decimal represeniation.
Reasoning
Code
Reasoning
A. Verification
Answer
Verifid:
Verited:: Uncertain
Verfildi False
thuen sequence moceste every dights,
frrt few go will see potterr emerge Let's cakulate this using Pytton.
Heput
the the sama the 21"
resresentation of
Code
macctior.....
The 20th digit be the same 2ar1 requentee repeat ohe desimal
The result New let's answer.
Code apeastion....
example
The 5thh digit in the ducimal representation pp
Code exccution
The patter repeating
dig tn
dgg.
Code snooution....
The vorifiaaiion
The un ritaaion result Tharolans,
reprrenttation
contidently
Dur fimal iesult
39th dagit inceed be
S
Majority Voting
Verification guided weighted majority voting
Wi =
Wy 0.2
True
Uncartain Fale
Output
Gueput
Candidate arowers:
0
xureff) 122 222
() CoT
4:c6
Verliaaten-guiled
3.+
Mojorttyvoting
5 X
wrighted mojortty
2
voting:
[a]
(b)
Figure 4: (a) Illustration of the Naive majority voting (Wang et al. 2023) and our Verification guided weighted majority voting. (b) The full pipeline of the proposed Verification guided Weighted Majority Voting (VW- voting) framework. We detect the self-verification state each solution and classify them into three states: True, Uncertain, and False. According to the state of the verification, we assign each solution a different weight and use the classified result to vote the score of each possible answer. (For more examples, see Tab. 12 and Tab.
13 in Appendix G
with our proposed CSV prompt consistently surpasses that of the Base Prompt across all designated difficulty level.) Meanwhile, the Code Usage Frequency receives a clear increase.
Before the advent of GPT4-Code, prior frameworks (Lightman et al., 2023 Cobbe et al. 2021 relied external Large Language Model (LLM) and well-construeted few-shot prompts for natural language verification. In contrast, GPT4-Code's 's robust capabilities enable our approach to depend solely on straightforward prompt, thereby operating in zero-shot manner. This enables GPT4-Code to autonomously verify and independently rectify its solutions using the advanced code execution mechanism, thereby eliminating the need for customized few-shot examples.
Given that CSV can effectively verify wooblem-solving answers, we can naturally integrate the verification states into majority voting, akin to the methodology embraced in self-consistency CoT (Wang et al., 2023). Answers deemed True through verification are generally more trustworthy, reflecting the problem-solving approach seen in human cognition (Newell & Simon 1972, Wang & Chiew 2010). This improved reliability can be leveraged in the widely-used majority voting process. To exploit this insight, we introduce verification-guue guided weighted majority voling, which assigns different weights to the states of the verification process.
In practice, sometimes occurs that once an answer is confirmed as False, no additional verification is conducted, yielding a False verification state. We allocate corresponding weights these states of True, Uncertain, False: Wr, Wu, and Wp, respectively.
Similar to the Seff-consistency with CoT (CoT-SC) (Wang et al., 2023) in Fig. 4 (a)(i), our framework can sample l paths. For simplicity, we extract pairs of final answers and their corresponding verification results from & solutions, represented (o', a²), 1,2,..., k. where of and a² denote the i-th final answer and final verification result, respectively.
So the voting score for each candidate answer can be expressed as:
Score(a) Σ w. (+ a' a and of v)); E (True, Uncertain, False),
Humann perceveee easier problems categorized under Level-1 difficulty as per (endryykks et al.
2021

---

Published as a conference paper at ICLR 2024
Table 1: Accuracy (%) on MATH dataset. VW-voting is an abbreviation for Verification- Weighted Majority Voting. Voting is abbreviation for Naive Majority Voting. (Overall: The results across various MATH subtopics)
Code-bused Vertfication
Intermediate Algehra
Precalculus
Geumetry
Number Theory
Couning & PreAlgebra
Algebra
Overall MATH
Probubdliny
GPT-4 OpenAL 20261
12.20
GP hengg
16.8
33.4
29.7
49.1
GPT-4 (Complex Ca
23.4
26.7
36.5
49.6
53.1
71.6
70.5
50.36
GPT-4 (P11)) (hhng et M223
26.3
29.8
41.9
55.7
56.3
73.8
74.3
53.90
GPT4-Cade (haselinc)
51.5
53.4
77.2
70.6
86.3
83.6
69.00
53.9
85.6
77.3
86.9
Improsement
+2.4
.0.6
+8.4
46.7
+0.2
43.3
+3.85
GPT4-Code+ Votlnng (6-16, baseline)
64.1
61.7
89.1
84.6
92.9
79.85
GPT4-Cade Voting (k=16)
66.5
91.1
88.8
95.3
83.54
-2.8
+4.0
GPT4-Code+ CSV+ VW-Yoting (k=16)
94.1
Improvement
+3.7
+5.0
+2.7
Here, a represents a candidate answer, v denotes the state of verification, and w. is an element from the set (wr, Wu, ). Each w, signifies the degree of confidence associated with its corresponding verification state. Finally, we select the answer with the highest score from all candidate answers.
It should be noted that when w, 1 for all we E fwr, Wu. wp ) Eq. becomes equivalent to the naive majority voting employed in Seff-Consistency with CoT (CoT-SC) (Wang et al. 2023). Typically, we set Wr > Wy > Wp. which means that an answer verified true has greater confidence than the one with an uncertain state of verification, while an answer verified false has the lowest degree of confidence. An example of the calculation process within verification-guided weighted majority voting is illustrated in Fig.
EXPERIMENTS
Datasets and Baseline. We evaluate GPT4-Code using CSV on three datasets: MATH (Hendrycks) et al. (021), GSM8K (Cobbe et al. 2021). and MMLU-Math (Hendrycks et al. 2020). We primarily compare our code-based verfiveeriicttion (CSV) approach to standard zero-shot prompting using GPT4-Code validate the effectiveness of the self-verification ability. To more comprehensively evaluate the zero-shot capabilities of both GPT4-Code and GPT4-Code with CSV in mathematical reasoning tasks, we also compare our method with the state-of-the-art few-shot in in-context-learning method using GPT-4 from PHP (Zheng et al., 2023) and Model selection (Zhao al. 2023)
Prompt. The proposed prompt is presented in the caption of Fig.
4.1 PERFORMANCE ON MATH
The MATH dataset (Hendrycks et al. 2021 is recognized as the most challenging math word problem dataset, also highlighted by Chen al. (Chen et al. 2023a). Most of our experiments and the corresponding analyses are performed on the MATH benchmark. Tab. [ compares the performance of the GPT4-Code against other models. GPT4-Code reaches 69.69% on MATH (Hendrycks et al. 2020). largely surpassing the previous state of the art result (53.90%), which shows that GPT4- exhibits strong abilities in solving math problems and used as our baseline. On top of GPT4-Code, our method further improves its accuracy, raising the result to 73.54% after adding explicit code-based self velf-verification. GPT4-Code with naive majority voting reaches an accuracy of 79.88%, which we set as the baseline for methods that used voting. When using code-based self-verification with majority voting, the accuracy is 83.34%, while adding both explicit code-based self-verification and verification- -uuiddd weighted majority voting reaches an accuracy of 84.32%. Note that this astonishingly high result is based on the strong abilities of the base model GPT4-Code, and our method amplifies its good qualities of GPT4-Code with the ability to verify solutions.
Note that although adding CSV can improve the performance of every individual subject, the extent of improvement varies, from 8.4% to only 0.2%. In particular, the Geometry problem only has an increased accuracy of 0.6%, even though the original accuracy only 53.4%, which is low among the subjects. This discrepancy may be attributed to the fact that solving geometry problems often requires multi-modality (Chen al. 2023a). concept beyond the scope of this paper.
7

---

Published as a conference paper at ICLR 2024
Table 2: Performance on GSM8K dataset.
Table 3: Performances on MMI.U-Math dataset.
Method
Sampled paths
(ccumeyy(5)
Methol
Samplel pulhs
Aucuracy((
Fou-shox
GPE55 (S-hot) (peenAl) 2025
57.1
Goper pacetal
shot
GPT4 (5-boo Cola ا m
92.0
GPT4 (PHP) Theng ou BU663
40
96-5
Lian-20
5-shat
GPT4 (Madef vereatoor Pacurel ml 3025
15
96.8
Galactics
41.5
zero-shot
GPT4-Code
GPT4-Code + CSV VW-Nuting
GPT4- Code + CSV
GPT4 Code Noting
92.9
$1.9
94.5
97.0
GPT4-Call GPTS-Code Nirting
GPT4 Code + CSV VW-Yoiing
GITTs-Cade + CSV
87.5
94.5
89.2
92:1
zcro-shot
neno-dhoe
zeno-shor
acro-dus
(a) Level
(b) Subject
90
Level
Level 2
90
Level 3
Level
Accuracy
80-
Accuracy
Level
Overall
70
number theory
60-
precakculus
algebra
prealgebra
50-
50-
intermediate
algabra
counting and probability
40
geometry
Code Usage Frequency
Code Usage Frequency
Figure 5: The four points on each curve correspond results using Prompt 1. Prompt Base Prompt and Proposed Prompt, respectively. (a) The accuracy of different levels at various code usage frequencies. (b) The accuracy of different subjects at various code usage frequencics.
4.2 PERFORMANCE ON GSM8K AND MMLU-MATH
In addition to the challenging MATH dataset, we have also performed our method on other reasoning datasets such as GSM8K (Cobbe et al.] [021). MMLU-Math (Hendrycks et al.) 2020). The corresponding results can be viewed in Tab, 2 and Tab. 3 When integrated on top of GPT-4-code, our method outperforms other methods, achieving state-of-the-ar results across all datasets. Other subjects in MMLU benchmarks are provided in Appendix D A comparative analysis of our results with those of previous state-of-the-art techniques and open-source models also provided.
Tab. 2 illustrates that verification-guided majority voting is an effective framework to reduce the number of sampled paths, compared to GPT-4 with model selection (Zhao et al., (023) and PHP (Zheng et al. 2023). Tab. presents a comparison of our model's performance with existing models (Hoffmann et al., 2022 Taylor et al., 2022; Toovroon el al., 2023) on the MMLU-Math dataset. The open-source models remain significantly outpaced by their closed-source counterparts.
4.3 CODE USAGE FREQUENCY OF PROPOSED PROMPTS
Analogous to the approach taken in Sec. 3.1 we gather data to clucidate the correlation between accuracy and Code Usage Frequency across various dimensions prompts (proposed CSV prompt and prompts used in pilot experiments), subjects, and difficulty levels. As shown in Fig. [ the model's behavior is in line with our expectations when adding the code-based verification prompts. Each line in Fig. has an obvious trend of going upwards, suggesting a possible positive correlation between Code Usage Frequency and accuracy. The performance gain when using more code is more obvious in the higher difficulty levels. while in lower levels, the performance gain is not very prominent, shown in Fig. 5 (a). The Code Usage Frequency increases with the increase of difficutty levels. This shows that the harder math problems require more frequent code usage, which implies that invoking code multiple times might be an important reason why GPT4-Code have such an advantage in solving difficult math problems. There is similar trend in Fig. (b).
4.4 ABLATION STUDY AND DISCUSSION
Comparisons between Natural Language and Code-based Self-Verification. To underscore the significance of code in the self-verification stage, employed distinct natural language self- verification, where GPT4-Code directed to verify the solution through natural language instead of relying on code-based verification, as presented in Tab, 4 The accuracy achieved with this method was slightly lower than that of the Base Prompt. Moreover, observed decline in accuracy
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison Self-verification with/without explicit code-hased prompt (Overall:The results across various MATH subtopics (Hendrycks al. 2021
Verification Method
Intermediate Algebra
Preculculas
Geometry
Number Theory
Counting & Probubility
PreAlgebra
Algebra
Overall
GPTc-Codc
Interpreter
Natural I Language
52.6
48.7
50.8
79.9
72.5
83.1
82.6
69.29
+2.5
-2.8
-2.6
+2.7
+1.9
-3.2
-1.0
-0.40
Code-based
73.54
+6.5
+2.4
+0.6
+8.4
+6.7
+0.2
+3.3
+3.85
(6)
(b)
Prectsoo
84
Recall
Assuracy
82
80
Accuracy
Value
78
76-
1/000
74-
72
Precision
Accusayy
Different Metrics
Sampled Reasoning Paths
Figure 6: (a). The average precision, recall, and accuracy of five sampled paths on the MATH dataset. (b). The Acc MATH in response to the number of sampled reasoning paths when the weight is set different values.
for four of the seven subtopics, indicating that relying solely on natural language self- self-verification, which appears to have a negative impact on the accuracy, is less reliable than using code-based self-verification. Examples of natural language self-verification can be seen in Tab, [4 and Tab. [5 (Appendix IF). In contrast, code-based verification enhances accuracy across all seven subtopics when compared to the Base Prompt.
Analysis of Verification-guided Weighted Majority Voting. We initially compiled the confusion matrix (TP/TN/FP/EN). capturing solutions with self- verffcciiiittion that matches the True and False states mentioned in Eq. Eq.omrom five distinct sampled paths. The details are presented in Appendix [A] From this data, we computed Precision, Recall, and Accuracy (Solutions in the True state are seen positive). The results are presented in Fig. 6 (a). We note that Precision exceeds Accuracy by 22.34% (increasing from 73.54% to 95.88%), whereas Recall surpasses Accuracy by 5.57% (rising from 73.54% to 79. %). In particular, the average Precision registered at 95.88 35.88% This implies that the Accuracy has the potential to become much higher if more solutions reach the verified True state before giving the final answer.
Slyper-parameters ablation in Verification-guided Weighted Majority Voting. We also performed ablation studies on the hhper-yaramanne parameter w. E [Wr, Wo, We} in Eq [ As shown Fig (b). When the hyper-parameter setting satisfied wtr > Wu 2 Wp. the performance of the verification- guided weighted majority voting consistently surpassed that of the naive majority voting methods across all sampled paths. contrast, when we the hyper-parameter (w, 0.5, Wh 0.5, wpp 1). the performance under this configuration was worse than the naive majority voting. Therefore, proposed method, verification-guided weighted majority voting, is easy tune and robust.
5 CONCLUSION
In this paper, we begin with pilot experiments on GPT4-Code to explore how its use of code impacts its performance in mathematical reasoning. By analyzing Code Usage Frequency and accuracy, we determine that GPT4-Code's 's skill in solving math problems can be largely attributed to its ability to generate and execute code, as well as its effectiveness in adjusting and rectifying solutions when confronted with implausible execution outputs. Expanding this understanding, ideas of explicit code-based self-verification and verification-guide weighted majority voting, with the goal of enhancing GPT4-Code's 's mathematical capabilities. We hope this work could shed light math problem-solvin in open-source L.LMs, especially when advanced code usage is involved.
introduce the

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This project funded in part by National Key R&D Program of China Project 20227I0111110, and in part by General Research Fund of Hong Kong RGC Project 14204021.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10400, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem- driven question answering dataset, 2023a.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023b.
Karl Cobbe. Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 11Xiv:2110.1416. 2021.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-basee prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aide -ided language models. In International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. ArXiv, 0at/2009.03300
2020.
URL https://apl. semanticscholar wwwpooorrr.r//oouusSDD
221516475
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 1:X3v:2103.03874, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pp. 22199-222222 2022.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances Neural Information Processing Systems, 25:3843-38577 2022.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5315- 5333, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.2003: 2023.
10

---

Published as a conference paper at ICLR 2024
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint MxXiv:2209.16611 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651, 2023.
A. Newell and H.A. Simon. Human Problem Solving. ACS symposium series. Prentice-Halll 1972. ISBN 9780134454030. URL. https://books: goog le. com. hk nkkbbooke ?id- 3A3AAAAMMAAJ
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katic Millican. Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptist Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume. Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero. Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. ArXiv, abs/2112. 14466, 2021. URL https://apls www.seteeaatcnnccoeeoeeeoe www.:ar...oo.oooo....oo.... 1D:2333337
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science. arXiv preprint arXiv:2211 .00085, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fin-tumed chat models. arXiv preprint arXiv:2307.09288 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://opeerrewe net. htt////ooumiiiooeeo Pit-rummmII--PPP11IMWM
Yingxu Wang and Vincent Chiew. On the cognitive process of human problem solving. Cognitive Systems Research, 11(1):81-92, 2010. ISSN 1389-0417.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL wwt.hhtt///// httpt://///:ene /otpps:///penrviee. net/forum? v
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Large language models are better reasoners with self- naff-verificttoon, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan, Tree of Thoughts: Deliberate problem solving with large language models, 2023.
Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie. Automatic model selection with large language models for reasoning. arXiv preprint arXiv:2305. 14333, 2023.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hin prompting improves reasoning in large language models. arXiv preprint a a9X1v72304.09797, 2023.
11

---

Published as a conference paper at ICLR 2024
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, and Ed Chi. seast-to-most prompting enables complex reasoning in large language models, 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXPLANATION OF CONFUSION MATRIX
A confusion matrix is specific table layout that allows visualization of the performance of an algorithm. It's particularly useful for classification problems, and utilize it analyze the performance of our verification process.
The matrix itself is two-dimensional grid, 2x2, for the binary classification of verification results. Each row the matrix represents the instances in a predicted class, which is determined by the verification results given by the language model, while each column represents the instances in an actual class, which determined by the actual correctness of the answer given by the model. Tab. shows how the matrix looks for our verification process:
Table 5: Confusion Matrix of Verification
Answer Correct
Answer Wrong
Verification True
TP
FP
Verification False
FN
TN
Here's what the four terms mean:
True Positive (TP): The cases in which the model's verification result is "True', and the answer is actually correct.
True Negative (TN): The cases which the model's verification result is 'False', and the answer is actually wrong.
False Positive (FP): The cases in which the model's verification result is 'True', but the answer is actually wrong.
False Negative (FN): The cases in which the model's verification result is FFalse, but the answer is actually correct.
This matrix is very helpful for measuring more than just straightforward accuracy, based on which Precision and Recall are two important metrics. They are defined in Eq. and their meanings are as follows:
Precision is the fraction of relevant instances among the retrieved instances. It measure of the accuracy of the classifier when predicts the positive class.
Recall is the fraction of the total amount of relevant instances that were actually retrieved.
It is measure of the ability of a classifier to find all the positive instances.
TP
TP
Precision
Recall
(2)
TP + FP
TP +N
In other words, precision answers the question "What proportion of verified TRUE answers was actually correct?" while recall answers "What proportion of actual correct answers was verified TRUE?" Given its meaning, crification-guider voting is bound to be effective when the precision of verification is high.
13

---

Published as a conference paper at ICLR 2024
B PYTHON PACKAGE USAGE ANALYSIS
Tab. outlines the usage of various Python packages in our experiments. Among them, we found that the sympy package is utilized most frequently, highlighting its central role in the computational tasks performed.
Table 6: Python package usage frequency on MATH dataset.
All
Correct
Correct per code
Wrong
Wrong per code
c/w per code
sympy math
0.4168
0.3907
0.3323
0.4724
0.3194
104%
0.1590
0.1638
0.1393
0.1493
0.1009
138%
numpy
0.0284
0.0241
0.0205
0.0383
0.0259
79%
fractions
0.0094
0.0110
0.0094
0.0058
0.004
238 %
itertools
0.0034
0.0029
0.0025
0.0045
0.0031
80%
emath
0.0034
0.0026
0.0022
0.0052
0.0035
63%
scipy
0.0016
0.0009
0,0007
0.0032
0.0022
34%
matplotlib
0.0010
0.0003
0.0003
0.0026
0.0018
14%
functools
0.0004
0.0003
0.0003
0,0007
0.0004
57%
collections
0.0004
0.0006
0.0005
0.0000
0.0000
NaN
statistics
0.0002
0.0003
0.0003
0.0000
0.0000
NaN
Table 7: Performance of CSV GSM8K and MATH based on CodeLlama-7B and CodeLlama-34BB
GSM8K Accuracy (%)
MATH Accuracy (%)
CodeLlama-77B
17.44
6.56
CodeLlama-74 -B + CSV
20.85
10.18
CodeLlama-333 -3BB
28.96
9.12
CodeLlama-3443 CSV
37.60
13.36
C CSV ON OPEN SOURCE MODELS
have tested our method on CodeLlama-771 and CodeLLama-34B by applying the same prompting method of CSV to these weaker models in zero-shot manner. The results are shown in Tab. 7 As be seen, there is noticeable improvement the accuracy of both GSM8K and MATH, though the accuracy is much lower compared to GPT4-Code.
14

---

Published as a conference paper at ICLR 2024
D DETAILED EXPERIMENT RESULT ON MMLU DATASET
Fig. 7 iillustrates that GPT4-Code performs relatively poorly in certain domains, such as engineering and the humanities, with a particularly marked deficiency in virology, where it achieves score of less than 60%. These observations delineate specific areas that call for further investigation and refinement, thus outlining the direction for future improvements in the model.
Accuracy by Subsubcaleyory
(b) ALluraly by Calegury
mécallanecus
90-
medical_eenetcc
High school governmenn end and politics
88.
school pamputer
90
markating
legnaschol prychology
30.01
Nihhschool jah_school_issooy
us foreigrr pollcy
Uverall: $0.32
agg_achool_ in erneconamec
hight aottsccool_ure history
callage biclogy
elementery_ machemahes
hight high_schoo__o htapsschoooogggooo Leeography
02"
professional medicine
prchiztory
80.02
sociology
80
sexualiyxuaity
kalza fallacies
SUmarittes STEM
estronomy
(c) Accuracy by Subcategory
religions
high scheol history
ju juriuuueence
Weleg
hight snt_sshoel__mmcc mecroeconomics
conceptual physics
other
nutrition
lusiness
high_school_ mathematiccs
geography
cinical knowledge ldgee
global fucts
business
internationaallnn Jaw
highh school statistics
cuture
collgge puter, science
meral diisptes
hitery
management
professional paychology
paychelogy
philosophy
public
economics
protessioral accounttnn
high wwccsccchhcceeee puropean history
humen .ging
physks
righ, schocl chemistry
high school__hyaae
eol eelingnmee medicine
compute: secunty
moral pcenarios
anatomy
computer science
college me.ttematiss
destrical ongincerng
machine leaming
philsaaphy
securnty studes
'omo logic
chemistry
abstract alaebra
colege_ctemsstty
enginwerng
economatric
professional bw
law
GPT4 Code Interpreter
viroloay
Overall: 85.32
Overall: 85 32
80
90
70
90
95
Accuracy
Accuracy
Figure 7: GPT4-Cod performance on MMLU dataset on: (a) subsubcategory, (b) category, (c) subcategory
15

---

Published as a conference paper at ICLR 2024
(a) Accuracy by Subsubcategory
(b) Accuracy by Category
high_ gavamment, politics
88.11
88
dementary_ mathomaties
87.04
miscallaneuus
06.6
college
microocconomicc
medical _geeeticc
high, school, psychology
hig_schhoo__o gph_school_comuteee cempoleeepppeeeeeee
Accuracy
84-
marketing
surenieeruality
astronomyy
high school geography
ligh_ _cchool biology
world religioss
High_sshe sigh_ychoo__ttee siatisiis
high school us jistory
professional medicine
78
prehistory
hummanitees other
STEM sciences
social
dinical knowledge
(c) Accuracy Subcategory
saciology
college wwwputorrreecocccccccccccc science
high school macronconamics
business ethics
foroorrgggpeee policy
goography
management
conceptual, physics
high schooll _tt_choo__mtttematees mathhematics
other
philoophhy
nutrition
business
college_ physics
logical fallacies
jurisprudecce
international Jaw
computer pecurity
professional, psycholooyy
psychriogy
human, _aging
high, school europsan history
economics
highh school chemistry
college, medicine
computer
moral dispures
machine learning
physics
high, school, physics
high school world histary
hstory
professional accounting
moral scenarios
politis
fomal logic
globelfects
philosophy
anstomy
aboeract_olgebra
health
public relations
college_ma mathemmtttsssiss
electrical Lenginering
chemistry
security studies
college, chemistry econometrics
engineering
professional Jaw
GPT4 Code Interpreter
viroloy
Overall: 84.43
Overall: 84.43
With Verification
300
60
Accuracy
Accuracy
Figure 8: Performance of GPT4-Code with CSV prompt on MMLU dataset on: (a) subsubcategory, (b)
category, (c) subcategory
16

---

Published as a conference paper at ICLR 2024
E EXAMPLES OF SELF-DEBUGGING MECHANISM IN GPT4-CODE
Math Problem: 0.54 as fraction in lowest terms.
Sure, we can solve this problem by setting up an equation, solving it, and simplifying the fraction.
Let's call 0.54 as æ.
The decimal 0.54 means that the digits "54" repeat indefinitely. So, we can multiply by 100 (since "54" is two-digit number) get another equation.
This gives us 100z 54.54.
By subtracting the original equation from this new one, we eliminate the repeating decimal: 100. 54.54 0.54.
Let's perform these steps in the next Python cell.
deZine the ayabol eymbolls
detine the equatioon e2l Eqil,
approxinatine the inp decimal
RESULT
TypeRrrar
Traceback
Imost
recent
call leet]
Apologies for the mistake. We can't directly subtract two equations in Python. Instead, we should subtract the right-hand sides and left-hand sides of the equations separately to get equation. Let's correct that.
subtract Eight sides wimpli (w/22.2.. mgl.cla) the simpliry (ogglliyyy/gg2211 (eq2.1ha eqi.
sides equations soparately
RESULT
E99 54.0)
After subtraction, we get the equation 99r
Next, solve for express 0.54 fraction.
solve For
sclution colve
sclution[0]
RESULT
0.545454545454545
The solution for is approximately
0.545454545454545, but we need to express it a fraction.
Let's convert this decimal into a fraction.
convert the cocimal solution traction fractioon (o
Fraution
RESULT
6/11
The decimal 0.54 can be expressed as the fraction 6
IT
Answer:
Table An example which GPT4-Code self-debugs when the code execution emits an emoor
17

---

Published as a conference paper at ICLR 2024
Math Problem: A bus comes by Jerry's bus stop every 20 minutes starting exactly 5:13 a.m. If Jerry shows up at exactly 8:35 a.m., how many minutes will he have to wait for the next bus?
Step I: Calculate the total minutes from 5:13 a.m. to 8:35 a.m.
Total minutes from 5:13 a.m. 8:35 a.m. 180 + 22 minutes.
Step 2: Calculate how many buses have come by during this time.
Step 3: Calculate the time when the last bus before 8:35 a.m. came by.
'The time when the last bus came by 5 a.m. + 20 x [n] minutes.
Step 4: Calculate the waiting time.
Now, let's calculate each step.
Step 1: Calcuiato the total ninutes fron 5:13 a.m. to 8:35 a.n. miinn_n 180 22
RESULT -311
It seems that there's 's mistake in the calculations. The waiting time cannot be negative. This error might have occurred in Step 3, where calculated the time of the last bus in minutes past midnight, but then in Step 4, subtracted this from the total time from 5:13 a.m. to 8:35 a.m. I should correct Step 4 to calculate the time until the next bus, which is either 0 if a bus has just arrived 8:35 a.m., or 20 minutes minus the time since the last bus if a bus arrived before 8:35 a.m.
Let's correct this.
Conrect Seep Caloulate the time unt the next bus sine_stncs__asee last__baa_______ total last_bus_ting
RESULT
331
There still seems to mistake in the calculations. The waiting time calculated significantly more than 20 minutes, which is not possible given that the bus comes every 20 minutes. Looking back the calculations, seems that the error is in Step 3 when calculating the time the last bus came. The last bus time should be calculated from 5:13 a.m., not from midnight. Let's correct this.
lest becuuw 8:35 Coe Step Celoulate the bune 20
bast_bua ciiee_oorreeeeoeeeeooeoooeeeeoee porsected
RESULT
331
It seems there is still an error. Let's try a different approach.
1. Calculate the total minutes that have passed since the first bus at 5:13 a.m. until 8:35 a.m.
2. Since the buses come every 20 minutes, calculate how many complete cycles of 20 minutes have passed in the calculated total minutes.
3. The time Jerry will have to wait is the remainder of the total minutes divided by 20 (which is the time since the last bus) subtracted from 20 (which is the bus cycle). This approach should give us the correct waiting time. Let's calculate it.
calculate total minaiminutes 60
Zrom 5:13 8:35
heuza and minutes
RESULT
18
So, Jerry will have wait for 18 minutes for the next bus.
Answer: 18
Table 9: An example in which GPT4-Code adjusts its solution based on the result of code execution.
18

---

Published as a conference paper at ICLR 2024
F EXAMPLES OF NATURAL LANGUAGE VERIFICATION
Math Problem: What is the sum of the digits in the terminating decimal representation of the fraction 57-28 ?
code 0
code
code more
The given fraction is
Answer:
Answer:
First, need simplify the fraction.
The denominator can be rewritten 5muz(u.7-B) 10mix(7.8) 2maz(0.8--1)
4321
57.20
Given that 4321 = 2a 3
To simplify this fraction, we can
5" 7 11 13' we can
where min(a, b) the minimum a and and maz(a, the maximum of a and b. Now let's calculate the value of min (7,8), 8), ma 7 - 8), and max (, 7).
notice that 5² 2
10".
simplify fraction 4321
which would mean the fraction is equivalent to 4321 x 10"
3 13'
67.23 2-
i.e. number 4321 with decimal point moved places to the left. This would result in number 0.0004321.
Next, we need to compute the decimal representation and sum the digits.
min_ab ninte, bt max m_nnmin max 2-81 max D. minuu max beal
To get the sum of the digits in the terminating decimal representation of this number, we just need to add up the digits in 4321. since the zeros do not contribute the So, the sum is 4+3+++4- 2+1== 10. Therefore, the answer is 10
Compute decinal represeneation
traction
RESULT:
value (3++3 13++2) (2++2
(7, 0,
5++3)
Now, let's simplify the fraction 4321
Corvest to string and replace
point ste_value str ssoplacs(".
X2T
nonerator
danoninator
t Compute the sun of the digits dig-t__uum sun(ind (digit) fuz digit in etr_value)
aumerator 4321
denominato: 20 5..0 2
digit_suu
RESULT: 0.00021605
RESULT: 18
We exclude the zeros hefore the first non-zero digit and the decimal point contribute to the sum.
they do not
Thus, the sum of the digits is the of 2, 1, 6, O, and 5. Let's calculate
digita
6, 51
i colculate the sun of the digits sum of of_.iiiitettee sonteigits)
RESULT: 14
So, the final answer is: 14
Answer:
Table 10: An example in which no restriction on code use succeeds while not using code and using code only once fails.
19

---

Published as a conference paper at ICLR 2024
Math Problem: Solve for æ 32" + 19 10".
code 0
code
code more
This is a difficult equation to solve and there's no simple analytical solution using elementary functions. In real world scenarios, such equations typically solved numerically. For example, using graphing calculator, computer, mathematical software.
Answer:
Answer:
The equation you're asking to solve, 3 19 10", cannot be solved algebraically due the nature of the functions and operations involved. Therefore, use numerical
Rewrite the equation as:
3 10 + 19
Let's denote 3 10" so a²
a, and
we'll need methods such as Newton's method or Bisection method for root finding.
now we have two equations:
10" Let's solve this with Python.
However, if you want approximation of the solution, numerical
it advisable methods such as bisection method, Newton's method, or others.
Crom scipy. sipy.optimnize import root
From acipy. optinize import fsolva sumpy
systen equatione def equet lone (cars)
Detining function def func DO)
x, 3, va2s
If you would like me to provide an algorithm to find the solution numerically, let me know.
sacuzn 3x- 19 10+
eql 34xx
e92 10rwx
Initial quess fox it can be
a.12 b [eqi, wg3]
anything, 1.0 is chosen
arbitrarily
RESULT:
t Joing focive to find the root
0.0
roos Esulve (Funm, x0;
3 19 10
root
The left-hand side becomes 1+19=20 and the right-hand side
RESULT 1.001831315399663
becomes 1. Thus, the original equation does not hold true for x=0.
So the answer is 1.00.
Let's try a different initial guess for the solution to see if that helps.
Adjust inilial For 11,
ayalen of equatioon Liens solution root laquationa, x0)
sclution.x:0]
RESULT:
2.00000000004446
We know that 3 = 3 81 and 10 = 100. The left-hand side becomes 100 and the right- hand side is also 100. Thus, the original equation holds true for 2.
Answer: 2
Table 11: An example in which restriction code succeeds while not using code and using code only once fails.
20

---

Published as a conference paper at ICLR 2024
G EXAMPLES OF DIFFERENT VERIFICATION RESULTS
Math Problem: Let A = (1, -4,-10 11, 2), = (3, 4, 1), and C = (- Compute LABC, in degrees.
True
Uncertain
False
The angle ABC is the angle between the vectors BA and BC. We can find these vectors as follows:
the cosine of the angle 0
The vector AB is given by B A, and the vector BC is given by C B.
between them is given by
v
|
The cosine of the angle between two vectors and v given by
In the given problem, we want to find the angle between vectors AB and BC, where
BA = --
cos(O)
1u |
Let's calculate.
BC - (-5,5,
AB A (2,7, = CC-
import nuppy as op
The angle 0 between two vectors 7 and e can be found using the formula
detine the pointe A og.array (T2, -11, 21)
Now, we'll calculate the angle using the formula for A.
RESULT:
cos (os(0)
60.0000000000001
2
|o
tron numpy import acroy, dot Exon nurey.linalg impost noen Impozt math
The angle ABC is
Let's calculate these values.
approximately 60.0". Let's
verify this. We can solve for cos(C))
Derining arrayill, 21)
Derine vecters WA op.acray (E), 11, 21)
cos(C) = a² 6² c²
1l -dy 111
RESULT:
2ab
60
RESULT:
The angle LABC is 60%.
10.000000000000001
calculace vector AC
Let's now verify this answer.
colculate magnitudes
7 7= |ل||| || cos(O)
negnitude AC ng. inw.iinnlg.ommm(AA
Substituting our values into this formula, should get the same dot product (-27) that calculated earlier. Let's check this.
RESULT:
False
I apologize for the confusion; there seems to have been a mistake in the verification.
The discrepancy arises due to the directionality of the vectors. Therefore, the initial calculation was correct: the angle ABC is approximately 60.0".
Corpute the dot product using the formula
dot pwwpppocttteriiiiotto.. negui aegnit.de_6C rp..oos itheta ap.pi
RESULT:
-27
which matches the dot product we calculated carlier. Therefore, the computed angle 120" conrect.
Answer: 120"
Table 12: An example in which True succeeds while Uncertain and False fails.
21

---

Published as a conference paper at ICLR 2024
Math Problem: What the sum all positive integers r that satisty lem(r,
700] 7000
True
Uncertain
False
The least common multiple
First, let's find the prime factorization of 7000.
this problem, we are given that lem(r, 700) = 7000. Let's first find the prime factors of 700 and 7000.
(LCM) two numbers a and is given by the formula:
Then, we'll generate all the factors of 7000 and find which those factors are multiples of 700.
LCM(a, b) = Ja bl
GCD(a,
impazt
Finally, we'll sum up all the positive integers that satisfy the condition.
Find prine Factori
Hence, the equation can be rewritten
700
ractors 700 sympy cmmpytacttorne
1700)
r x 700
GrD 700)
7000
import εупру
RESULT:
the prine cactorinacion
(2: 2, 5: 2, 7: 3, 1) Since the LCM of and 700
3,
We also know that must be a divisor of 7000. So, we'll find all the divisors of 7000.
pa gri_e___aai faclorx - syapy, Caclorini syngy,Cacuocint
103C
17300]
is 7000, r must contain all the
Gererate all factors of 3000
prime factors of 7000 that are not already in 700.
set
ror tactor_2_court
ranger
pz pzina_ factorr 2)
Find all divisers 700C divisors_7000 syrey.divisor 17000)
for factor___ccunt in zanzel prine [as 11:
conbinations 14ss (produect in. axtra wat.oa_aaouoree
Filter out the divisors
r. Me laee corrected Ir for
RESULT:
divisors_7000 12 1030
7002 7001
(2: 3, 5: 3, 7: 1, [3500, 700,
RESULT: ([1, 5, 2, 10), 18)
7000, 1400], 12600)
RESULT:
The prime factorization of 7000
The possible values r are 1, 2,
(7000, 1000)
is 2s 2.255..77 The factors of 7000
and 10.
Let's verify our solution by checking whether the LCM of
that are multiples of 700
To verify our answer, we can
3500, 700, 7000, and 1400. So,
check that the least common
each possible and 700 is indeed 7000.
the sum of all positive integers r
multiple of and 700 is indeed 7000.
satisfy lem(r, 700) 7000
3500 700 7000 + 14100
Verlfy the son
12600.
Reneva values pessible possible_r_valon www.oa.......a.c...
con_vaiues__oo cocretted [aympy Jem In, 7003 for e in
So the final answer is 12600.
RESULT:
RESULT:
False
(True, True)
seems that there's mistake. Not all the calculated values of satisfy the given condition.
Upon verification, we find that
the least common multiple
(LCM) of each possible value of
Let's correct the calculation.
and 700 is indeed 7000. Henee,
the of all positive integers
$ All Lhe values of abould have the factore ot 700 [300]
that satisfy lem[r. 700] 7000
is 1000 + 7000 8000. The answer is 8000.
RESULT:
(7700, 1400, 3500, 7000J, 18720) Therefore, answer is 18720.
Answer: 8000
Table 13: example which True succeeds while Uncertain and False fails.
22

---

Published as a conference paper at ICLR 2024
H FAILURE EXAMPLES OF NATURAL LANGUAGE VERIFICATION
Math Problem: Let = 4n + 6n + 4n + 1. Find 0s + 02 + 030 + +
Qus-
The sequence dn 4n + 6rl² + 4n 1 is cubic sequence. To find the sum + ag + d1o + + ag3, we can use the property of cubic sequences which states that the sum of consecutive terms in a cubic sequence forms a quartic sequence. Given that the sum of the first n terms Let's calculate this.
Define the Funct for sun of the First berne det Sin):
return need - 2xreed 24 + caloulate the aum of tomns crom 9_8 to aj23 23 (23) 5(7)
B_10__3
RESULT 302064
The sum of the terms from to 023 is
302064
Now, let's verify this result. The sum of the terms from ag to 023 is equivalent to the sum of the terms On from
23. This is equal
4 8 6 8 + 4 + 9 + 6 92
4 23 + 6 -23 4 23
We can't calculate this manually without calculator or any programming language. However, still verify the result by checking the properties of the sequence and the sum formula. Given the sequence a, = 4n 6n +4n+ 1, the sum of the first terms is quartic sequence, which is given by S₂ n + 2n + 2n2 + This is derived correctly from the sequence by term-by-term integration. The sum of terms from ag 023 is found by subtracting the sum of the first terms from the sum of the first 23 terms, which is S23 - Sz. This is also a correct approach. Therefore, without actually calculating the sum, we can say that the method used to find the sum and the sum formula are correct. Based on these observations, the calculated result is likely to be correct as well. So, the verification result is:
TRUE
Answer: 302064 X
Table 14: An example in which only natural language verification is used and the model failed to find the mistake.
23

---

Published as a conference paper at ICLR 2024
Math Problem: Compute Ex k-1
R+2 (++(++1)+(++2)11
The sum
k + 2
kl
(k 1)! + 2)!
can be rearranged recognize pattern
where e is the base of the natural logarithm.Non Now let's verify the result. Recall that the series
1
1)!
is shifted version of the series
k!
k=
which is known to converge to the base of the natural logarithm. The original series is missing the 0! term, which is equal to we subtract from to get the of the original series. Therefore, based on the properties of the series and the value e, it seems reasonable to conclude that the solution is 1. Therefore, the verification result is:
TRUE
Answer: e X
Table 15: An example in which only natural language verification is used and the model failed to find the mistake.
24

---

Published as a conference paper at ICLR 2024
I DISTRIBUTION OF THE CONSISTENCY BETWEEN REASONING PROCESS AND VERIFICATION PROCESS
To further analyze the consistency between the verification process, NL reasoning process, and output correctness, have collected examples of four kinds from MATH shown Fig. 9 The reasoning process is correct if the answer before the verification is correct. The verification process is correct if the verification process itself is correct. The four kinds of examples are explained below:
Reasoning: Correct, Verification: Correct. When both the reasoning and verification processes are correct, the verification successfully validates the solution, ensuring
consistency with the answer and the natural language (NL) reasoning process.
Correct, Verification: Wrong. the reasoning is correct but the verification is emoneous, the verification fails to recognize the correct solution.
Reasoning:
Wrong, Verification: Correct. If the reasoning is incorrect but the verification is accurate, the verification process amends the erroneous reasoning, leading to a correct answer.
Reasoning:
Reasoning:
Wrong, Verification: Wrong. If both the reasoning and verification processes are incorrect, the verification is unable to rectify the flawed solution.
Fig. demonstrates that in most of the samples, reasoning process and verification process are consistent with each other.
7.0%
12.0%
5.0%
Bewsoning: Cornect
Verification: Correct
Reasoning: Correct
Verification: Wrong
Reasoning: Whong
Verification:
Rewsoning: Whong
Verification: Wrong
76.0%
Figure 9: The distribution of the consistency between reasoning process and verification process on 100 randomly sampled math word problems from MATH.
ANALYSIS OF DIFFERENT TYPES OF VERIFICATION CODE
To further analyze what types verification code are generated, we randomly sampled 50 problems from MATH as well as their solutions generated with CSV. We counted types of verification code: Substitution, Alternative Method, Double Checking, and Approximation. Their distribution is show Fig. I0
The model substitutes the answer back the question to check if fits all the conditions. By inspecting the solutions we randomly sampled from the results, we found that this is the most common type of verification.
Substitution:
Alternative Method: The model uses different method to solve the problem and check if it reaches the same final answer.
25

---

Published as a conference paper at ICLR 2024
Double Checking: The model double-checks each part of the solution see if they correct.
The model computes approximation of the answer. This method is often used when there is infinity involved the problems. The model uses a large number to approximate the infinity to check if the result is close to the answer.
Approximation:
Examples of the types above are shown in Tab. 16
Tab. 17
Tab.
18
Tab. 19
18.0%
Substhuue Bon
Altemattve Method
22.0%
Double Checking Appresimation
50.0%
Figure 10: The distribution of verification types generated by GPT4-Code 50 randomly sampled math word problems from MATH.
Math Problem: Suppose that the minimum value f(x) cos 2x 2a + cos x) Find
Next, we will verify this solution by substituting a = 0.25 into the function f(x), and checking whether the minimum value f(x) is indeed We will evaluate f(x) over the interval [0, 2T] because the function is periodic with period 2n.
Substiture -0.25 pusstituued t. 13, -0.25)
Define a function for numorical evaloation Fuum sp. sp.bandd:1 18, "_aubet ttaee,, "rampy")
Evaluate
interval 2xpi]
aurey
x_values tp.1incpase 2xng.pi, 1600)
Lvaluus f E_nunx_yvalues
Rind the misimm value of (())
min ap.nia iop-niate_valvee
Table 16:
An example of verification code that used substitution. The model substitutes the answer back to the question to check if it fits all the conditions.
26

---

Published as a conference paper at ICLR 2024
Math Problem: What is the value of the following expression: 1 - 99 + 98 97 + 96 95 +
4 - 3 2 1- 1?
Now, let's verify this result by actually computing the expression term by term. We'll create a list of numbers from 100 down to 1, then add the even-indexed numbers and subtract the odd-indexed numbers.
create numbers Tiat Inwrge 9, -131
numbers
cuaguie Lhe expreasion Lern by Leen
cetal_check For:
(numbers):
totel_check nun
elees totck_chock - mun
cetal_check
Table 17:
An example of verification code that used alternative methods. The model different method solve the problem and check if reaches the same final answer.
Math Problem: What is the greatest prime factor of the sum of the arithmetic sequence 1+2+3+ .4. +00
Now, let's verify the solution. First, we will generate the sequence from to 80, sum it up, and confirm that it is equal to 3240. Then, we will calculate the product of the prime factors and confirm that it is also equal to 3240. Finally, will check that the maximum prime factor is indeed 5.
create 11st of numbera Ison 100 down to
Generate the seguende Fron 1ist (range
80
Conflen the aspert sun (ssquance) en 8, *The pun of the sequenes is not correcz."
sequence
Contien the prosuct oz the prine tactore
from Functools import reduze
impost
assert reduce (oporator. mal, reima_tactora__
prodect poima factors not courect."
Confirm the neximum prine factor
assert max (pe ine___eet_ttt Facturs_8) =- ge inct The meximum prine factor is not correct,"
Table 18: An example of verification code that used double checking. The model double checks each part of the solution to see if they are correct.
Math Problem: What the value of Do0 (Lan
vni
Lan Vn I)?
Now, let's verify the solution. We can do this by calculating the sum of the first few terms of the series and see if it is close to our solution. Note that the series is infinite, so we can't 't compute the exact value in this way, but Let's calculate the sum of the first 10,000 terms.
check if it seems to be converging towards our solution.
calculate the sum of the riest 10000 torms
in pange/1. 10001):
sun_of_terns
(nath.agrt (n)) hnarh
cun cun_ot tememt
Table 19:
An example of verification code that used approximation. approximate the infinity to check if the result is close to the answer.
The model use a large number to
27


Reviews:
[{"review_id": "vlsGuwEwdw", "reviewer": "Reviewer_iedh", "summary": "This paper proposes a zero-shot prompting strategy for LLMs that can execute code during a completion (e.g. GPT-4 Code Interpreter). The strategy, referred to as CSV (code-based self-verification), is to use the following zero-shot prompt: \"Solve the problem using code interpreter step by step, even in every sub-step. And following your answer, please verify it using code interpreter by yourself.\" They find that this prompt boosts performance on several math word problem datasets. The authors then present a variation on majority voting based on weighting samples that were verified as True more heavily. The authors explore various ablations/variations, and also find a strong correlation between amount of code use and accuracy in general (ablations that can only use their code interpreter zero or one times perform worse).", "questions": "- Can you address or respond to the main 4 points made in the weaknesses section?\n    - The majority voting ablation addition to Table 1\n    - The majority voting + no CSV ablation addition to Table 1\n    - The missing rows of Table 2 and Table 3\n    - The confusion around Fig 6a\n\nMore minor questions:\n- In Fig 2 why are some of the red dots different sizes given that the prompt says no code is allowed – does that mean in some cases it's using code anyways?\n- \"Moreover, we observed a decline in accuracy for 4 of the seven subtopics, indicating that relying solely on natural language self-verification can not only compromise accuracy but also negatively impact performance\" What does this line mean – aren't accuracy and performance the same thing? \n- Do you have a sense for why the natural language verification row in Table 2 can fairly significantly *decrease* performance (by 2-3% in some domains)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The results on correlations between code usage frequency and accuracy are a nice analysis, and breaking it down by difficulty level of problem is useful (e.g. at higher difficulty levels amount of code usage seems to matter more)\n- The CSV setup itself is quite simple, which is a good thing – \"Solve the problem using code interpreter step by step, even in every sub-step. And following your answer, please verify it using code interpreter by yourself.\" A simple way of getting a decent boost in performance from zero-shot prompting is a fairly useful contribution. The boost of 3.85% accuracy on the MATH dataset from using CSV over the base prompt is solid – that's a reasonable gain.\n- The Verification-guided weighted majority voting setup is also quite simple (just a weighted majority vote with human-picked parameters) which is good. The weighting seems to provide a boost of around +1% accuracy over a standard majority voting approach (at 16 samples, MATH dataset, per Fig 6b).\n- The precision-recall-accuracy analysis and also analysis of weighted majority vs naive majority are good to see, I was preparing to suggest that, and I'm glad it was done.", "weaknesses": "Overall I'm marginally below acceptance on this paper, but would certainly consider raising my score if the following points are well addressed in revisions/rebuttal.\n- As-is the current paper has only one row in Table 1 that uses 16 samples with GPT4-Code, so the +14.63% improvement is a comparison between a 1 sample and 16 sample method. This can be a bit misleading to readers (or at least it was to me) since much of that improvement comes from using additional samples, which is something that could also be done through prior work via naive majority voting (which is actually evaluated later in the paper, in Fig 6b). Therefore, Table 1 should include a line for **GPT4-Code + CSV + Majority Voting** (which does about 1% worse than weighted voting, per Fig 6b).\n- For the same reasons as in the previous comment, adding a line to Table 1 with **GPT4-Code + Majority Voting (no CSV)**, an ablation that done with neither CSV nor weighted voting, would be an informative baseline to include. These two baselines would considerably strengthen the analysis.\n- There doesn't seem to be a **GPT4-Code + CSV** row in Table 2, nor a **GPT4-Code + CSV + Voting** row in Table 3. This use of different setups on different datasets is confusing – an explanation should be given, or preferably both should be added. Having both of these novel methods from the paper evaluated on all 3 datasets would strengthen the paper.\n- Figure 6a is quite confusing to me:\n    - Which dataset is used? I believe it's MATH overall dataset, but this should be in the caption and optionally also main text\n    - I don't understand why there are 5 separate paths graphed, given that there's nothing special differentiating path 0 from path 1, etc. The different path indices are just IID samples, so of course when you calculate precision/accuracy/recall averaged over a whole dataset of problems, all the histogram bars will look the same for different sample path indices. So having 5 separate paths doesn't seem to add anything – why not just collapse them all into a single graph with three bars (accuracy, precision, recall)?\n\nminor:\n- Consider weakening the statement \"Each line in Fig. 5 has an obvious trend of going upwards, proving that the increase of Code Usage Frequency induces a general improvement in accuracy\": *Proves* is a strong word there, when it's just a nice correlation (not causal – there are many ways of including meaningless code in an output that wouldn't not induce an improvement in accuracy). Just state it as a correlation and that's good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LAj677FfFc", "reviewer": "Reviewer_qqMb", "summary": "This paper studies the effectiveness of code writing and self-validation for improving the reasoning capability of large language models (LLMs). Given that GPT-4 can generate Python programs, instead of providing step-by-step reasoning entirely in natural language text, this paper suggests a new prompt that encourages GPT-4 to generate both text and Python code snippets as well as a self-validation in the end. Additionally, a weighted majority voting is proposed to aggregate multiple runs or samples. The evaluation on three math datasets shows that GPT-4 with proposed prompting techniques significantly outperforms the previous state-of-the-art results.", "questions": "Would the same idea improve other LLMs like Llama 2? Or what factors may prevent the same idea from improving other LLMs? \n\nIn Table 3, the last row only mentions _CSV_ (but not Voting), is it a typo? If not, why voting is not enabled?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Like many recent works, this work provides strong evidence that prompting LLMs in a proper way could significantly influence the performance\n- This work also makes another natural but still interesting finding -- the frequency of code usage has a strong correlation with the accuracy of the final answer. The rationality is that executing code is more accurate/reliable than performing reasoning through natural language text. \n- This work presents the new state-of-the-art results on three datasets (i.e., MATH, GSM8K, MMLU-Math).", "weaknesses": "- In terms of improving LLMs, the ideas of code writing, self-validating, and majority voting have been already explored in recent literature. The novelty of this work seems a simple combination of all three ideas together. \n\n- Only one particular and proprietary LLM (i.e., GPT-4) is used for evaluation. Whether the results and findings reported in this work may generalize to other publicly available LLMs (e.g., Llama 2) or relatively smaller language models is unclear. \n\n- Another similar concern is that only one specific kind of benchmark (i.e., grade school math problems) is used, thus the proposed prompt might be overfitting to simple math problems.  For the chosen benchmarks, improvements due to coding and CSV on GSM8K and MMLU-Math are already fairly small (i.e., 2-4%). \n\n- The improvement seems largely due to the superior capability of GPT-4. For instance, GPT-4 itself can outperform the state-of-the-art by a large margin without any sophisticated prompting and majority voting. It is already well-known that even a simple prompt could dramatically influence performance (see ). The particular finding of GPT-4 on the MATH benchmark is not very surprising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7uD78Lo0gM", "reviewer": "Reviewer_boig", "summary": "This paper presents a prompting based self-verification technique to improve GPT-4 CodeIntepreter model's performance on math tasks. The key ideas are: (1) by prompting the model to perform code-based self-verification, the model improves its performance, and (2) taking advantage of self-verification result for weighted-voting, the model further improves its performance.\n\nThe paper is inline with existing lines of work of leveraging model's self-debugging ability to improve math reasoning ability. The experiments showed that (1) by leveraging multiple invocation of code call, GPT-4 Code achieve higher performance comparing to NL or 1-code call variances, (2) self-verification provides the opportunity for the model to discover its own error thus to fix it, and (3) voting solutions based on self-verification results achieves better performance over vallina voting scheme.", "questions": "I think the paper lacks a qualitative analysis of the self-verification code and analysis of consistency between answer and verification code. Otherwise a great paper showing the ability of self-verification.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper is timely. It's a novel approach of leveraging code self-verification to improve math reasoning ability. It's simplicity means it can be easily used by GPT-4 users, and the thorough study provides solid evidence of adopting this technique.\n\nConcretely, the paper's strengths include:\n1. Novel technique that combines self-debugging and test generation for math reasoning tasks.\n2. Quite complete study that highlights effectiveness of the technique.\n3. Weighted voting technique is quite unique, and it can possibly used in other self-debugging techniques beyond Math reasoning tasks.", "weaknesses": "The paper lacks some insights into the quality of self-verification results, and how that matters for model performance.\n1. The paper can potentially dive deeper into analysis of consistency between verification process and NL reasoning process as well as output correctness. As shown in prior work like CodeT, some self-generated test-cases or verification code can either be wrong or inconsistent, but it may or may not affect model output quality. Guessing from the paper's results on weighted voting versus simple voting, such inconsistency exists and they could benefit model performance (or affect model performance if we simply reject such answers). I would suggest the authors perform some qualitative analysis to dive into this problem.\n\n1b. Some deeper qualitative analysis into what types of verification code are generated would also be helpful.\n\n2.The paper considers both sampling and sequential self-repair. It would be great if the authors can analyze the tradeoff between the depths of self-repair (e.g., if self-verification continues to fail, how much can the model benefit from continuing self-repair until verification succeed) verse breaths of self-verification (e.g., simply repair once, but generate multiple samples to do weighted sampling).\n\n3. The paper plots are visually misleading: Figure 2 accuracy numbers should all start from 0 as opposed to 60 in figure 2a and 40 in figure 2b. Otherwise the improvement looks like 10x as opposed to 6% comparing prompts 1 and 2. Similarly for other figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q933vlNNOH", "reviewer": "Reviewer_UL1M", "summary": "The paper attempts OpenAI's new GPT4 with Python Interpreter version. They reveal that incorporating code interpreter as an external could increase LLM's performance on math tasks. The more times using the interpreter, the higher performance would have. In addition, by using the self-debug method with the new GPT4 w/ code, they further push the SoTA of MATH dataset to 84.3%.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "1. A good demonstration of OpenAI's new GPT4 with Python Interpreter version. The experiments show the great improvement of the new model comparing w/ the traditional GPT4. It also shows the potential of using code interpreter as an external tool to enhance performance on math reasoning tasks.\n2. They authors push the SoTA of MATH to 84.3%, which is a very high number considering its complexity.", "weaknesses": "1. Most of the methods discussed in the paper are proposed by existing works and the main contribution of this paper is to try them out using the new OpenAI model. The contribution and novelty could be a weakness of the paper.\n2. Most credit of the huge improvement on MATH should be given to the better capability of GPT4-code itself. The authors seem to over claim their own contribution throughout the paper. For example in Page 6, the authors said \"Before the advent of GPT4-Code, prior frameworks (Lightman et al., 2023; Cobbe et al., 2021) depended on an external LLM to use natural language for verification and well-designed fewshot example prompts. In contrast, our approach simplifies the process by relying solely on a straightforward prompt for GPT4-Code, all in a **zero-shot** manner.\" But the root reason that zero-shot is applicable is because you are using OpenAI's instruction-tuned model. In addition, it is very likely that OpenAI has already used PRM (Lightman et al., 2023) to RL tune it's new model, therefore the API you used could be based on top of Lightman et al., 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)