SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning
Authors: Ning Miao
Yee Whye Teh
Tom Rainforth
Venue: ICLR 2024 poster (2024)
Keywords: large language models
LLMs
reasoning
Primary area: generative models

Abstract: The recent progress in large language models (LLMs), especially the invention of chain-of-thought prompting, has made it possible to automatically answer questions by stepwise reasoning. However, when faced with more complicated problems that require non-linear thinking, even the strongest LLMs make mistakes. To address this, we explore whether LLMs are able to recognize errors in their own step-by-step reasoning, without resorting to external resources. To this end, we propose SelfCheck, a general-purpose zero-shot verification schema for recognizing such errors. We then use the results of these checks to improve question-answering performance by conducting weighted voting on multiple solutions to the question. We test SelfCheck on math- and logic-based datasets and find that it successfully recognizes errors and, in turn, increases final answer accuracies.

Published as a conference paper at ICLR 2024
SELFCHECK: USING LLMs TO ZERO-SHOT CHECK THEIR OWN STEP-BY-STEP REASONING
Ning Miao Yee Whye Teh
Tom Rainforth'
ABSTRACT
The recent progress large language models (LLMss, especially the invention of chain-of- prompting, has made it possible automatically answer questions by stepwise reasoning. However, when faced with more complicated problems that require non-linear thinking, even the strongest LLMs make mistakes. To address this, we explore whether LLMs are able to recognize errors in their own step-by by- step reasoning, without resorting external resources. To this end, propose SelfCheck, a general-purpose zero-shot verification schema for recognizing such errors. We then the results of these checks to improve question-answerii performance by conducting weighted voting on multiple solutions to the question. We test SelfCheck on math- and logic-based datasets and find that successfully recognizes errors and, in turn, increases final answer accuracies.
INTRODUCTION
Recent years have witnessed dramatic changes in the areas of NLP and AI brought on by significant advances in LLMs. From GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), Llama (Tou- vron et al., 2023) and Falcon (Almazrouei al., 2023) to GPT-4 (OpenAI, 2023) and PaLM-2 (Google, 2023), the increasing model sizes and exploding amount of training data have empowered LLMs to achieve human-level performance on a large range of tasks, including summarization, translation, and question answering. The invention of Chain-of- ofhin-of-Thouuht prompting (Co'T, Wei et al. (2022)) has further enhanced LLMs ability to solve complex problems by generating step-by-step solutions.
However, the performance of even the largest LLMs is still unsatisfactory more difficult reasoning problems. For example, GPT-4 with CoT prompting only correctly answers 42.5% of problems in the MATH dataset (Bubeck et al., 2023; Hendrycks et al., 2021), which is far below the human level. Such problems require careful multi-step reasoning to solve, and LLMs are consequently prone to make mistakes: even though their error rate individual steps may be low, the probability of generating at least one erroneous step can still be quite high, undermining the final answer.
Recent works have tried to overcome this limitation by checking for errors in these step-by-step solutions (Cobbe et al., 2021: Li et al., 2022; Ling et al., 2023: Jiang et al., 2023). Such checks can then be used to provide confidence scores in answers and select between different possible alternatives. This checking has typically been performed either by using an external verification model (Cobbe al., 2021: Lyu et al., 2023; Peng et al., 2023), or through few-shot in-context learning (Brown et al., 2020) of an LLM (Weng et al., 2022; Ling et al., 2023). Unfortunately, existing methods generally require extra training data and/or domain-specific exemplars, which often makes them inconvenient to use in practice and restricts them to specific domains or data formats. The aim of our work is thus instead provide a general-purpose, zero-shot, approach checking that relies only on the original LLM, without the need for additional external resources.
To this end, we introduce SelfCheck, a zero-shot step-by-step checker for self-identifying errors in LLM reasoning chains. SelfCheck uses the L.MM to individually check the conditional correctness of each step based on directly related information from the question and the preceding steps, in a manner similar human going back to check their work. The results of these individual checks are then integrated to form an overall correctness estimation for the whole reasoning chain.
Key SelfCheck's 'ss success is novel mechanism for performing the checking of we will show, the naive approach of directly asking the LLM to check a step is typically ineffective.
individual steps. As
Department of Statistics, University of Oxford. 'Email: <ning miao@ stats.ox.eo ac..ukk.

---

Published as a conference paper at ICLR 2024
Instead, we introduce multi-stage approach that breaks the problem down into series of simpler tasks, leverages the generative strengths of the LLM, and decorrelates errors between the original generation and checking, Specifically, using separate calls to the LLM we first extract the target and relevant context for the step, then regenerate an independent alternative step from these, and finally compare the two. The original step is then deemed to pass the check if it matches the regeneration.
Besides providing an estimation of correctness for each solution, SelfCheck can also boost final answer accuracies for the original questions by weighted voting. Namely, given multiple solutions to a question, it uses confidence scores as weights to vote among the answers, which provides a soft way to focus on more accurate solutions.
We evaluate SelfCheck on three math tasks and one logical reasoning task. For all datasets, we find that using SelfCheck achieves a significant increase in final answer accuracies compared with simple majority voting and other baselines. We also see that SelfCheck provides an accurate confidence estimation for LLM's solutions, which decreases the proportion of incorrect solutions by between 9% and 23% (depending on dataset) when filtering out solutions with low confidence scores. We further perform a number of ablations to justify some of our key design choices in the SelfCheck approach.
To summarize, we introduce SelfCheck step-by-step reasoning in L.M Unlike previous methods, SelfCheck does not need any finetuning or example crafting, can be directly applied to reasoning tasks in different domains. Our experiments confirm that it can, in turn, be used to improve final predictive performance of LLMs.
novel and effective zero-shot schema for self-checking
RELATED WORK
How to automatically check the correctness of sequence of reasoning steps is long-standing question. We now discuss how previous methods have tried to tackle this in an LLM context. We note that none of these works are able work in the zero-shot setting covered by SelfCheck, requiring either problem-specin examples, an external model, and/or finetuming.
Few-shot verification Though our focus will be on zero-shot checking, for some problems one may have hand-crafted exemplars available that are specifically designed for that particular question- answering task. Previous methods have been designed to perform checking of LLMs' generated solutions in this few-shot checking scenario.
For example, the Self- Verification (SV) approach of Weng et al. (2022) verifies the whole solution by backward prediction. That is, it uses the conclusion from CoT reasoning to predict a masked condition in the question. However, it only supports single-step checking and is based on the assumption that every piece of information in the question can be recovered using a correct solution of it, which is often not the case. Consequently, it is only applicable simpler tasks, such as GSMSK.
The Deductive Verification (DV) approach of Ling et al. (2023) instead looks to verify independent sub-tasks, as per SelfCheck. However, its verifier only supports checking reasoning chains in special format called Natural Programs. As a result, it can only work with specific specialised generator, without serving general verifier for multi-step reasoning.
Verification with external resources In some cases, there might be external resources available to verify the logical correctness or faithfulness of LLM outputs. Lyu et al. (2023) translate a question into a symbolic reasoning chain using an LLM and solve the problem by symbolic logic solver. Peng et al. (2023) introduced an external database to check for incorrect knowledge in LLM outputs. These methods are limited by the availability of external resources and are typically restricted to checking for certain types of errors.
Training/finetunnng verifier A few other methods train or finetune separate verifier model to check reasoning chains. Cobbe et al. (2021) finetuned GPT-3 model on GSM8K to predict the correctness of solution as whole. Li et al. (2022) trained a binary deberta-v3-large (He et al., 2020) classifier on each domain to predict step correctness. More recently, Lightman et al. (2023) built a large dataset, which contains step-wise correctness labels from human labelers, and finetuned GPT-4 model on it. Unlike SelfCheck, all of these methods require extra data and external computational resources, restricting their applicability and ease of use.
2

---

Published as a conference paper at ICLR 2024
3 SELFCHECK: USING LLMS TO CHECK THEIR OWN REASONING
Rather than relying on external resources or problem-speciffe data like the aforementioned approaches, it would be highly beneficial if we could develop self-contained checking schemes that require only the original LLM itself. other words, we would like to use the LLM to identify errors in its own step-by-step reasoning, analogously to how a human might go back to check their working.
Unfortunately, directly asking the LLM to check its own reasoning largely ineffective: it almost invariably declares that the original answer is correct, with Ling et al. (2023) finding answers checked in this way are deemed correct more than 90% of the time regardless of whether they actually are. As we will show in Section 5, individually prompting the LLM to check each step in the CoT reasoning fares slightly better, but is still only able to offer marginal gains compared to not checking at all.
A more nuanced method to perform this checking is thus required. To this end, we introduce SelfCheck, a ceneerl-ppuroose zero-shot, checking schema for self-identifying errors in LLM CoT reasoning. Given a question, 9. and its step-by-step solution, 8, produced by some generator (which will generally be an LLM with appropriate CoT prompting), SelfCheck considers each step of in turn and tries to establish its individual correctness based on the preceding steps. This checking is done by leveraging an LLM (which can either be the same LLM used to generate s or a separate one), but rather than directly asking the LLM to perform the check, we instead introduce a novel step checking method (see Section 3.1) that exploits their generative modeling strengths. The results of the checks on individual steps are then combined into a single confidence score, we E [o, 1], for the whole solution. These confidence scores, in turn, allow us to improve predictive performance, by using them to perform weighted voting on multiple solutions to the same question.
3.1 STEP CHECKING
To check individual steps of reasoning process, the first thing we should note is that the correctness of each step is highly dependent on its context, namely the question and previous steps in the solution. For example, we usually need to refer to previous steps for the definition of variables and the meaning of specific numbers. If each step conditionally correct based on the provided context and the last step provides an answer in the required format, then the overall reasoning will itself be correct. The target of the step checking is thus simply to check the conditional correctness of each step based on the context provided by previous steps. That is, we only care about catching errors at the current step, and can assume all information from its context to be correct.
A simple idea to achieve this would be to feed the current step well as all its context an LLM and directly ask it to 'check the correctness of the step'. However, in practice, we find that this task is too difficult for current LLMs to do effectively, even with careful prompting that exemplifies how to do the checking in detail (see Section 5). This difficulty comes first from the fact that there are multiple aspects to the checking problem that the checker must deal with simultaneously: it needs to understand the key content in the step and then collect all related information from the context, before actually checking for its correctness. Second, LLMs are trained generative models, rather than directly as supervised approaches for checking. such that it is a problem that does not necessarily play to their strengths. Finally, there are likely be strong correlations between the errors such a checker will make with the errors made in the original generation, undermining its usefulness.
To address these difficulties, SelfCheck instead decomposes the checking task for each step into four stages: target extraction, information collection, step regeneration, and result comparison. The LLM used to execute each stage successively, with the outcome of the result comparison providing the correctness prediction.
The idea behind this decomposition to make the LLM focus on an easier task at each stage and ensure the individual tasks carried out are more closely aligned to the LLM's strengths. Moreover, by focusing on regenerating and then comparing, we hope to reduce the correlations between the errors of the checking and the original generation.
At a high level, the stages work by first prompting the LLM to figure out the target of the current step and what information it uses achieve the target; find that the LLM is usually able to perform these tasks extremely accurately. Then we ask the LLM to re-achieve the target using only the collected information, providing an alternative the original step that maintains the same purpose
3

---

Published as a conference paper at ICLR 2024
Step checking
Question:
Let the set of complex numbers such that the real part of H2js is equal to (1 This set forms a curve. Find the area of the negion inside the curve.
Stage Target extraction
step aquare equatioon
Stage Information collection
The nact direcly kcloo from Step
Generator
Checker
Stage Step regeneration
To reenite the ucuurionn standard forn of clicle, reed to completee The square for both Sx5 and Sy6.
Samalieine: 30ear9 yo2 - ig. This a now the standard form of dircle with center $13,0 and Taskus $38.
Multi-step reasoning:
SemieytS be complex number, real numbers.
Stage Result comparison
Stop 1. We are givan that the real part of 13/25 is equal to (
solutions.. However Solution
completion
results center the circle being $ and redius 588. In Selution the completion the squure rasuis the center of the circle being #(3/2,015 and a radius of $325.
Cross-muttillyng reamanghng in get the equatiio =x22* y22.5
Therefore, Solutton contradicte the conclusion Solution 2 as the centere and rddi of the ordee are cifferent
Stop & Completing the aquare, we obtain Silffi'
(ieeignn + y (hac
Step checking results: 1, 1, 1,
1,
region cirde S1rec(Be))))8S
Integration function: &
the 8tt (11411)))11118.
Confidence score:
Figure I: Example of using SelfCheck, focusing on the checking of a particular step (Step 5). To check the correctness of the step, SelfCheck goes through 4 stages. First, in the target extraction stage, it figures out that the main purpose of Step 5 is to complete the square. In the information collection stage, it then establishes that Step 5 only directly relies on Step 4. Next, the step regeneration stage instructs the LLM to complete the square independently, only using Step 4 as context. The regeneration result shows that the center and radius of the circle are (3, 0) and 3, which is different from what is implied by the original Step 5. Consequently, the result comparison stage concludes that Step 5 is likely to be wrong. After checking all the steps, SelfCheck integrates the results to form an overall confidence score, W. See Appendix A for complete version of the example.
in the overall reasoning process. Here the clear description of the target and the simplified context we provide make the regeneration stage less challenging. As result, we hope its output will be more reliable and thus serve as a useful reference. Even this not the case, it will still hopefully provide viable alternative, with a distinct generation, that can be used for comparison. The last stage then uses the LLM to compare the original step with the regenerated output. If their main conclusions match/mismatch. this provides evidence that the original step correct/incorrect.
worked example of this step-checking process is provided in Figure 1. In the following, we describe each of the subtasks in detail and provide our specific instructions to the LLM. We note here that the different LLM queries are made independently, rather than keeping the queries and answers from previous stages in context. Thus, for example, when the LLM is called carry out the step regeneration, it does not have access to the original generation. The same prompts are used across LLMs and datasets, thereby providing a general-purpose approach.
Target extraction To check step (for example, Step 5 in Figure 1), first need figure what the step is trying to achieve. Without a specific target, the regeneration stage would proceed in a random direction, making it impossible to serve as a reference to the original step. We thus use the LLM itself to extract the target of step using the question and all previous steps (Steps 0-4 in Figure 1) with the following prompt (we omit some line breaks due to space limitations):
The following is a part of the solution to the problem [Question]: [Step O..... Step ij. What specific action does the step [Step il take? Please give a brief answer using a single sentence and do not copy the steps.
During execution, we copy the question and steps into [Question] and [Step 0, ... Step i] to form the actual input to the LLM. The reason for requesting brief answer is to try and keep the amount of information retained to the minimum needed, thereby avoiding unnecessary influence on the regeneration and hopefully reducing correlations in errors in turn.
Information collection To reduce the difficulty of the regeneration stage and avoid unrelated information from affecting the result, filter out information that is not directly related to the
4

---

Published as a conference paper at ICLR 2024
eurrent step. Specifically, we ask the LLM to select useful items from the question and all previous items with the following prompt, where [Information] j] is simply the j-th sentence in the question:
This is a math question: [Question]. The following is information extracted from Information O: [Information 0] Information I: [Information 1] The following are the first a few steps in solution to the problem:
from the question:
Step 0: [Step 0] Step 1: [Step
Step i-1: [Step i-1]
Which previous steps or information does the next step [Step il directly follow from?
After retrieving the free-text response from the LLM, extract step information ids by regular expression. For example in Figure 1, the current step requires Step 4 and no information from the question context. The selected steps and information then fed into the regeneration stage.
Step regeneration Given the target and necessary information of the step, we now ask the LLM achieve the target independently with only the collected information, without seeing the original step. Because the step is usually a small jump from previous conclusions, and the information collection stage has already filtered out irrelevant information, we can usually trust regeneration results. The prompt for this stage is:
We are the process of solving math problem. We have some information from the problem: Information [Information Iol Information I: [Information It]
The following are some previous steps: Step 0: [Step Sol Step 1: [Step S.J
The target for the next step is: [Target]
Please try achieve the target with the information from the problem or previous steps.
Here [Target] is the output from the target extraction stage. [Information T,] and [Step S.] correspond to the specific items selected by the information collection stage. Figure only Step 4 and no information from the question is directly related to the current step, so SelfCheck simply copies the content of Step into [Step Syl and removes the block containing [Information I,].
Result comparison The last step is to compare results from the regeneration stage and the original step with the following prompt:
The following are 2 solutions to math problem. Solution 1: |Regeneration output) Solution 2: [Step i] Compare the key points from both solutions step by step and then check whether Solution I 'supports', 'contradicts' or 'is not directly related the conclusion Solution Pay special attention to the difference in numbers.
If the regeneration output 'supports' / /'contradiets' the original step, we can conclude that the original step is likely correct/incorret respectively. Sometimes, the correctness of the original step cannot be directly inferred from the regeneration output. For example, when the target is to simplify an equation, then there may be multiple valid solutions. In such cases, we are not sure about the correctness of the original step, which makes 'is not directly related to' the third possible outcome of the check.
3.2 RESULTS INTEGRATION
After running step-checking and getting a checking result for each step, we need an integration function á to give confidence score, w E [0, 1), for the overall correctness of the solution. The input of é should be vector in the form of [ro, T1, .... rrl., where each item r; represents the step checking result for Step i. We will use r; = -1, and to represent the step-checking results 'contradict', 'is not directly related to' and 'support" respectively. We find that the following simple integration function works well in practice
1 1,,-0
w <([o, (M(Voo,i,..... rn)) = 22 Sigmoid
1 r-=-1
An
(1)
==0
1=0
where A-1 and Ao are two non-negative hyperparameters with A_1 > Ao; we found nearly any sensible pairs of A_ and Åo worked similarly well in preliminary tests, so we simply fix A = and >o 0.3 throughout our experiments. The rationale of this setup is that the more failed checks we see, the more likely the overall reasoning process, and thus the final solution, are wrong. Note here that, because the checks are themselves imperfect, we do not necessarily want to immediately reject the whole solution from a single step-check failure, especially for r; 0 cases. This is why we take a 'soft' approach to the verification with a confidence score. The number of successful checks, Le. Σi= 1,,=1, is deliberately not included in integration function successful checks does not actually increase our confidence in the overall solution: shorter reasoning chains generally preferable to longer ones for given question and L.M.M
increased number of
5

---

Published as a conference paper at ICLR 2024
Once calculated, the resulting confidence score be directly used as weight for voting between different possible solutions. We can thus use SelfCheck to increase the accuracy of an LLM's 's answers by generating multiple possible solutions, calculating confidence scores for each, and then choosing our final answer through weighted voting.
4 EXPERIMENTS
We now run experiments on three wath-reasoning datasets to evaluate SelfCheck's effectiveness in checking multi-step reasoning and improving final answer accuracies. Note here that our focus on math-reasoning problems is due to ease of performance evaluation and dataset availability; SelfCheck is directly applicable to other arrestion-anwweriggg problems with nominal changes to our prompts. An additional experiment based on a logic reasoning task is provided in Appendix D.
Datasets GSM8K (Cobbe et al., 2021), MathQA (Amini et al., 2019), and MATH (Hendrycks et al., 2021) consist of math problems on primary school, middle school, and competition levels, containing 1319, 2985, and 5000 test samples, respectively. For GSM8K and MathQA, we evaluate SelfCheck on the whole test sets. Due limited resources, we use subset of MATH test set taken from Ling et al. (2023).' Besides the levels of difficulty, the three datasets differ from each other in the following aspects. Firstly, MathQA provides 5 options choose from for each problem. while GSM8K and MATH have no options. Secondly, GSM8K only has arithmetic problems, while MathQA and MATH contain more diverse problems in geometry, physics, probability, and algebra.
LLMs We use GPT-3.5 (gpt-3.5-0301) and GPT-4 (gpt-4-0613) as our LLMs, focusing in particular on the former due to budget restrictions. Note that the same prompts are used for all datasets with both LLMs during evaluation; no dataset-specific customization or tuning has been performed. When devising the prompts, small number of training samples from MathQA dataset were utilized. An additional experiment using Llama2 (70B, 4-bit, Touvron et al. (2023)) is provided in Appendix E.
Baselines We use majority voting (also known as Self-Consistency Decoding (Wang et al., 2022) in the context of CoT reasoning) as our main baseline following Ling et al. (2023) and Lightman et al. (2023). Despite its simplicity, this is still quite strong baseline in the current literature. In particular, most existing few-shot methods report similar results compared with it (Weng et al., 2022; Ling al., 2023). We also compare with quoted results from Self Verification (SV, Ling al. (2023)) and Deductive Verification (DV, Weng et al. (2022)) when possible. We note though that these approaches are not directly comparable SelfCheck in general, as they require additional exemplars which will often not be available in practice. Despite this, we will find that SelfCheck outperforms them when comparisons are possible.
We omit results from Faithful-CoT (Lyu et al., 2023), because it has already been shown to decrease the accuracies on GSMSK and MATH by 11 11.8% and 4.2%. respectively compared to majority voting (Ling et al., 2023). It is also impossible for us to compare with training/finetuning based methods such as Lightman et (2023), because we have neither access to their finetuned models nor computation resources to repeat their training/fieetnnngg The significant extra data and resources they require also means their contributions are somewhat tangential to SelfCheck regardless.
4.1 FINAL ANSWER CORRECTNESS
Figure 2 shows the performance gains using the confidence scores from SelfCheck to do weighted voting compared with baseline methods. The upper plots show that accuracies of both SelfCheck and majority voting have the same increasing tendency the number of generated solutions per question increases, which is a result of the variance reduction provided by averaging over more solutions. The bottom plots show the difference in accuracy between the two including the standard error in the estimate. We can see that by allocating higher weights to correct solutions, SelfCheck achieves significantly higher accuracies than majority voting for all solution numbers per question. We also find the improvements of SelfCheck (compared with majority voting) to be higher than Deductive Verification and Verificaiin in their reported settings, despite the use of in-context learning from additional examples. We will perform additional experiments on how performance changes when ensembling over larger number of solutions in Section 5.1.
tes / whw.. com/ con/1 logean /ver Fy. pot ren//ei in /reaull s/ gpt 3. 5/natural, progren/HHTH http://HAllLieeeeee. paor

---

Published as a conference paper at ICLR 2024
0.84
0.78
0.51
0.81
0.75
0.48
Accuracy
0.78
Accuracy
0.72
Ancuracy
0.45
0.69
0.42
0.72
0.66
0.39
Majority Voting SelfCheck
Majority Voting SelfCheck
Majority Voting
0.69
0.63
0.36
SeliCheck
0.66
0.60
0.33
AAccuracy
0.04
AAccuracy
0.04
AAccuracy
0.04
SV
0.02
0.02
0.02
0.00
0.00
0.00
10
8 10
#Solutions per question
# Solutions per question
Solutions per question
(a) GSM8K
(b) MathQA
(c) MATH"
Figure The upper plots show the accuracies of SelfCheck and majority voting for different numbers of generated solutions per question with GPT-3.5. The lower plots show the accuracy gaps between each method and majority voting, where DV and SV stand for Deductive Verification (Weng et al., 2022) and Velf-verifiiation (Ling al., 2023), respectively. It is difficuult to compare with DV and SV with respect to absolute accuracies because they are using different generator models. However, we that SelfCheck achieves higher relative performance gains than both in their reported settings.
Table 1: SelfCheck significantly increases final answer accuracies with both GPT-3.5 and GPT-
4, even we only have 2 candidate solutions for each question. AAce is the performance gain of SelfCheck compared with majority voting (MV), with the = indicating the standard error. Xx, and / //represen the proportions of questions with 0, or 2 correct solutions. Unlike in Figure 2, single generations. We see that the gains from SelfCheck typically larger in where it is common for only of the solutions to be correct, as these are the cases using weighted voting can influence the final answer.
Acc(MV) here is essentially the average accuracy
Dataset
Generator
Checker
XX (%)
X/ (%)
/ (%)
Acc(MV, %)
Acc(SeliCheck, %)
AAccc (%)
GPT-3.5
GPT-3.5
16.8
23.0
60.2
71.7
74.3
2.8:10.9
GSM8K
GPT-4
GPT-4
8.8
8.2
83.0
87.1
86.9
-0.24:022
GPI-4
GPT:3.5
8.2
83.0
1,0:10.3
GPT-3.5
GPT-3.5
27.6
26.4
46.0
59.2
64.6
5.4.1.11
MathQA
GPT-4
GPT-4
16.2
11.0
72.8
78.3
80.9
2.6::0.44
GPT-4
GPT-3.5
16.2
11.0
72.8
78.3
81.2
3.0::0.4
GPT-3.5
GPT:3.5
52.6
23.2
24.2
35.8
38.0
2.2:10.77
MATH
GPT-4
GPT4
42.0
20.2
37.8
47.9
51.3
3.4:10.6
GPT-4
GPT:3.5
42.0
20.2
37.8
47.9
48.9
1.0.0..8
To investigate the effect using more powerful LLMs, and of using a different LLM for generation and checking, we further conducted experiments with GPT-4 and a mix of GPT-4 and GPT-3.5. Because the high cost of calling the GPT-4 API, we randomly sample 500 questions from each dataset to form the test sets and generate (instead of 10) answers to each question. In Table 1, we see that SelfCheck significantly outperforms majority voting with both GPT-3.5 and GPT-4. We also notice that using GPT-3.5 to check GPT-4 generated answers yields surprisingly good results, actually outperforming checking with GPT-4 on the simpler GSM8K and MathQA tasks. This is likely because using different LLMs helps further decorrelate the errors of the generator and the checker, and shows that using a cheaper LLM can still often be sufficient for the checking. For the more difficult problems in MATH, using GPT-4 as checker always produces better results, but even here the checking from GPT-3.5 is beneficial compared to doing no checking at all.
4.2 VERIFICATION PERFORMANCE
Besides serving as a confidence score caleulator to improve the performance of voting, SelfCheck also predict the correctness of single solution. To do so, simply set threshold to the confidence score, where solutions with confidence scores w 2 I. are classified as correct.
7

---

Published as a conference paper at ICLR 2024
1.00
1.00
1.00
Proportion
0.80
Proportion
0.80
Proportion
0.80
0.60
0.60
0.40
0.40
0.40
0.20
Prad
0.20
Boal
0.20
Real in Frod
Rual - in Pood
Rosd 20 Pood
0.00
0.3
0.6
0.6
0.9
0.3
0.6
Threshold
Threshold
Threshold t
(a) GSM8K
(b) MathQA
(c) MATH
Figure 3: When raising the classification thresholds L, the proportions of real correct solu- tions in predicted correct solutions (Real in Pred +) increase for GSM8K MathQA (59.4%-882.29)) and MATH -50.8)....
1.0
Figure 4 shows the ROC curves for each dataset. As com- parison, directly prompting GPT-3.5 to verify whole reasoning chains leads to no meaningful control on the false and true pos- itive rates (FP and TP): they are always both 100% on MATH and 98% on GSM8K, as observed by Ling et al. (2023). In other words, the checker always predicts the answer as correct, providing no useful information.
0.8
TP rate
0.6
0.4
0.2
GSMBK
MathQM
As well as verification accuracies, we may also care about the solution quality after filtering out solutions with low confidence scores w. Figure 3 shows that by increasing the threshold t, SelfCheck can filter out more incorrect solutions, such that a higher proportion of the solutions that pass the check are indeed correct (Real + in Pred +). Though this is at the cost of misclassifying more of the real correct solutions as incorrect, this can useful feature in cases where the risk of choosing incorrect solution is higher than rejecting correct one.
0.0
MNTTI
FP rate
Figure 4: True positive rates (TP) vs. false positive rates (FP) as clas- sification threshold, t, is varied.
5 ANALYSIS
We now perform some ablations to justify some of the key design choices made by SelfCheck and provide insights on its behavior. Limited by budget and time, all experiments in this section are performed on subset of the MathQA test set with 100 randomly selected questions.
0.85
5.1 MORE SOLUTIONS PER QUESTION?
Accuracy
0.80
Serving as method to reduce variance, majority voting increased final answer accuracies different datasets when we increased from 2 to 10 solutions in Figure 2. In cases where only care about final predictive performance and majority voting can be applied (see Appendix D for example where it cannot, but SelfCheck can), one might thus question whether is better to simply use our computational resources to keep increasing the size of this ensemble, rather than relying on checking. Note here that the cost of running the verifier is around twice that of the original generation for zero-shot generators (it.
0.75
0.70
Majority
0.65
SetCCheck
10
20
30
40
50
#Solutions per question
Figure 5:
SelfCheck achieves significantly higher final answer accuracies than majority voting for large ensembles of solutions.
can cost. less than the original generation for few-shot generators with longer prompts).
However, as shown in Figure 5, the improving performance from using a larger ensemble saturates for majority voting, with the accuracy never going above that achieved when n = 9, thereby never reaching the performance we already achieved by SelfCheck for an ensemble of size n 5. Moreover, the performance of SelfCheck continues to increase as the ensemble grows. By lowering the weights (confidence) of incorrect solutions, SelfCheck increases the chance of selecting the correct answers, even when their generation probabilities the generator LLM are low. Therefore, with SelfCheck, LLMs effectively rectify their biased beliefs by themselves.
8

---

Published as a conference paper at ICLR 2024
5.2 ALBATION STUDIES
In order to pick apart the effect of several critical design choices for SelfCheck, we compare SelfCheck with some of its variants with respect final answer and verification accuracies MathQA.
Global v.s. step-by-step checking The first question is can we simply ask an L.M to check the whole solution without taking steps into consideration. To answer it, we prompt the LLM to perform global checking with the following instruction:
The following is a question and solution to it from a student. Carefully check whether the solution correct step by step. End your response with your conclusion that starts with "Correct", "Wrong" or "Not Sure".
Question: [Question] Solution: [Step 0, Step I,..., Step n]
Similar to the findings of Ling et al. (2023), we find that the global checker outputs "correct" most of the time and rarely recognizes an error. Consequently, its final answer accuracies are very close to majority voting (in
0.8
Accuracy
SeffChock
Figure 6) and its verification accuracy (55.0%) is only marginally above random guess (50.0%). This lack of ability to deal with the difficulty of global checking is what makes step checking necessary.
0.7
Global Check
Single Stage Check
Error Check (0 shot)
Ernur Check (1-chet)
Majonty Voking
0.6
1
2
3
4
5
8
9
10
Single-stage v.s. multiple-stage step checking Next, we ask whether we really need to decompose the step checking into several stages. To answer this, we design the following prompt to use the LLM directly.
#Solutions per question
Figure 6: Generation accuracies for variants of SelfCheck on MathQA with GPT-3.5.
The following is question and the first a few steps in its solution.
Question: [Question] Solution: [Step 0, Step l,.., Step i-1] Check the correctness of the next step: [Step i] Please consider the information it relies on and check step by step. Please end your response with your conclusion that starty with "Correct". "Wrong" or "Not Sure".
Figure 6 and Table 2 show that although this is better than global checking, it is still significantly worse than SelfCheck with its stule checking. This indicates that checking step in single stage is still too challeng- ing for the LLM, is necessary to further decompose step checking into pipeline of easier sub-tasks.
Table 2: Verification accuracies for vari- ants of SelfCheck on MathQA with GPT- 3.5. The reported verification accuracy is the average of true positive and true nega- tive rates.
Error check v.s. regenerate and compare We now justify the choice to perform step regeneration and com- parison instead of direct error checking for each step. To do so, replace our regeneration stage and com- parison stage with single error- cror-chhekiig stage. We first compare with a zero-shot version of the variant with the following prompt:
Method
Accuracy (%)
SelfCheck Global Check
66.7% %
55.0%
Single stage Check
57.2%
Error Check (O-shot)
63.1%
Error Check (I-shot)
64.2%
Given the following information:
Information 0: [Informaionn Tol Information 1: [Information T1]
Step O: [Step So] Step 1: [Step S.]
Check the correctness of the next step [Step i]
Please check for grounding errors, reasoning errors and calculation errors step by step. Please end your response with your conclusion that starts with "Correct", "Wrong" or "Not Sure".
We then add an examplar from Ling al. (2023) (see Appendix B) make more powerful one-shot error checker. However, results Figure 6 and Table show that even with a very detailed and instructive example, direct error checking still performs worse than regenerate and compare approach, which supports our previous argument that LLMs are better at generation than checking.
CONCLUSIONS
In this paper, we have introduced SelfCheck, general-purpose, zero-shot, step-by-step checking scheme for LLMs. Unlike previous approaches, SelfCheck does not require any additional data external resourees: it uses the LLM to identify emrors in its reasoning, leveraging novel regenerate-ane sriitte--n--com con-compare approach. By using the results of this checking to perform weighted voting over different solutions, find that SelfCheck is able to, in turn, increase final predictive accuracy.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Tom Rainforth is supported by the UK EPSRC grant EP/Y037200/1. Ning Miao and Yee Whye Teh are supported by ELISE European Network of Al Excellence Centres and the European Union's Horizon 2020 research and innovation programme under grant agreement No 951847 as well as by Tencent through the Tencent Collaboration on Large Scale Machine Learning. Ning Miao is also supported by China Scholarship Council University of Oxford Scholarship.
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi. Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-400 an open large language model with tfte-offtheean performance. 2023.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedzioske Yejin Choi, and Hannanch Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-bile based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.16653/11/11912444 URL https: https////aaaaaahhlocoooc
org/N1 1245.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 13:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv: 2303. /2712, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Google. Palm 2 technical report. arXiv preprint arXiv:2303.08774 2023.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T Kwok. Backward reasoning in large language models for verification. arXiv preprint 20870:2308.07758. 2023.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models hetter reasoners. arXiv preprint arXiv:2206.02336. 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of-thought reasoning. arXiv preprint arXiv:2306.03887 0332320000038220020 2023.
10

---

Published as a conference paper at ICLR 2024
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang. Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison- Dollison-Burch. Faithful chain-of thouuhht reasoning. arXiv preprint arX1::2301.1337, 2023.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Baolin Peng, Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang. Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 228.1133.2./81, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning language models. In The Eleventh International Conference on Learning Representations, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. Large language models are reasoners with self-verification. arXiv preprint 2xXiv:22/2.0956) 2022.

---

(Step 7) The area of the region inside the circle is 95 So the answer is 95 4 Published as a conference paper at ICLR 2024
APPENDIX A A COMPLETE EXAMPLE OF SELFCHECK
To help readers get a better idea of how SelfCheck works, we show a complete version of the example in Figure I, including the exact inputs and outputs of different stages. Please note that we have compiled all latex codes for readability, though the inputs and outputs to LLMs are in their original form. The question is
Let S be the set of complex numbers such that the real part of is equal to g. This set forms curve. Find the area of the region inside the curve.
We feed the following instruction the LLM that serves the generator:
Solve the following problem step by step. Please start each step with "Step :" and split sentences in each step with "vin". Please finish you response with "So the answer is ...". Problem: Let be the of complex numbers such that the real part of is equal to G. This set forms curve. Find the area of the region inside the curve.
After getting the following solution, start checking Step 5 as example:
Solution:
(Step 0) Let z = + yi be complex nunber, where and y are real numbers.
(Step We given that the real part of 2 is equal to 1.
(Step 2) Using the fact that the real part of complex number a + bi is a, we have
Re (+) Re
(Step 3) Setting Re (-)= G we get z²4y²
(Step 4) Cruss-mdtipyying and rearranging, we get the equation 6x = x² y².
(Step S) Completing the square, we obtain (a: )) + 5²
(Step This is the equation of circle with center 3 0) and radius
oz (Step 7) The area of the region inside the circle is T. So the answer
Dx 4
Variable definition extraction This optional stage extracts the definitions of variables that appear in the reasoning chain. SelfCheck feeds the definitions to the regeneration stage to avoid LLM not understanding the meaning of variable. The input the LLM is:
The following is the solution to math question: Question: Let S be the set of complex numbers such that the real part of 1 is equal to n. This set forms a curve. Find the area of the region inside the curve.
Solution:
(Step Let x + yi be a complex munber, where and y are real numbers.
Are there variables in the solution? If so, please list the definition of variable in the form of "I. variable is defined as..."
The underlined parts are the type of question, the question itself and the steps its solution, respectively.
The output from the LLM is:
Yes. There are variables in the solution.
1. is defined as complex number of form yi, where and y real numbers.
The bold part is then saved to form a part of the input in the regeneration stage.
Target extraction To get a brief and clear target of the current step, the input to the LLM is:
The following part of the solution the problem: Let S be the of complex numbers such that the real part of is equal to 1. This set forms curve. Find the area of the region inside the curve.
(Step Let + yi be complex number, where and y are real numbers.
(Step 5) Completing the square, we obtain (x 2)
What specific action does the step "Completing the square, obtain (z - 9)² + 2." take? Please give brief answer using a single sentence and do copy the steps.
12

---

Published as a conference paper at ICLR 2024
The underlined parts are the question and reasoning steps before the current one, including the current one.
The output of the LLM is:
The step completes the square to rewrite the equation in standard form of circle. The whole sentence saved and forms the most important part of the input in the regeneration stage.
Information Collection To get sentences in the question and previous steps in the solution that are directly related to the current step, the input to the LLM is:
Let be the set of complex numbers such that the real part is equal to 8. This set forms curve. Find the area of the region inside the curve.
This math question: Question:
The following information extracted from question:
Information Let S be the set of complex numbers z such that the real part of 1 is equal to nz
Information 1: This set forms curve. Information 2: Find the of the region inside the curve.
The following the first a few steps in solution to the problem: Step Let 2 + yi complex number, where and y are real numbers.
Step 4: Cross-multipyiing and rearranging,
the equation 6z x²² + 1/2
Which previous steps information does the next step "Completing the square, we obtain (x ( 2
1. directly follow from?
The underlined information (contents after Information X:) is simply list of sentences in the question. The underlined steps are reasoning steps before (not including) the current one in the solution.
The output from the LLM is:
The next step "Completing the square, obtain (x 2x- -" -
directly follows from Step 4.
Then we use regular expression to extract the information and step id from LLM's output. Some steps may only rely on information from the question while others may only use results from previous steps.
Step regeneration In this stage, we instruct the LLM to achieve the extracted target with only collected information with the following prompt:
are a process of solving a math problem.
Variables defined as: 1. is defined complex number of the form + yi, where x and y are real numbers.
The following are some previous steps: Step O: Cross-multiplinng and rearranging, we get the equation + y²
The target for next step is: The step completes the square to rewrite the equation standard form of circle.
Please try to achieve target with the information from previous steps.
The underlined parts are the type of question, variable definition (if any), collected information from the question (if any), collected previous steps (if any) and the extracted target. Please notice that the current step does not directly rely on any information from the question, removed the corresponding block from the input.
The output from the LLM is:
To rewrite the equation in standard form of circle, need complete the square for both and y. Starting from the given equation: 6z x² +
We add and subtract (6 (6/2)² on the right-hand side:
6x 9 x² + 9 Simplifying: 3.
3) 2 9. This is now in the standard form of a circle with center (3, 0) and radius
Then directly compare the regenerated step with the original one in the next stage.
13

---

Published as a conference paper at ICLR 2024
Result comparison The input to the LLM is:
The following are solutions to a math problem:
Solution 1: To rewrite the equation in standard form of a circle, we need to complete the square for both and y--
Starting from the given equation; 6o = x² 8².
9. Simplifying: (ar 3) vf This now in standard form of a circle with center (3.0) and radius
We add and subtract (6/2)² 9 on the right-hand side: 6z +
+
Solution 2: Completing the square, we obtain (x
3
Compare the key points from both solutions step by step and then check whether Solution "supports",
"contradicts" or "is not directly related to" the conclusion in Solution 2. Pay special attention difference in numbers.
The underlined parts simply the problem type, the regenerated step and the original step.
The output is:
In both solutions, the goal is to rewrite the given equation in standard form a circle. However, in Solution the completion of the square results in the center the circle being (3, 0) and radius 3. In Solution completion of the square results in center of the circle being (3/2,0) and radius of 3/2.
Therefore. Solution I contradicts the conclusion in Solution as the centers and radii of the circles are different.
the word 'support' or "contradict" appears in the last line of the output, SelfCheck outputs step score 1 or -1- Otherwise, SelfCheck outputs meaning that two steps are not related.
APPENDIX B EXAMPLAR FOR ONE-SHOT ERROR CHECKING
The following example from Ling et (2023) used to perform one-shot error checking.
Here is some information:
"I Alice gets 2 apples.
Alice gets twice as many oranges as bananas.
3. Original price per apple: $2
Discount: 50%
Price per apple after discount: $2 * (1 0.50) $2 0.50 $1"
Based on the given information. here reasoning process: "Calculate Alice's 's cost of the 5 apples after a 50% discount. Price per apple after discount: $1
Apples be eaten:
Oranges be eaten:
Total applex:
Total oranges: x
Total bananas: 2x
Alice's 's total number of fruits: 5 + 2x 5+ Alice's 's total cost of apples: (5 3) $2"
Double-check the reasoning process, let's analyze its correctness, and end with "yes" or "no".
Answer:
Let's think step by step.
Grounding check
Purpose: Find references for numbers in reasoning process sequentially
Rule: Any ungrounded number makes the check fails
"x" is variable, we do not care about variables. "y" is variable, we do not care about variables.
14

---

Published as a conference paper at ICLR 2024
"5" is quantity from the given information that represents the number of applex.
"50%" is quantity from the given information that represents discount.
"$1" is a quantity from the given information that represents the price per apple after the discount.
"3" is quantity not from the given information, it is ungrounded.
"6" is quantity not from the given information, it ungrounded.
"$2" is quantity from the given information that represents the original price per apple.
Result: Based on the check, there are ungrounded numbers, so grounding check fails.
Reasoning check:
Purpose: Check whether the reasoning logic correctly answers the question
Rule: Any logic error makes the check fails
To calculate the total cost of apples after 50% discount, should multiply the number of apples by the discounted price. But to answer the question, we do not need calculate the total number of fruit Alice gets. Result: Based on the check, there are some logic mistakes, so the reasoning check fails. Calculation check:
Purpose: Check whether the calculation process is consistent Rule: Any inconsistent calculation makes the check fails calculationI:
equation: $1 * 3)) answer: $2
(5-3)= =
$1 $ $2 is consistent with the answer, so the calculation is correct.
calculation2:
equation: x 2x, answer: 4x 2x 3x
+ 3x is inconsistent with the answer, so the calculation is incorrect.
Result: Based on the check, the calculation process is inconsistent, so the calculation check fails.
Check results: Ground check fails, Reasoning check fails, Calculation check fails. Rule: Any failed check makes the reasoning incorrect. So the answer "no".
APPENDIX C MORE EXAMPLES ON TARGET EXTRACTION
The following are some real examples of extracted targets on MathQA with GPT-3.5.
Step: According to the problem, the profit was divided the ratio of 2:3, which means x received 2/5 of the profit and received 3/5 of the profit. Extracted target: It calculates the rutio of the profit earned between x and y bused on the information given in the problem.
Since the interest has increased by 4%, the new of interest (r2) is 16.67% 4% 20.67%. Extracted target: The step calculates new rate of interest after increasing by 4%.
Step:
Step: The sum of the ages of the first 14 students is 56 + 160 216 years. Extracted target: The step calculates the sum of the ages of the first 14 students of the class.
Step: We can rearrange 8a 9b to a/b 9/8.
Extracted target: The step rearranges the given equation express in terms of and then obtains the ratio of a
Step: So, the equation becomes: (1+y/100)=50=
Extracted target: The step sets up the equation to calculate the percentage increase in Shyam's weight.
Step: Find the ratio of investment of each person in the business
Invesiment of A Investment of B Investment of C
=6500 1300 7800
=::
Extracted target: The step finds the ratio of investment of each person in the business.
15

---

Published as a conference paper at ICLR 2024
APPENDIX D AN EXPERIMENT ON LOGIC REASONING TASK
In this section, we verify whether SelfCheck can generalize to the task of logic reasoning using the dataset of SNR. Each test sample of SNR consists of a logic question, as well as its target answer. Different from other datasets shown in the paper, the target answers of SNR samples are latural-language sentences (such as "The hedgehog cold and scary."). instead of multiple-choie options (for MathQA) or single math expressions (for GSM8K and MATH). Because of the variability of natural languages, exact matching among generated answers becomes infeasbile. Majority voting thus cannot be used for this problem, but we can still use SelfCheck to pick out the best solution from all candidates.
To test performance, we randomly select 100 test samples from the SNR test set and generate 2 answers for each the questions. Because of the nature of natural language answers, it is not possible to perform automatic comparison between the generated answers and the target answers, we invited volunteers to perform human evaluation. To better deal with logic problems, we replaced all 'math' with 'logic' in the prompts and deleted the instruction 'pay special attention to the difference in numbers' in the result comparison stage. The prompts used were otherwise identical to those outlined in Section 3.
From the results shown in Table D.I, we can see that SelfCheck increases the accuracy on SNR by 3.5%, which a statistically significant improvement 5% level.
Table D.I: SelfCheck increases predictive accuracy on SNR by picking out correct answers from two independent generations. Base Acc means single solution accuracy here, since majority voting not feasible for SNR.
Dataset
Generator
Checker
Rase Acc (%)
SelrChock Acc (%)
Acc (%)
SNR
GPT-3.5
GPT-3.5
44.5
48.0
3.5-2.0
APPENDIX E AN EXPERIMENT ON LLAMA2
To test SelfCheck's generalization to other L.Mss, we also tested it with Llama2 (70B, 4-bit, Touvron et al. (2023)) on GSM8K (the generative accuracy of Llama2 on the other datasets was too low to perform any meaningful checking). Restricted by the speed of Llama2 (70B, 4-bit) on our server (which is only 20 tokens/s), we randomly select 500 test samples and generate 3 solutions for each question.
From Table E. we see statistically significant improvement in accuracy compared with majority voting using this different LLM; the gains here are actually larger than those observed for GPT-3.5.
Table E.1: SelfCheck significantly increases predictive accuracy on GSM8K with Llama2.
Dataset
Generntor
Checker
MV Acc (%)
SelfChock (%)
Acc (%)
GSMBK
Llama2
Llama2
40,3
43.2
2.9-0.3
ht hetpps /hwgggingg ng higfaee fecee.eeeee www.ee.o//eeeee cos/det ssets/1 git //artt we rewsoni ing.. netural I/b ol/ wrd wtwps//ttttttteeeeeoo//e/ss 1aan1
16


Reviews:
[{"review_id": "9HycegqbA9", "reviewer": "Reviewer_wFF7", "summary": "The paper introduces SelfCheck, a zero-shot verification framework designed to identify errors in Large Language Models (LLMs) when applied to mathematical reasoning tasks. Instead of the simplistic approach of self-querying LLMs, the paper presents a multi-stage strategy that deconstructs the problem into a sequence of simpler tasks. This method capitalizes on the LLM's generative capabilities and reduces the correlation between errors in the original generation and the subsequent verification process.\n\nThe approach involves separate interactions with the LLM, where it extracts the target and relevant context for each step, then generates an independent alternative step based on this information. Subsequently, it compares the original step with the regenerated one. If they match, the original step passes the verification check. The approach also employs confidence scores as weights to evaluate and prioritize more accurate solutions, offering a flexible alternative to traditional majority voting methods.\n\nNotably, this approach operates in a zero-shot manner, necessitating neither illustrations nor training.", "questions": "Questions are mentioned in the weakness section. Please refer to that.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Using the LLM to identify errors in its own step-by-step reasoning, analogously to how a human might go back to check their working is a very interesting research problem. Self checking and further refinement of its own answers is a challenging task for LLMs when reasoning is involved. The paper proposes a general-purpose, zero-shot, checking schema for self-identifying errors in LLM CoT reasoning. Some past methods have dealt with this problem but almost all of them use few shot setting. It is interesting to see the method work in zero shot setup.\n\nThe paper introduces an innovative step-checking method that leverages the generative modeling capabilities of modern LLMs to determine the correctness of step-by-step reasoning. To achieve this, the paper divides the checking process into four key components: target extraction, information gathering, step regeneration, and result comparison. Each step is carried out with carefully designed prompts, and the collective outcomes of these steps determine the overall accuracy of the answer.", "weaknesses": "- The paper presents a general purpose validation framework designed to assess the accuracy of multi-step mathematical reasoning tasks. Although this approach performs effectively well within a zero-shot prompting setup, it essentially relies on well-crafted prompts. It remains uncertain whether these four steps constitute the exclusive and comprehensive way of rigorously assessing the accuracy and the reasoning process in such tasks. There might be more ways to asses the quality of the reasoning chain and there is no gurantee that this is the only way or the best way. These four steps are not acquired through learning, generated by a model, nor rooted in learning science or based on any other theory. So it is very hard to judge it. It is plausible that their effectiveness is coincidental in the context of mathematical reasoning tasks and may not readily apply to other types of reasoning. Consequently, for broader applicability, the prompts would need to be rethought and rephrased, raising questions about the approach's generalizability.\n\n- Also, I dont see the usefulness of zero shot setting as if this comprehensive prompt for verification can be written, then 1 or 2 examples can also be provided. Utilizing a few-shot configuration could have yielded improved outcomes and enhanced the model's ability to adhere to the prompt, potentially resulting in better overall performance. Moreover, in a few-shot setup, the approach could have been benchmarked against previous methods. If the intention behind the zero-shot setup was to demonstrate the broad applicability of the approach, it would have been much better to include datasets that covered a wider range of domains, rather than solely focusing on mathematical reasoning.\n\n- The paper lacks a comparative analysis with other methods that have employed extensive prompts, such as PHP , faithful decomposition, verification based on subquestion decomposition, and similar approaches. So it is hard to judge if this prompting strategy is the best one for mathematical reasoning. \n\n- No comparison has been drawn to the cost of the prompting method as for each sample, 4 extra verification steps are needed which makes 5 API calls (at least) per sample. Compared to one step verification tasks that take 2 API calls, is the cost vs accuracy tradeoff worth it?\n\n- Table 1 is a bit confusing as 2 samples for majority voting does not make much sense. Should be at least 3 so that there are some agreements between the samples.\n\nPapers:\n[PHP]: https://arxiv.org/abs/2304.09797\n[Faithful decomposition]: https://www-files.anthropic.com/production/files/question-decomposition-improves-the-faithfulness-of-model-generated-reasoning.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bTjNWdswMA", "reviewer": "Reviewer_PScZ", "summary": "This paper proposes a zero-shot verification schema named SelfCheck to recognize errors in the step-by-step reasoning of large language models (LLMs) in answering complex questions. The proposed SelfCheck implements the self-checking by following a regenerate-and-compare procedure and then conducts weighted voting on multiple solutions, thereby improving the performance. Experimental results on three maths datasets including GSM8K, MathQA and MATH demonstrate the effectiveness of SelfCheck, which alleviates the need for additional data or external resources.", "questions": "1. In Paragraph 2, Section 3.1, the authors state that ‘checking’ is a less common task in the training corpus of most LLMs. Can the authors provide some evidence of citations of this statement?\n\n2. In Paragraph 5, Section 3.1, the authors claim that LLM is usually able to perform the tasks of figuring out the target of the current step. Missing evidence or supplementary experiments should be added.\n\n3. In Section 3.1, the authors mention Information in both stages of Information collection and Step regeneration but the Information should refer to different contents according to the authors’ descriptions. A minor suggestion would be to modify the terms in the prompts in order to better distinguish them.\n\n4. The comparison between the proposed SelfCheck and DV/SV in Figure 2 is quite confusing. Could the authors give relevant explanations on how they compare these methods with varying scores and different generator models?\n\n5. Why did not the authors conduct experiments with GPT-3.5 as the Generator and GPT-4 as the Checker (in Table 1)? It would be helpful to provide appropriate descriptions.\n\n6. Minor comments on writing:\n\n(1)\tParagraph 2 in Section 1: ..., which is far below human level -> , which  is far below the human level\n\n(2)\tParagraph 5 in Section 1 & Paragraph 1 in Section 3: to check their working -> check their work\n\n(3)\tParagraph 1 in Section 3.2: , and thus final solution -> , and thus the final solution\n\n[1] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou. LARGE LANGUAGE MODELS CANNOT SELF-CORRECT REASONING YET", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper explores the self-checking capabilities of LLMs by elaborating a regenerate-and-compare schema to assess their reasoning and recognize errors, which potentially has broader insights beyond a wider range of tasks.\n\n2. The authors introduce an intriguing and reasonable ensemble method for multiple solutions, which employs an integration function and calculates confidence scores in the final voting stage.\n\n3. The authors provide a comprehensive and detailed analysis to validate the effectiveness of each stage of the proposed SelfCheck.", "weaknesses": "1. The authors claim that they provide a general-purpose, zero-shot approach to checking. However, the proposed schema is more tailored to maths problems to some extent as answering mathematical questions requires explicit computational steps and numerical formulas, which can be easily checked by LLMs. Hence, it is a little confusing to state the proposed SelfCheck is general-purpose. I wonder if the authors could conduct experiments on other types of reasoning tasks such as logical reasoning or commonsense reasoning.\n\n2. Confusion about the methodology design.\n\n- The authors check each step by evaluating the conditional correctness of each step based on the provided question and previous steps in the solution. I doubt whether there could be a phenomenon where step i is wrong, but the following steps are perfectly correct, i.e. the model is anot affected by the step i error. But if the authors evaluate using conditional errors, it’s quite possible that the model would determine all subsequent steps (after step i) are wrong. Could the authors provide more analysis or explanations?\n\n- The design of the integration function seems helpful as shown in the experimental results. Could the authors explain the insights of this design and the choices of hyperparameters?\n\n3. A lack of clarification regarding the experimental results.\n\n- The authors claim that SelfCheck significantly outperforms majority voting with both GPT-3.5 and GPT-4. However, results in Table 1 (Row 2) show that SelfCheck exhibits inferior performance in GSM8K with GPT-4 as the generator and checker.\n\n- Results in Figure 2 demonstrate that the SelfCheck did not exceed majority voting by much, but the inference cost of SelfCheck should be larger in comparison. Could the authors illustrate in detail the superiority of SelfCheck over majority voting?\n\n4. The writing can be improved. There are some typos and unclear descriptions. Please refer to the questions below for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I184Buv2Mq", "reviewer": "Reviewer_NsFJ", "summary": "The paper focused on the task of refining CoTs when applying LLMs to reasoning. The main contribution is to propose a self-checking solution on each step in the reasoning chain of CoTs one by one. The proposed operations include target extraction, information collection, step regeneration, and results comparison. The main tech employs some specifically designed prompts for powerful LLMs, such as GPT 3.5 and GPT-4. The experimental comparisons on three datasets (GSM8K, MathQA and MATH) show that the proposed method could outperform existing majority voting methods, self-verification, and deductive verification.", "questions": "1) Why the authors do claim the solution is self-checking? In Table 1, the authors also employ different LLMs on the Generator and Checker. \n\n2) Are the proposed checking components and prompts in each step useful for other LLMs?\n\n3) In equation (1), why lamda_-1 is setting to 1 and lamda_0 to 0.3? How to determine such parameters？\n\n4) Why there is no dot for DV and SV in Figure 2 (b)\n\n5) What is the meaning of the yellow curve in Figure 3?\n\n6) I think that the discussion in subsection 5.2 is of no use. The performance heavily relies on the prompts. So I think if we adopt an optimal prompt in the global checking setting, better results may be obtained.\n\n7) Why were the experiments conducted on the math-related tasks? I wonder where the performance goes when other types of reasoning are explored, such as logical reasoning, deductive reasoning, and temporal reasoning.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) Correcting each reasoning step through self-checking is interesting.\n2) Employing LLMs to perform self-checking based on designed prompts may be useful for some applications.\n3) The experimental results on three open-accessed datasets show the effectiveness of the proposed solutions.", "weaknesses": "1) I think discussing how to prompt some \"incantations\" to the LLMs (especially for such black-box LLMs) has no insight into this community. The proposed prompts for each component in the self-checking process are well-designed, especially for GPT-serial LLMs. In my opinion, they could not be well extended to other LLMs, such as LLama or Claude.\n\n2) Step-by-step checking may have the problem of error propagation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LoTJAaqgW7", "reviewer": "Reviewer_6G8r", "summary": "This paper proposes a Self-Check, a method that allows LLMs to check their reasoning. The method involves four steps (target extraction, information collection, step regeneration, and result comparison.), each involving querying LLM for a different step. The basic idea involves probing the model to understand what a substep is doing (Steps 1 and 2), and then solving a substep again with this increased understanding (Step 3). If the generated answer does not match the conclusion of the subset, it indicates a contradiction in the output at the step. The contradictions are accumulated to generate a single score for the correctness of the solution. During inference, multiple solutions are generated, and SelfCheck scores are used to weigh and pick the best solution. This score is used to weigh solutions. The authors also analyze whether the SelfCorrect predictions correspond to the actual correctness and find that it is generally the case.", "questions": "Q1. If my observations in Weakness A are accurate, could Figure 5 simply result from comparing incompatible data points? Should we compare SelfCheck @ k with SelfConsistency @ m with m ~ 10 x k?\n\nQ2: What are the false positive rates at various thresholds?\n\nQ3: When devising the prompts, a small number of training samples from MathQA dataset were utilized. How were these examples used? \n\n\n---\n\n### Comments\n\n- _Therefore, with SelfCheck, LLMs can effectively rectify their own biased beliefs by themselves._ While this insight is crucial, it might not be immediately apparent to a casual reader. In my opinion, this is an important takeaway from the paper. A more extensive analysis of this topic would be highly beneficial.\n\n- Missing citation: \n\nJiang, Weisen, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T. Kwok. \"Backward reasoning in large language models for verification.\" arXiv preprint arXiv:2308.07758 (2023).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**A:** The method is straightforward yet proves to be effective. Although there are questions regarding its efficiency and effectiveness in comparison to majority voting (refer to weaknesses), the analysis confirms that SelfVerification is indeed taking place. \n\n**B:** The paper demonstrates that LLMs can self-correct for reasoning problems in certain cases, providing a valuable answer to a significant question currently of interest to the community.", "weaknesses": "**A:** SelfCheck requires 3 LLM queries per substep. Consequently, the comparison between SelfCheck and SelfConsistency appears skewed, as SelfCheck inherently utilizes approximately 10x more samples (assuming an average of 3 substeps). Furthermore, the results with GPT-4 are based on a limited sample size, and given the analysis with Self-Consistency, the findings should be interpreted cautiously. \n\n\n**B:** While the method is presented as 0-shot, it requires training samples (as seen in section 3.2) to learn hyperparameters, and thus, the criticism of approaches that require examples is unfair (We note though that these approaches are not directly comparable to SelfCheck in general, as they require additional exemplars which will often not be available in practice.). Additionally, the authors had to meticulously craft instructions for each step, which somewhat undermines their critique of other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)