Title: Self-Evaluation Guided Beam Search for Reasoning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Models

Multistep Reasoning

Stochastic Beam Search

LLM Self-Evaluation
Abstract: Breaking down a problem into intermediate steps has demonstrated impressive performance in Large Language Model (LLM) reasoning. However, the growth of the reasoning chain introduces uncertainty and error accumulation, making it challenging to elicit accurate final results. To tackle this challenge of uncertainty in multi-step reasoning, we introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of LLMs. We propose a decoding algorithm integrating the self-evaluation guidance via stochastic beam search. The self-evaluation guidance serves as a better-calibrated automatic criterion, facilitating an efficient search in the reasoning space and resulting in superior prediction quality. Stochastic beam search balances exploitation and exploration of the search space with temperature-controlled randomness. Our approach surpasses the corresponding Codex-backboned baselines in few-shot accuracy by $6.34$%, $9.56$%, and $5.46$% on the GSM8K, AQuA, and StrategyQA benchmarks, respectively. Experiment results with Llama-2 on arithmetic reasoning demonstrate the efficiency of our method in outperforming the baseline methods with comparable computational budgets. Further analysis in multi-step reasoning finds our self-evaluation guidance pinpoints logic failures and leads to higher consistency and robustness. Our code is publicly available at [https://guideddecoding.github.io/](https://guideddecoding.github.io/).

Self-Evaluation Guided Beam Search for Reasoning

Abstract

Breaking down a problem into intermediate steps has demonstrated impressive performance in Large Language Model (LLM) reasoning. However, the growth of the reasoning chain introduces uncertainty and error accumulation, making it challenging to elicit accurate final results. To tackle this challenge of uncertainty in multi-step reasoning, we introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of LLMs. We propose a decoding algorithm integrating the self-evaluation guidance via stochastic beam search. The selfevaluation guidance serves as a better-calibrated automatic criterion, facilitating an efficient search in the reasoning space and resulting in superior prediction quality. Stochastic beam search balances exploitation and exploration of the search space with temperature-controlled randomness. Our approach surpasses the corresponding Codex-backboned baselines in few-shot accuracy by 6 . 34% , 9 . 56% , and 5 . 46% on the GSM8K, AQuA, and StrategyQA benchmarks, respectively. Experiment results with Llama-2 on arithmetic reasoning demonstrate the efficiency of our method in outperforming the baseline methods with comparable computational budgets. Further analysis in multi-step reasoning finds our self-evaluation guidance pinpoints logic failures and leads to higher consistency and robustness. Our code is publicly available at https://guideddecoding.github.io/.

1 Introduction

The remarkable empirical achievements of Large Language Models (LLMs) have recently ushered in a new era in machine reasoning through few-shot prompting techniques (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a; OpenAI, 2023). In particular, breaking down a problem into intermediate stages, or a reasoning chain, can significantly improve model performance on reasoning tasks (Cobbe et al., 2021). Various prompting approaches have been proposed to define these chains, such as scratchpads (Nye et al., 2021), chain-of-thought (CoT) (Wei et al., 2022b), least-to-most (Zhou et al., 2023), and program-aided language models (PAL) (Gao et al., 2023; Chen et al., 2022). However, as the complexity and length of reasoning chains increase with the difficulty of tasks, LLMs struggle with errors and imperfections that accumulate across multiple intermediate steps (Wu et al., 2016; Guo et al., 2018; Chen et al., 2022). Furthermore, the growing number of steps leads to an exponential growth in the search space for reasoning, making it exceedingly difficult to obtain accurate final outcomes.

Confronted with the challenges of uncertainty in multi-step chaining, several previous studies have worked on different aspects to alleviate the impact of reasoning errors. For instance, Wang et al. (2023) introduce self-consistency as a method to determine the final answer through majority voting using multiple sampled reasoning paths, while Li et al. (2022) investigate various prompts to diversify the sampling outcomes. Gao et al. (2023) and Chen et al. (2022) utilize Python programs to

Multi-step Reasoning Chain

total_sold = 88000

marilyn_harald_ratio = 10

answer = total_sold / (marilyn_harald_ratio + 1)

Multi-step Reasoning Chain

marilyn_copies = 88000

ratio = 10

answer = marilyn_copies // ratio

Question

1

st

record sold

10

times as many

copies as Harald's. If

they sold

88,000

copies

combined, how many

copies did Harald sell?

_total_sold = 88000_

_marilyn_copies = 10_

_marilyn_copies = 88000_

_ratio = 10_

_marilyn_harald_ratio = 10_

_copies_mar = total_sold / 10_

_…_

_…_

_answer = total_sold / (marilyn_harald_ratio + 1)_

_…_

Answer

:

8,800

Answer

:

8,000

❌

✔

Self-Evaluation Guided Stochastic Beam Search

Basic Greedy Decoding

Generation

Evaluation

🤖

LLM

achieve higher accuracy in mathematical computations. While these approaches have contributed to significant performance improvements in reasoning, the process of generating reasoning chains has been parameterized as a standard autoregressive process and intrinsically faces the challenge of sampling within an exponentially large search space.

Motivated by this challenge, we employ LLM self-evaluation (Kadavath et al., 2022) as a bettercalibrated criterion to automatically guide the search in the reasoning space, drawing inspiration from prior works on utilizing LLMs for self-evaluation (Rae et al., 2021; Paul et al., 2023; Madaan et al., 2023; Shinn et al., 2023). We integrate the self-evaluation guidance for reasoning in a stepwise and generalizable manner. Specifically, we formulate the reasoning chain generation as a decoding process consisting of multiple intermediate steps. Unlike traditional text decoding where each step produces a single token, we consider each decoding step as a reasoning logic composed of a sequence of tokens. This framework enables us to employ beam search (Jurafsky and Martin, 2009; Graves, 2012) decoding tailored for intermediate steps and guide the beam searching process by controlling the error of each reasoning step to prevent potential error accumulation throughout the chaining. Figure 1 illustrates an example of decoding a chain of program-aided reasoning steps. Furthermore, we incorporate temperature-controlled randomness (Ackley et al., 1985; Kool et al., 2019; Meister et al., 2021) into the traditional (deterministic) beam search to balance the quality-diversity trade-off in searching for better reasoning chains. Our approach has resulted in respectable improvements across various arithmetic, symbolic, and commonsense reasoning tasks. For instance, by guiding the reasoning decoding process of the Codex model (Chen et al., 2021), we achieve accuracies of 85 . 5% , 64 . 2% , and 77 . 2% on the GSM8K, AQuA, and StrategyQA benchmarks, compared to the vanilla reasoning-enhanced Codex performance of 80 . 4% , 58 . 6% , and 73 . 2% , respectively. Our further analysis on Llama-2 (Touvron et al., 2023b) demonstrates the efficiency of our method in surpassing the self-consistency baseline under equivalent computational budgets.

2 Self-Evaluation Guided Stochastic Beam Search

Considering the input prompt and question Q represented as x , we formulate the answer distribution P ( a | x ) by decomposing it as a reasoning chain generation process P ( R | x ) and an answer generation process P ( a | R,x ) :

where R is the intermediate reasoning chain variable that is typically a text sequence. P ( a | R,x ) = ✶ A ( a ) max( | A | , 1) , where A = execute ( R ) represents the set of predicted answer(s) interpreted from R , and ✶ A is the indicator function of the subset A . In practice, | A | ≥ 0 can be 0 or larger than 1 when the reasoning R returns no valid answer or produces more than one possible answers, respectively.

s

1

s

2

…

s

T

R

= [

s

1

,

s

2

, …,

s

T

] =

s

1:

T

Q

…

…

…

…

…

…

…

Timestep

1

Timestep

2

Timestep

T

s

1

1

s

1

2

s

1

n

s

1

n

-1

s

2

1

s

2

n

s

2

n

+1

s

2

n

+2

s

2

2

n

s

T

1

s

T

n

s

T

n

+1

s

T

2

n

s

T

2

Reasoning Chain

Predicted Final Answer

a

Multi-step Reasoning

Q

Q

s

1

…

s

t

-1

s

1

…

s

t

-1

Kept Candidate

2

s

t

2

n

-1

s

t

1

…

s

t

n

-1

s

t

n

+1

…

s

t

2

n

s

t

n

Kept Candidate

1

Self-Eval

Stochastic

Beam Search

(

k = 2

)

P

LM

(

s

1:

t

1

)

C

(

s

1:

t

1

)

…

…

P

LM

(

s

1:

t

n

-1

)

C

(

s

1:

t

n

-1

)

P

LM

(

s

1:

t

n

)

C

(

s

1:

t

n

)

P

LM

(

s

1:

t

n

+1

)

C

(

s

1:

t

n

+1

)

…

…

P

LM

(

s

1:

t

2

n

-1

)

C

(

s

1:

t

2

n

-1

)

P

LM

(

s

1:

t

2

n

)

C

(

s

1:

t

2

n

)

Decoding at Timestep

t

Self-Eval

…

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is:

(B), because Marilyn sold 10 times as many …

Prompting Framework

🤖

LLM

Q

R

s

t

Select

k

Paths

Prior research has modeled the reasoning chain generation P ( R | x ) by prompting LLMs to explicitly elaborate on the required intermediate steps R . Through setting different prompting schemes, the reasoning process P ( R | x ) can be modeled as chain-of-thought free-text reasoning (Kojima et al., 2022; Wei et al., 2022b), a two-stage question decomposition and answering pipeline (Zhou et al., 2023), or program-aided reasoning to generate a python program (Gao et al., 2023; Chen et al., 2022). While effective, previous work mostly uses a single sample of R from the LLMs to approximate the expectation in Eq. 1 - the generated reasoning chain is often unreliable and causes incorrect answers. To mitigate this issue, Wang et al. (2023) conduct majority voting to approximate the expectation via sampling and aggregating multiple reasoning chains. Li et al. (2022) take a further step to diversify the sampling and calibrate P ( R | x ) with a task-specific fine-tuned verifier. Another line of work focuses on improving P ( a | R,x ) instead. For example, Gao et al. (2023) and Chen et al. (2022) employ Python programs for more accurate calculations in math word problems.

In this work, we focus on improving P ( R | x ) to enhance the consistency of the sampled reasoning chains. To this end, we propose to explicitly break down the reasoning process into multiple steps, as shown in Figure 2, where each step yields a semantically integrated sequence of tokens, representing a single step within the overall reasoning chain. From this perspective, we can approach the task of enhancing P ( R | x ) as a decoding problem over the reasoning chains. Considering the exponentially large search space and the potential unreliability of LLM-produced chains in reasoning, we propose a constrained stochastic beam search decoding approach to improve the reasoning step by step and obtain high-quality reasoning with a limited number of samples. We detail our approach next.

2.1 Multi-step Reasoning via Stochastic Beam Search

In multi-step reasoning, a reasoning chain of T steps is sequentially generated through several timesteps as R = [ s 1 , s 2 , · · · , s T ] = s 1: T , where s t represents a sequence of tokens as the t -th step. Formally, the reasoning generation process P ( R | x ) can be factorized in an autoregressive manner:

which resembles the typical token-level autoregressive distribution of language models. Stepwise reasoning allows us to formulate the process as a step-by-step decoding problem, where we can utilize widely-used strategies such as beam search for the generation. Different from the typical text

decoding process where each step consists of a single token, here we view a sequence of reasoning tokens as a single step. One of the most severe issues in LLM-based reasoning is the potential unreliability and inaccuracy of each reasoning step generated by the model. Furthermore, errors from individual steps may accumulate throughout the reasoning chain, exacerbating the problem. To address the issue, we define a constraint function C ( s t , s 1: t -1 ) ∈ [0 , 1] within each reasoning step 3 that outputs the LLM confidence in the correctness of the reasoning sequence s t based on the previous context s 1: t -1 . Then, we present a constrained decoding approach that combines the language model probability and the correctness confidence as a new decoding objective function E ( s 1: T ) :

where P LM G is the language model distribution 4 . λ ∈ [0 , 1] is a weight hyperparameter to balance the LM score and the confidence score. We will detail the design of C ( s t ) in Section 2.2. Eq 3 follows an autoregressive factorization form, and thus traditional token-level decoding methods such as beam search can be applied here on the chain level. As it is desirable to obtain high-quality reasoning chains with limited samples that are scored high by E ( s 1: T ) , it is natural to utilize greedy or beam search decoding to approximate the reasoning sequences that maximize E ( s 1: T ) .

Additionally, multiple diverse reasoning chains could be aggregated to further improve the final accuracy, as suggested by Eq 1 and empirically confirmed by self-consistency reasoning (Wang et al., 2023). To this end, we propose a variant of stochastic beam search (Kool et al., 2019; Meister et al., 2021) to strike a tradeoff between exploration and exploitation. Concretely, for beam size k , at each reasoning step we draw n samples of s t following P LM G ( s t | x, s 1: t -1 ) for each beam, and we end up with nk chain hypotheses of s 1: t to form the candidate set S , then we perform beam pruning through sampling - we sample k reasoning beams without replacement, rather than finding the arg max k , following a distribution defined by the accumulated score: 5

where the temperature τ is a hyperparameter to control the randomness in stochastic beam search; when τ → 0 , stochastic beam search becomes the vanilla beam search algorithm. The reasoning beams s 1: t can be sampled efficiently since |S| = nk is a finite set. To enable fine-grained control of sampling randomness in decoding, we also introduce a hyperparameter α ∈ [0 , 1] so that τ can decay step by step as τ → ατ . By annealing τ with α , we can mitigate the error accumulation due to aggregated randomness throughout chaining, as discussed in Section 3.4.

By incorporating controllable randomness, we not only achieve a more reliable single reasoning chain generation by setting randomness to be small, but also leverage multiple diverse reasoning chains with larger variance. Next, we introduce our constraint function C ( s t , s 1: t -1 ) that utilizes a self-evaluation scheme to improve the consistency of each reasoning step.

2.2 Self-Evaluation as Correctness Control

Inspired by the recent success of self-evaluation (Kadavath et al., 2022; Shinn et al., 2023; Madaan et al., 2023; Paul et al., 2023), a scheme to prompt LLMs to evaluate their own generation, we use LLMs to judge the correctness of s t based on s 1: t -1 . Specifically, the evaluation and generation models use the same backend LLM with different prompts, which consist of few-shot exemplars. We follow previous works of CoT (Wei et al., 2022b) or PAL (Gao et al., 2023) to formulate the generation prompts. To construct the in-context exemplars prompt C for the self-evaluation LLM LM C , we provide stepwise evaluation examples (as question answering with rationales) in each instance. Inspired by Kadavath et al. (2022), we design prompt C in the form of multiple-choice questioning (as shown in Figure 2) to better calibrate the model predictions, where we adopt the token-level probability of option A to represent the correctness score as:

3 Experiments

3.1 Setup

We present and analyze the results of our self-evaluation guided beam search with different LLM backbones on various reasoning benchmarks. Implementation details including prompt examples and hyperparameter setup can be found in Appendix A.3.

Benchmarks. We evaluate the effectiveness of our approach across three types of reasoning tasks: (1) Arithmetic Reasoning on five math word problem benchmarks, including GSM8K (Cobbe et al., 2021) on math word problems, AQuA (Ling et al., 2017) on algebraic word problems, SVAMP (Patel et al., 2021) on structure variations of math word problems, ASDiv (Miao et al., 2020) on diverse math word problems, and TabMWP (Lu et al., 2023) on tabular math word problems; (2) Symbolic Reasoning on BIG-Bench (Srivastava et al., 2022), which involves Date Understanding of contextbased date inferring and Object Counting of enumerating and counting objects of different types; (3) Commonsense Reasoning on three benchmarks, including CommonsenseQA (Talmor et al., 2019) of commonsense questions that require prior world knowledge to answer, StrategyQA (Geva et al., 2021) of questions that require a multi-hop strategy to answer, and Sports Understanding from BIG-Bench (Srivastava et al., 2022) to determine whether a sports-related sentence is plausible.

Baselines. We consider two types of baselines: (1) Chain-of-Thought (CoT) (Wei et al., 2022b) prompting in free-text reasoning and (2) Program-Aided Language models (PAL) (Ling et al., 2017) and Program-of-Thought (PoT) (Chen et al., 2022) prompting in program-aided reasoning. We also include their self-consistency (Wang et al., 2023) variants for multiple-chain reasoning. For generation, we follow the few-shot exemplars of baselines. For self-evaluation, we manually create a set of few-shot exemplars based on the baseline outputs on corresponding training data. We formulate self-evaluation as a task of multiple-choice question answering, following Kadavath et al. (2022). For baselines, we represent the cost as the number of generated tokens. For the cost of our method, we also include the number of additional input tokens in self-evaluation for a fair comparison.

Backboned LLMs. We assess our approach on closed- and open-source LLMs using both PAL and CoT prompting. For closed-source LLMs, we choose Codex ( code-davinci-002 ) (Chen et al., 2021) to report and compare the results on all datasets. We use Llama-2 ( 13B ) (Touvron et al., 2023b) as our open-source LLM to conduct cost-performance analysis on different datasets.

3.2 Main Results

Arithmetic and Symbolic Reasoning. Table 1 shows the results for arithmetic and symbolic reasoning. Our method achieves significant performance improvements on most benchmarks in both single- ( τ = 0 ) and multiple-chain scenarios, with PAL as the baseline. For arithmetic reasoning, we observe absolute increases in accuracy of 5 . 3% , 8 . 3% , and 0 . 7% on GSM8K, AQuA, and SVAMP, respectively. One possible explanation for this discrepancy in improvements is the reduced diversity in LLM generations due to higher confidence in predictions, as evidenced by the relatively high performance on the tasks. This highlights the importance of incorporating controllable randomness into the candidate generations to expand the search space for self-evaluation guided decoding. We further explore the impact of generation diversity by varying the temperature γ in Section 3.4.

For symbolic reasoning, our approach also leads to consistent performance gains. However, when the baseline itself performs well on the task ( e.g. , 96 . 7% on Object Counting ), our approach may not yield substantial improvement. This can also be attributed to the constrained accessible search space for self-evaluation guidance to refine the generation distribution. This limit suggests the inherent deficiency in our LLM-based prompting method that it becomes increasingly challenging to calibrate the generation direction when the model LM G is more confident in its predictions. In other words, the high baseline performance usually indicates lower diversity in the LLM generations even with a large temperature γ , resulting in a limited accessible search space for the model to find a better solution.

Commonsense Reasoning. Table 2 compares methods using CoT prompting in commonsense reasoning. Our approach shows consistent performance improvements across several tasks. For example, on StrategyQA, we achieve an accuracy of 77 . 2% compared with 73 . 2% of the baseline.

Likewise, the performance of our approach is constrained by the low diversity of LLM generations on Sporting Understanding , as we observe on Object Counting in symbolic reasoning.

Computational Cost Overhead. Despite the fact that our approach achieves significant improvement on various benchmarks, we observe an overhead of computational cost compared with the corresponding baselines. For example, the single-chain version of our approach using PAL costs about 3 times more than the self-consistency baseline on GSM8K. As detailed in Appendix A.3, this is due to a relatively large hyperparameter - the number of rollouts per beam n - which we set as 16 for better performance. To strike a balance between performance and cost and present a complete picture, we adopt n = 2 and conduct cost-performance analysis on our approach in Section 3.3.

3.3 Cost Analysis

Table 3 compares the baseline and our approach under comparable computational budgets (measured in # Tokens). Our method consistently outperforms self-consistency on the arithmetic reasoning tasks even when benchmarked for relatively less computational cost. For example, we achieve 46 . 1% on GSM8K with a cost of 12 . 6 k tokens, compared with the accuracy of 41 . 8% of self-consistency which costs 13 . 9 k tokens. Figure 4 further illustrates the cost-efficiency of our approach on GSM8K using different prompting methods under various levels of costs. Our approach significantly outperforms the corresponding equal-cost baseline especially when the computational budget increases, indicating the improvement in the performance upper bound brought by our method.

However, our approach lags behind the CoT baseline on commonsense reasoning. This implies the limitation of our method when applied to shorter reasoning chains, i.e. , decreasing the number

2000

4000

6000

8000

10000

12000

14000

34

36

38

40

42

44

46

35.0

38.9

40.4

41.7

41.8

34.6

35.7

36.1

40.5

42.9

45.2

46.1

PAL-SC

Ours-PAL (single)

Ours-PAL (multiple)

2000

4000

6000

8000

10000

12000

34

36

38

40

42

44

37.1

40.8

42.9

43.0

43.2

34.5

36.5

38.8

40.6

43.3

44.9

45.4

CoT-SC

Ours-CoT (single)

Ours-CoT (multiple)

(a) PAL Prompting Methods on GSM8K

(b) CoT Prompting Methods on GSM8K

0.0

0.2

0.4

0.6

0.8

1.0

generation probability

0

500

1000

1500

0.88

0.83

overall

correct

wrong

0.0

0.2

0.4

0.6

0.8

1.0

correctness confidence

0.84

0.71

overall

correct

wrong

0.0

0.2

0.4

0.6

0.8

1.0

score

=

(1

)

,

=0.5

0.86

0.76

overall

correct

wrong

(a) Score distribution of PAL baseline predictions on GSM8K.

0.0

0.2

0.4

0.6

0.8

1.0

generation probability

0

1000

2000

0.63 0.65

overall

correct

wrong

0.0

0.2

0.4

0.6

0.8

1.0

correctness confidence

0.75

0.70

overall

correct

wrong

0.0

0.2

0.4

0.6

0.8

1.0

score

=

(1

)

,

=0.5

0.66 0.69

overall

correct

wrong

(b) Score distribution of CoT baseline predictions on StrategyQA.

of intermediate steps weakens the effect of stepwise self-evaluation in beam search in reducing error accumulation. On the other hand, self-consistency can directly improve performance through instance-level aggregation without additional cost for self-evaluation. We analyze how our method benefits longer reasoning chains on different tasks in Section 3.4.

3.4 Further Analysis

We now provide a detailed analysis of why our method achieves significant gains.

Generation and Self-evaluation Calibration. We investigate the distributions of generation confidence ( i.e. , the LM probability P ) and correctness confidence C in our self-evaluation score E . By

2

4

6

8

10

Beam Size (k)

34

36

38

40

42

32.6

33.4

34.1

34.2

35.7

35.7

36.7

38.3

42.8

33.2

Ours-PAL (single)

Ours-PAL (multiple)

PAL

(a) effect of beam size

0.2

0.5

0.8

1.0

LM temperature

(

=0.5,

=0.5)

78

80

82

84

79.1

79.2

78.4

77.9

80.8

83.3

84.5

84.8

Ours-PAL (single)

Ours-PAL (multiple)

0.0

0.5

Sampling temperature

(

=0.5,

=0.5)

70

72

74

76

78

80

82

71.8

71.4

77.8

81.3

Ours-CoT (single)

Ours-CoT (multiple)

(b) effect of generation and sampling diversity

comparing the score distributions for correct and wrong predictions, we aim to gain an intuitive understanding of whether these confidence scores are reliable. Figure 5 shows different score distributions on correct and wrong baseline predictions. The difference in distribution between the two prediction sets is substantial for arithmetic reasoning, but negligible for commonsense reasoning. Notably, in both instances, correctness confidence is more discriminatory than generation confidence.

To achieve a balance between these two confidence scores, we utilize a tunable hyperparameter λ , setting λ = 0 . 5 for all datasets. Nevertheless, varying its value can lead to distinct outcomes. For instance, when setting λ to 1 ( E = C ) or 0 ( E = P ), the performance on GSM8K decreases from 80 . 2% to 74 . 5% and 77 . 1% , respectively. This indicates that both scores play a crucial role in our final performance. A more comprehensive analysis of λ can be found in Appendix A.2.

Reasoning Complexity. We investigate if our approach is more beneficial for instances needing more reasoning steps. Table 4 shows that performance gains (in absolute accuracy % increase) increase as reasoning chains become longer on both GSM8K and StrategyQA. Notably, the improvement on StrategyQA primarily comes from improvements in longer reasoning chains, showcasing the effectiveness of our method in navigating lengthy and intricate reasoning chains.

Hyperparameters in Stochastic Beam Search. We examine the significance of hyperparameters associated with stochastic beam search, including the beam size k and the temperatures γ and τ controlling the generation and sampling diversity, respectively.

Figure 6a shows the trend of performance improvement with the increase of beam size k . Notably, our beam search approach inherently enables majority voting on the final beam without additional cost, resulting in a more significant performance improvement in the multiple-chain reasoning when the beam size is larger ( e.g. , 42 . 8% compared with 35 . 7% when k = 10 ).

For generation and sampling diversity, it is clear that more diversity resulting from higher temperatures generally leads to a decline in performance when only considering a single reasoning chain. However, diversity significantly benefits majority voting on multiple reasoning chains 6 . This benefit comes

(b) Examples of self-evaluation score distribution of different predictions on the StrategyQA dataset. We also provide explanations corresponding to the ground-truth answers for reference.

Figure 7: Comparisons among predictions of high and low self-evaluation scores on arithmetic (7a for GSM8K) and commonsense (7b for StrategyQA) reasoning tasks. Scores from low to high are visualized from orange ( 0 . 0 ), yellow ( 0 . 4 ), to green ( 1 . 0 ). Here C , P , and E represent the evaluation confidence, the generation confidence, and their combination as the final score, respectively.

from the improved coverage of the plausible generations and the ensembling effect. Nevertheless, one can adjust the sampling-related parameters ( i.e. , τ and α ) to incorporate more randomness into the generations. In practice, we find that a moderate temperature decay ( e.g. , α = 0 . 5 ) results in improved performance. We conduct further analysis of the effect of sampling diversity in Appendix A.2.

Qualitative Analysis. We examine particular instances to investigate the behavior of correctness confidence scores C and generation probabilities P in different scenarios. From the comparison shown in Figure 7, we have the following main observations:

In general, the correctness confidence is more effective at identifying logical errors, taking into account the accumulated mistakes from prior steps, while the generation probability focuses more on text perplexity as the confidence of the generation LLM.

When comparing arithmetic and commonsense tasks, LLMs exhibit greater confidence in dealing with structured and objective reasoning chains such as problems in GSM8K, for both generation and self-evaluation, as opposed to reasoning chains in StrategyQA.

Reasoning chains that appear logically plausible can achieve high correctness confidence scores but still result in incorrect answers, as demonstrated in R 41 in Figure 7b. Moreover, the correctness confidence can be influenced by minor details ( e.g. , imperfect variable naming in PAL reasoning) and assign low scores regardless of the correctness of the final answers as shown in R 22 in Figure 7a.

Incoherence due to a sudden jump in reasoning ( e.g. , R 32 in Figure 7b) can lead to low correctness confidence. Additionally, the correctness confidence tends to be lower when the generation LLM makes a probability statement with less certainty, such as 'it seems' as illustrated by R 42 in Figure 7b.

4 Related Work

Reasoning Formulation. Several studies have attempted to better formulate the reasoning problem. One approach is to generate rationales to enhance model interpretability (Zhou et al., 2020; Wiegreffe and Marasovic, 2021; Wiegreffe et al., 2021). Recently, the focus has shifted towards decomposing the reasoning process into intermediate steps before reaching the final answer (Wei et al., 2022b; Zhou et al., 2023; Gao et al., 2023; Chen et al., 2022). Various decomposition techniques have been explored, such as question reduction (Zhou et al., 2023; Yang et al., 2022), iterative prompting (Wang et al., 2022), and chaining the steps (Wu et al., 2022). While incorporating intermediate reasoning steps has resulted in substantial performance improvements, errors or imperfections can accumulate, especially when the chains become longer (Wu et al., 2016; Guo et al., 2018). As such, we utilize LLM self-evaluation as a stepwise criterion to improve the chaining process.

LLMSelf-Evaluation. Recent research on LLM calibration shows that current LLMs' probabilistic predictions correspond well with actual token occurrence frequencies, leading to well-calibrated predictions for specific tasks (Rae et al., 2021; Kadavath et al., 2022; Guo et al., 2017; Kadavath et al., 2022; Jiang et al., 2021; Kuhn et al., 2023). Notably, scaling model size plays a crucial role in enhancing calibration (Rae et al., 2021; Wei et al., 2022a). As LLMs exhibit good calibration, an increasing number of studies focus on prompting LLMs to perform self-evaluation as a means of verification (Zhang et al., 2023; Shinn et al., 2023; Madaan et al., 2023; Paul et al., 2023). Selfevaluation provides an effective and efficient assessment method without requiring task-specific verifier fine-tuning, which typically involves additional annotations (Li et al., 2022). In contrast to existing works that refine generation results through instance-level self-evaluation, our approach applies self-evaluation results as a stepwise criterion to calibrate generation at a finer granularity. By focusing on step-by-step self-evaluation, our method enables fine-grained guided decoding, addressing the challenges associated with complex or lengthy reasoning.

Decoding Strategies. A tradeoff typically exists between diversity and quality. Deterministic decoding methods such as greedy decoding and beam search (Jurafsky and Martin, 2009; Graves, 2012) often produce high-quality results but lack diversity (Stahlberg and Byrne, 2019; Meister et al., 2020). Temperature sampling (Ackley et al., 1985), topk sampling (Fan et al., 2018), and topp sampling (Holtzman et al., 2020) are various techniques used to enhance diversity. The recent work of tree-of-thought (Yao et al., 2023) explores different search algorithms such as breadth-first and depthfirst searches tailored for different tasks. Differently, we propose a unified framework of stochastic beam search (Caccia et al., 2020; Kool et al., 2019; Meister et al., 2021), which combines beam search and temperature sampling to balance the quality-diversity trade-off in multi-step reasoning.

5 Discussion

We have introduced a multi-step decoding method that calibrates reasoning with stepwise selfevaluation guidance via stochastic beam search for current large language models. The empirical success of our method across a broad range of tasks, from arithmetic and symbolic to commonsense reasoning, demonstrates its robustness and generalizability in various application areas. The significant performance gains of our method on long reasoning chains also highlight its applicability to other multi-step tasks, such as multi-hop question answering and more complex scenarios involving multi-modal understanding, reasoning, and planning. In future work, we will investigate how to utilize external tools to further enhance the calibration and explore its generalizability on other multi-step scenarios to deal with more complex information such as external knowledge and multimodalities.

Potential Impacts and Limitations

We propose self-evaluation guided stochastic beam search to facilitate multi-step reasoning. However, our approach, based on stepwise self-evaluation guidance, has certain limitations. It requires access to LLM logits to calculate the self-evaluation score, restricting its applicability to more powerful LLMs, such as GPT-4, which do not provide token likelihoods. Plus, multi-step decoding inherently causes additional costs from candidate sampling and self-evaluation. For optimal balance between efficiency and cost, our approach is best applied to longer reasoning chains, where the cumulative effect of calibration across multiple steps can improve the overall performance more significantly.

Acknowledgments and Disclosure of Funding

The computational work for this article was partially performed on resources of the National Supercomputing Centre (NSCC), Singapore 7 . We would like to thank Prof. Hwee Tou Ng for his insightful discussions that enhanced the depth and quality of our study.

References

David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. 1985. A learning algorithm for boltzmann machines. Cognitive Science , 9(1):147-169.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .

Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2020. Language gans falling short. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR , abs/2211.12588.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. CoRR , abs/2204.02311.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR , abs/2110.14168.

Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 889-898, Melbourne, Australia. Association for Computational Linguistics.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: program-aided language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 10764-10799. PMLR.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguistics , 9:346-361.

Alex Graves. 2012. Sequence transduction with recurrent neural networks. CoRR , abs/1211.3711.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 1321-1330. PMLR.

Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2018. Long text generation via adversarial training with leaked information. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1).

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net.

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve.

Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know When language models know? on the calibration of language models for question answering. Trans. Assoc. Comput. Linguistics , 9:962-977.

Dan Jurafsky and James H. Martin. 2009. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition . Pearson Prentice Hall, Upper Saddle River, N.J.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. CoRR , abs/2207.05221.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In NeurIPS .

Wouter Kool, Herke van Hoof, and Max Welling. 2019. Stochastic beams and where to find them: The gumbeltop-k trick for sampling sequences without replacement. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 3499-3508. PMLR.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. In NeurIPS .

Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners.

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers , pages 158-167. Association for Computational Linguistics.

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. CoRR , abs/2303.17651.

Clara Meister, Afra Amini, Tim Vieira, and Ryan Cotterell. 2021. Conditional poisson stochastic beam search. CoRR , abs/2109.11034.

Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. Best-first beam search. Trans. Assoc. Comput. Linguistics , 8:795-809.

Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 975-984. Association for Computational Linguistics.

Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. CoRR , abs/2112.00114.

OpenAI. 2023. GPT-4 technical report. CoRR , abs/2303.08774.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021 , pages 2080-2094. Association for Computational Linguistics.

Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. REFINER: reasoning feedback on intermediate representations. CoRR , abs/2304.01904.

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, PoSen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021. Scaling language models: Methods, analysis & insights from training gopher. CoRR , abs/2112.11446.

Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. CoRR , abs/2303.11366.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR , abs/2206.04615.

Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3356-3362, Hong Kong, China. Association for Computational Linguistics.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4149-4158. Association for Computational Linguistics.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288.

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback.

Boshi Wang, Xiang Deng, and Huan Sun. 2022. Shepherd pre-trained language models to develop a train of thought: An iterative prompting approach. CoRR , abs/2203.08383.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. Trans. Mach. Learn. Res. , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS .

Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual .

Sarah Wiegreffe, Ana Marasovic, and Noah A. Smith. 2021. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 10266-10284. Association for Computational Linguistics.

Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , CHI '22, New York, NY, USA. Association for Computing Machinery.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR , abs/1609.08144.

Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin, and Diyi Yang. 2022. SEQZERO: few-shot compositional semantic parsing with sequential prompts and zero-shot models. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022 , pages 49-60. Association for Computational Linguistics.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. CoRR , abs/2305.10601.

Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-Tau Yih, Daniel Fried, and Sida Wang. 2023. Coder reviewer reranking for code generation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 41832-41846. PMLR.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net.

Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang. 2020. Towards interpretable natural language understanding with explanations as latent variables. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .

A Appendix

A.1 Theoretical Analysis of Eq. 4

In Eq. 4, we use S sampled from the language model LM G generations. This is an approximation for sampling from the infinite set of all possible chaining paths. And the finite set S is constructed based on the generation LM P LM G , which is different from our target distribution as shown in Eq. 4.

Specifically, denote the infinite set of all possible generated completions till the t -th step as S ∗ , we approximate sampling from P ∗ beam ( s 1: t ) = exp ( E ( s 1: t ) /τ ) ∑ s 1: t ∈S ∗ exp ( E ( s 1: t ) /τ ) via P beam ( s 1: t ) = exp ( E ( s 1: t ) /τ ) ∑ s 1: t ∈S exp ( E ( s 1: t ) /τ ) , where S is the approximation of S ∗ with |S| = nk = M ≤ |S ∗ | .

Define the upper bound ¯ c and the lower bound c on each exp( E ( s 1: t ) /τ ) as ¯ c ≥ exp( E ( s 1: t ) /τ ) ≥ c for all s 1: t ∈ S ∗ . Define the ratio as r = ¯ c/ c . Note that c ≥ 1 since E ( s 1: t ) /τ ≥ 0 . Thus, we can take r ≤ ¯ c .

We now give the following proposition which shows that | P ∗ beam ( s 1: t ) -P beam ( s 1: t ) | decreases at the rate of O ( 1 -M/ |S ∗ | M ) toward 0 as M increases. Note that as M increases toward |S ∗ | , the numerator 1 -M/ |S ∗ | decreases toward 0 while the factor for the denominator 1 M also decreases.

Proof. We now prove the second statement by analyzing the absolute difference:

Since exp( E ( s 1: t ) /τ ) is nonnegative, using the upper bound on each exp( E ( s 1: t ) /τ ) , we have:

Similarly, using the lower bound on each exp( E ( s 1: t ) /τ ) ,

A.2 Extended Experiments

0.00

0.25

0.50

0.75

1.00

Sampling temperature

(

=0.5)

74

76

78

80

80.2

79.4

76.6

79.0

78.8

77.6

LM temperature

=0.8

LM temperature

=1.0

0.2

0.4

0.6

0.8

1.0

Decay ratio

(

=0.5)

77.9

79.4

74.1

78.4

78.8

73.4

LM temperature

=0.8

LM temperature

=1.0

Sampling Diversity. In accordance with Figure 6b, we observe similar results when ablating the sampling hyperparameters τ and α for the single reasoning chain case, as shown in Figure 8. Increasing τ and α generally adds more diversity to the decoding process, but excessive randomness negatively impacts the performance of the single-chain decoding. Generally, a moderate temperature decay results in improved performance. Therefore, we set α = 0 . 5 throughout our experiments for simplicity and only tune τ for randomness control.

0.2

0.4

0.6

0.8

0.74

0.76

0.78

0.80

0.82

AUC

=0.2

=0.4

=0.8

(a)

λ

-AUC curves of

E

λ

on GSM8K (PAL).

0.2

0.4

0.6

0.8

0.58

0.59

0.60

0.61

0.62

AUC

=0.2

=0.5

=0.8

(b)

λ

-AUC curves of

E

λ

on StrategyQA (CoT).

High-scored (

>0.90) Predictions

Low-scored (

<0.70) Predictions

0.0

0.2

0.4

0.6

0.8

1.0

Instance-level Accuracy

(a) GSM8K (PAL prompting)

High-scored (

>0.70) Predictions

Low-scored (

<0.50) Predictions

0.0

0.2

0.4

0.6

0.8

1.0

Instance-level Accuracy

(b) StrategyQA (CoT prompting)

More Analysis on Self-Evaluation. Recall that we use a combination of generation confidence and faithfulness score as E λ = C λ · P (1 -λ ) , with λ ∈ [0 , 1] . In our experiments, we set λ = 0 . 5 for all tasks for simplicity. However, we investigate its effects here since, intuitively, it is an important hyperparameter for distinguishing correct / incorrect predictions and might require different values for various reasoning tasks and datasets. Its effect is also coupled with the language model temperature γ .

Figure 9 demonstrates how λ functions on arithmetic (GSM8K) and commonsense (StrategyQA). In general, we observe that the performance remains relatively stable with different choices of λ on different datasets, although fine-tuning this hyperparameter might lead to further improvements. This stability suggests that the choice of λ is not overly sensitive across various reasoning tasks and datasets, but exploring its optimal value for specific tasks could potentially lead to even better performances.

To examine the influence of incorporating faithfulness on LLM final predictions, we plot the distributions of changes in different scores, specifically the faithfulness score C , the generation confidence P , and the overall decoding score E λ on the baseline reasoning chains and the reasoning chains generated by our method. We categorize the data points into 4 sets based on whether our approach changes the final prediction. Since the majority of the data points belong to the 'both correct' set (in blue), where both baselines and our method generate accurate predictions, we particularly highlight the last two sets (in green and red), where our method results in improvement and degradation, respectively.

As shown in Figure 11, faithfulness typically works by significantly increasing the evaluation confidence C of model predictions, while the generation confidence P remains similar to that of the baseline methods. Specifically, for the evaluation confidence C , our approach corrects the original predictions by increasing the confidence scores. This indicates that evaluation confidence plays a crucial role in guiding the decoding toward a better reasoning choice in decomposed reasoning. The increase is more significant for PAL when compared with CoT. This demonstrates that LLMs are generally better at judging the logic in reasoning that is more structured, while the free-text intermediate steps ( e.g. , CoT reasoning) may be challenging to conduct information extraction and soundness checking.

A similar conclusion can be drawn from Figure 10, where the difference in instance-level accuracy distributions between high-scored and low-scored predictions is more significant on the GSM8K dataset. For StrategyQA, while the incorporation of faithfulness helps, the level of the score value does not align well with whether the prediction is correct. For example, most of the low-scored predictions can still obtain the correct answers, as shown by the plot on the right of Figure 10b.

change in generation probability (

)

0

100

200

both correct

both wrong

wrong

correct

correct

wrong

change in correctness confidence (

)

both correct

both wrong

wrong

correct

correct

wrong

change in score (

),

=

(1

)

,

=0.5

both correct

both wrong

wrong

correct

correct

wrong

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

change in generation probability (

)

0

10

20

wrong

correct

correct

wrong

0.2

0.0

0.2

0.4

0.6

change in correctness confidence (

)

wrong

correct

correct

wrong

0.2

0.0

0.2

0.4

0.6

change in score (

),

=

(1

)

,

=0.5

wrong

correct

correct

wrong

change in generation probability (

)

0

25

50

75

both correct

both wrong

wrong

correct

correct

wrong

change in correctness confidence (

)

both correct

both wrong

wrong

correct

correct

wrong

change in score (

),

=

(1

)

,

=0.5

both correct

both wrong

wrong

correct

correct

wrong

0.4

0.2

0.0

0.2

0.4

change in generation probability (

)

0

10

20

wrong

correct

correct

wrong

0.1

0.0

0.1

0.2

0.3

change in correctness confidence (

)

wrong

correct

correct

wrong

0.1

0.0

0.1

0.2

0.3

change in score (

),

=

(1

)

,

=0.5

wrong

correct

correct

wrong

change in generation probability (

)

0

50

100

150

both correct

both wrong

wrong

correct

correct

wrong

change in correctness confidence (

)

both correct

both wrong

wrong

correct

correct

wrong

change in score (

),

=

(1

)

,

=0.5

both correct

both wrong

wrong

correct

correct

wrong

0.6

0.4

0.2

0.0

0.2

0.4

change in generation probability (

)

0

5

10

15

wrong

correct

correct

wrong

0.2

0.0

0.2

0.4

change in correctness confidence (

)

wrong

correct

correct

wrong

0.2

0.1

0.0

0.1

0.2

0.3

change in score (

),

=

(1

)

,

=0.5

wrong

correct

correct

wrong

LLMBackbone Study. We are interested in how stronger LLMs ( i.e. , ChatGPT, GPT-4 (OpenAI, 2023)) work, but they are not directly compatible with our approach since the API does not return token logits.

Table 5 compares the results of various backend LLMs ( i.e. , Codex, ChatGPT, and GPT-4) on GSM8K. In arithmetic reasoning with PAL prompting, our Codex-based method achieves competitive results ( 80 . 2% vs. 78 . 7% ) even when compared with ChatGPT. The results are consistent across other datasets, including AQuA ( 55 . 9% vs. 54 . 7% ), SVAMP ( 89 . 6% vs. 84 . 1% ), ASDiv ( 84 . 9% vs. 84 . 1% ), and TabMWP ( 79 . 1% vs. 80 . 6% ). In commonsense reasoning, our method using Codex significantly outperforms ChatGPT-based methods across different datasets, including StrategyQA ( 77 . 2% vs. 65 . 9% ), CommonsenseQA ( 78 . 6% vs. 75 . 2% ) and Sports Understanding ( 98 . 4% vs. 95 . 9% ). One possible explanation is that ChatGPT lacks sufficient world knowledge for effective fact checking and commonsense reasoning. Given the significant performance improvement of GPT-4, we conduct further analysis about how to synergistically combine it with our method.

GPT-4 Experiments The recently launched GPT-4 has demonstrated notable improvements in reasoning capabilities across a variety of tasks. In this section, we examine and compare the reasoning skills of different large language models (LLMs), specifically Codex and GPT-4, in assessing and determining the accuracy of each step in a reasoning chain. We contrast the confidence scores and corresponding explanations for Codex ( C ) and GPT-4 ( S ) in the context of both arithmetic and commonsense reasoning, as shown in Figure 13 and Figure 14, respectively. For ease of visualization, we employ the same colormap (shown in Figure 12) as in Figure 7 to represent the scale of scores. Since OpenAI has not provided access to the token-wise likelihood of generated text, we directly request GPT-4 to score the reasoning steps using binary values 8 . Moreover, we report the average of three evaluation results to reduce the variance of sampling discrete values, i.e. , S = ( S 1 + S 2 + S 3 ) / 3 , S i ∈ [0 , 1] .

As illustrated in Figure 13, GPT-4 demonstrates greater effectiveness in pinpointing the central logical error in arithmetic reasoning. For instance, we can observe that S < C for alex_total = alex_weight + weight_multiplier * grace_weight and S > C for answer = grace_weight + alex_total , where the former leads to an incorrect final answer. Additionally, GPT-4 typically offers detailed explanations and alternative solutions. As seen in the step answer = grace_weight + alex_total , GPT-4 can correct minor errors even when it arrives at the correct final answer. However, GPT-4 may still encounter difficulties in detecting small errors within the text, which can have a significant impact on logical consistency. This challenge is illustrated by the substantial variance in S for the step alex_total = alex_weight + weight_multiplier * grace_weight .

The benefits of well-crafted explanations in GPT-4 become more significant when handling complex reasoning tasks, as demonstrated in Figure 14. For instance, in the R 42 of Q 4 shown in Figure 7b, Codex generally assigns high evaluation scores for all steps. Although this reasoning chain leads to the correct final answer, it makes some overly definitive assumptions without reasonable justification (e.g., 'must have attributes that match both'). In such cases, GPT-4 can accurately identify these vague statements through meticulous analysis. Moreover, the comprehensive analysis helps address the growing uncertainty arising from the ambiguity in understanding commonsense questions. For example, in the final step, GPT-4 offers extensive explanations for assigning low S scores, considering the limited information available to comprehend the question.

In summary, GPT-4 demonstrates promising improvements in LLM evaluation by providing more rigorous and in-depth explanations. While there is significant variation in its generated explanations for some challenging cases, these explanations are typically beneficial as they elaborate on understanding and interpretation based on the given information. This suggests the potential for using LLM explanations to enhance self-evaluation in future research.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

self-evaluation scores:

|

|

[

Q

1

]

Grace

weighs

125

pounds.

weighs

2

pounds less than

4

times what

Grace

weighs.

What are their

combined

weights in pounds?

[Ground-Truth

a

1

*]

623.0

[Predicted

a

12

]

627.0

[

R

12

]

in

Python

grace_weight = 125

Ɛ

C

correct

S

S

1

correct

S

2

The line of code correctly assigns the given weight of Grace (

125

pounds) to the variable

grace_weight

S

3

The given line of code correctly assigns the value of

125

to the variable

grace_weight

, representing

Grace's weight in pounds.

alex_weight = 2

Ɛ

C

alex_weight

should be calculated based on

grace_weight

S

S

1

alex_weight

should be calculated as

4

times

grace_weight

minus

2

S

2

The line of code defining

alex_weight

should be calculated as

4

times Grace's weight minus

2

, which

is

(

4

*

grace_weight

) -

2

.

S

3

The calculation for Alex's weight is not correct. It should be calculated as

4

*

grace_weight

-

2

.

weight_multiplier = 4

Ɛ

C

correct

S

S

1

The line of code is correct as it defines the multiplier for Alex's weight based on Grace's weight.

S

2

Correct because it assigns the correct value (

4

) to the variable

weight_multiplier

, which represents

the factor by which Grace's weight is multiplied to calculate Alex's weight.

S

3

The code correctly assigns the value

4

to the variable

weight_multiplier

, which will be used in the

calculation of Alex's weight.

alex_total = alex_weight + weight_multiplier * grace_weight

Ɛ

C

correct formula to calculate

alex_total

S

S

1

The formula used to calculate Alex's weight is incorrect.

S

2

The above line of code calculates Alex's weight correctly as

2

pounds less than

4

times Grace's weight

(

4

*

grace_weight

-

2

)

.

S

3

2

pounds less than

4

times what Grace weighs, not

2

pounds more.

answer = grace_weight + alex_total

Ɛ

C

should be the sum of

grace_weight

and

alex_weight

S

S

1

The code calculates Alex's weight correctly using the given relationship and adds it to Grace's weight to

calculate the combined weight. The only minor issue is the unnecessary use of the separate variable

alex_weight

which could directly be replaced with the calculation (

4

*

grace_weight

-

2

)

.

However, this does not affect the correctness of the answer.

S

2

The result variable holds the correct combined weight of Grace and Alex, and this line of code correctly

calculates the value of

answer

.

S

3

The code above correctly calculates the total weight of Grace and Alex based on the given problem

statement. The only minor issue is that the

alex_weight

variable is initialized as

2

, which might be a

bit confusing. It would be better to directly calculate

alex_total

as

(

4

*

grace_weight

-

2

)

, but the

current code does not produce an incorrect result.

❌

[

Q

4

]

Is

a combination of

Athena

and

Aphrodite

?

[Ground-Truth

a

4

*]

yes

[Explanation]

Athena

was the

Greek

goddess of

war

.

Aphrodite

was the

Greek

goddess of

love

.

was the

Norse

goddess of

war

,

love

, and

fertility

.

[Predicted

a

42

]

yes

[

R

42

]

in

Chain-of-Thought

✔

Freya is the goddess of love, beauty and fertility.

Ɛ

C

correct

S

S

1

Freya indeed is associated with love, beauty, and fertility in

Norse

mythology. However, it is important to note that while the

comparison to Athena and Aphrodite is understandable, it is not a direct or complete equivalence.

S

2

The statement correctly identifies Freya. However, the statement does not mention that Freya is a

Norse

goddess, which could

slightly impact the understanding of the comparison to

Greek

goddesses Athena and Aphrodite.

S

3

The statement is mostly correct as Freya is indeed a goddess associated with love, beauty, and fertility. However, it might be

worth mentioning that she is a

Norse

goddess, while Athena and Aphrodite are

Greek

goddesses, to provide more context.

You can compare these to the Greek gods Athena and Aphrodite.

Ɛ

C

correct

S

S

1

It is true that Freya can be compared to the

Greek

gods Athena and Aphrodite in terms of their respective domains. However,

the comparison is not entirely accurate as Freya comes from

Norse

mythology, and the attributes of the gods may not be an

exact match. The comparison is a simplification, which is why the probability is not higher.

S

2

The statement establishes a comparison between Freya and the

Greek

goddesses Athena and Aphrodite. While this step of

reasoning is generally correct, it could be improved by specifically mentioning that Freya is a

Norse

goddess and by providing

more information about the roles and attributes of Athena and Aphrodite for a clearer comparison.

S

3

While it is true that Freya shares some similarities with Athena and Aphrodite, the comparison is not entirely accurate. Freya is

a

Norse

goddess, while Athena and Aphrodite are

Greek

goddesses. The comparison is correct in the sense that they share

some domains, but it might be misleading to suggest that Freya is a direct "combination" of these two goddesses.

It is not clear how their powers exactly compare.

Ɛ

C

correct

S

S

1

This step of reasoning acknowledges that it is not entirely clear how the powers of Freya, Athena, and Aphrodite compare.

Although there are some similarities, the differences between

Norse

and

Greek

mythologies make it difficult to determine

whether Freya is a direct combination of Athena and Aphrodite. The high probability reflects the acknowledgment of the

uncertainty in comparing the powers of the gods across different mythologies.

S

2

The statement acknowledges that it is not clear how the powers of Freya, Athena, and Aphrodite exactly compare. However,

the question asks if Freya is a combination of Athena and Aphrodite, which implies a synthesis or blending of their roles or

attributes. The reasoning could be improved by specifically mentioning the roles and attributes of Athena and Aphrodite, and

whether or not they overlap with Freya's roles and attributes.

S

3

The reasoning here is more accurate, acknowledging that while there are similarities between Freya and Athena and

Aphrodite, it is not clear how their powers exactly compare. It is important to consider that Freya is a

Norse

goddess and

Athena and Aphrodite are

Greek

goddesses, so their roles, domains, and characteristics may not be directly comparable.

However, it seems that Freya must have attributes that match both.

Ɛ

C

correct

S

S

1

incorrect because it assumes that Freya must have attributes that match both Athena and Aphrodite. While there may be some

similarities in their attributes, it is not appropriate to conclude that Freya is a combination of the two

Greek

goddesses. Norse

and

Greek

mythologies are distinct, and the deities in these pantheons may have different origins and roles.

S

2

The statement acknowledges that Freya has attributes that match both Athena and Aphrodite. However, it would be more

accurate to discuss the specific attributes of Athena and Aphrodite to determine the extent of the similarities.

S

3

It assumes that Freya must have attributes that match both Athena and Aphrodite, but this may not be the case. While there

are similarities between these goddesses, they are from different mythologies (Freya -

Norse

, Athena & Aphrodite -

Greek

),

and their roles, domains, and characteristics may not be directly comparable or combined. It is not accurate to assume that

Freya is a combination of Athena and Aphrodite without a more in-depth analysis in their respective mythologies.

So the answer is yes.

Ɛ

C

correct

S

S

1

This step of reasoning is incorrect because it is just based on some similarities in their attributes. While there may be some

similarities, it is not accurate to say that Freya is a combination of the two

Greek

goddesses.

Norse

and

Greek

mythologies

are distinct, and the deities in these pantheons have different origins and roles.

S

2

Although Freya shares some attributes with both Athena and Aphrodite, it is an oversimplification to claim that she is a

combination of the two. The question seems to imply that Freya might be a direct mix of the two, but this is not accurate.

S

3

The conclusion that Freya is a combination of Athena and Aphrodite is not accurate based on the information provided. While

there are similarities between these goddesses, they belong to different mythologies (Freya is

Norse

, Athena and Aphrodite

are

Greek

), and their roles, domains, and characteristics may not be directly comparable or combined.

A.3 Implementation Details

Similar to beam search, we maintain k distinct candidates in the beam and sample n completions for each one. Thus, for each reasoning step s t , the search space has a size of k · n . After acquiring k · n samples, we retain k candidates by sampling from P beam as Eq. 4. We set k = 5 , n = 16 with Codex backbone to balance the quality and efficiency. The maximum number of steps to decode is capped at 16 . To control the computational cost and time complexity, one can also reduce the number of rollouts per beam and the beam size to n = 2 and k ∈ [3 , 4] respectively, as we illustrate with Llama-2 backbone.

We set generation temperatures differently for various tasks and baselines. Regarding the generation temperature γ on Codex, for arithmetic and symbolic reasoning with PAL using deterministic beam search ( τ = 0 . 0 ), we find that γ ∈ [0 . 4 , 0 . 8] generally works well. In contrast, for commonsense reasoning with CoT, a lower temperature ( γ ∈ [0 . 1 , 0 . 5] ) is more effective, likely due to the increased randomness from the free-text format. Differently, when using Llama-2 backbone, PAL generally works better with lower generation temperature γ ≤ 0 . 5 , while CoT can tolerate larger γ > 0 . 5 with better or comparable performance. This difference between Codex and Llama-2 may come from the different training tasks and data adopted for the two models, where the PAL reasoning is especially enhanced in Codex.

In majority voting, higher γ is preferred to better explore the search space in reasoning, e.g. , γ ≥ 0 . 5 for arithmetic reasoning. To further introduce sampling randomness in stochastic beam search for majority voting on multiple reasoning chains, we use α = 0 . 5 for all datasets but different values of τ for each task. Specifically, we choose τ = 0 . 5 for PAL and τ = 0 . 2 for CoT, as PAL typically decomposes the reasoning problem into more steps than CoT. Here we tune τ instead of α to be smaller in CoT as CoT naturally contains more randomness due to its free-text formulation as we observe in practice, where a smaller τ is more efficient to balance this randomness.

In previous works, majority voting on multiple reasoning chains involves sampling N (usually ≥ 20 ) reasoning chains and conducting a vote to determine the final answer, which can be time-consuming. In our approach, we simply perform majority voting among the N candidates in the last step of beam search from only a few times ( ≤ 10 ) of searching. As a result, our method does not introduce additional time complexity compared to the conventional majority voting method, although we sacrifice some diversity in the final outcomes due to the similarity of candidates within a beam.

Prompts. We show examples of both the generation and evaluation prompts we use on different tasks in the following tables, where we only show one instance for each case. Full prompts and detailed formulations can be found in our code.

Table 6: Examples of few-shot exemplars of generation and evaluation CoT prompts on GSM8K. We adopt 8 -shot and 5 -shot learning for generation and evaluation, respectively.

GENERATION

Q : There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?

A :

There are 15 trees originally.

And there were 21 trees after some more were planted.

So 21 - 15 = 6 trees were planted.

So the answer is 6.

EVALUATION

Q : Allison brought some CDs online. Each CD cost $7. There was an additional charge of $4 per order for shipping costs. The total bill came to $60. How many CDs did Allison buy?

A :

Each CD cost 7 dollars.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is (A)

And there was an additional charge of 4 dollars.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is (A)

So the total cost of each CD is 7 + 4 = 11 dollars.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is (B), because each CD costs 7 dollars.

So 60 / 11 = 5.45.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is (B), because it cost 4 dollars for shipping costs. So the cost of CDs is 60 - 4 = 56 dollars. So Allison bought 56 / 7 = 8 CDs.

So the answer is 5.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is (A), but the value of the number of CDs is incorrect.

Table 7: Examples of few-shot exemplars of generation and evaluation PAL prompts on GSM8K . We adopt 9 -shot and 5 -shot learning for generation and evaluation, respectively.

GENERATION

Q : Olivia has $23. She bought five bagels for $3 each. How much money does she have left?

EVALUATION

Q : A piece of square paper has a perimeter of 32 centimeters. Nicky's dog, Rocky, tore off 1/4 of the paper. What is the area of the remaining paper?

Table 8: Examples of few-shot exemplars of generation and evaluation PAL prompts on AQuA . Here ... represents the same evaluation script as those in the previous steps. We adopt 8 -shot and 5 -shot learning for generation and evaluation, respectively.

GENERATION

Question : In a flight of 600 km, an aircraft was slowed down due to bad weather. Its average speed for the trip was reduced by 200 km/hr and the time of flight increased by 30 minutes. The duration of the flight is:

Answer Choices : A)1 hour; B)2 hours; C)3 hours; D)4 hours; E)5 hours

EVALUATION

Question : Two trains of length 150 m and 200 m are 100 m apart. They start moving towards each other on parallel tracks, at speeds 54 kmph and 72 kmph. In how much time will the trains cross each other?

Answer Choices : A)100/7 sec; B)80/7 sec; C)57/7 sec; D)110/7 sec; E)50/7 sec

Table 9: Examples of few-shot exemplars of generation and evaluation PAL prompts on SVAMP and ASDiv . Here we utilize the same prompts as they have the same task formulation. We adopt 7 -shot and 5 -shot learning for generation and evaluation, respectively.

GENERATION

Passage : James bought 93 red and 10 blue stickers, he used 31 red sticker on his fridge and 7 blue stickers on his laptop.

Question : How many red stickers does James have?

EVALUATION

Passage : A piece of square paper has a perimeter of 32 centimeters. Nicky's dog, Rocky, tore off 1/4 of the paper.

Question : What is the area of the remaining paper?

Table 10: Examples of few-shot exemplars of generation and evaluation PAL prompts on TabMWP . We adopt 4 -shot and 5 -shot learning for generation and evaluation, respectively.

GENERATION

Q : 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?

EVALUATION

Q : Yesterday was April 30, 2021. What is the date yesterday in MM/DD/YYYY?

GENERATION

Q : I have a drum, a flute, a clarinet, a violin, four accordions, a piano, a trombone, and a trumpet. How many musical instruments do I have?

EVALUATION

Q : I have a peach, an onion, two garlics, a nectarine, a yam, a carrot, a potato, a head of broccoli, a cabbage, a banana, and a stalk of celery. How many vegetables do I have?

Table 13: Examples of few-shot exemplars of generation and evaluation CoT prompts on CSQA . We adopt 7 -shot and 3 -shot learning for generation and evaluation, respectively.

GENERATION

Q : What do people use to absorb extra ink from a fountain pen?

Answer Choices : (a) shirt pocket; (b) calligrapher's hand; (c) inkwell; (d) desk drawer; (e) blotter

A :

The answer must be an item that can absorb ink.

Of the above choices, only blotters are used to absorb ink.

So the answer is (e).

EVALUATION

Q : John's took his appointment book with him to a checkup but couldn't find it anywhere.

He looked and he looked but he didn't know where it was. It wasn't until he bought a new one, went home, took a shower that he found it. Where was it?

Answer Choices : (a) office supply store; (b) wallet; (c) doctor's office; (d) handbag; (e) pocket

A :

The answer must be a place where John could have put his appointment book.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (B), because it fails to reason out that John will check the place when taking a shower (taking off clothes)

Of the above choices, only wallet makes sense.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (B), because wallet is too small to put his appointment book, and it cannot be the only one which makes sense (pocket is also plausible)

So the answer is (b).

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (A), but the previous step is incorrect

Table 14: Examples of few-shot exemplars of generation and evaluation CoT prompts on StrategyQA . We adopt 6 -shot and 4 -shot learning for generation and evaluation, respectively.

GENERATION

Q : Do hamsters provide food for any animals?

A :

Hamsters are prey animals.

Prey animals are food for predators.

Thus, hamsters provide food for some animals.

So the answer is yes.

EVALUATION

Q : While viewing "Scary Movie" is the viewer likely to experience an increase in adrenaline?

A :

Adrenaline is a hormone that is released when the body is under stress.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (A)

Scary movies are designed to make the viewer feel stressed.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (B), because Scary Movie is a film that is a comedy take on horror, intended to make viewers laugh

Thus, the viewer is likely to experience an increase in adrenaline.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (A), but the previous step is incorrect

So the answer is yes.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (A), but the previous step is incorrect

Table 15: Examples of few-shot exemplars of generation and evaluation CoT prompts on Sports Understanding on Big-Bench . We adopt 8 -shot and 2 -shot learning for generation and evaluation, respectively.

GENERATION

Q : Is the following sentence plausible?

'KYLE PALMIERI WAS CALLED FOR SLASHING.'

A :

Being called for slashing is part of hockey.

So the answer is yes.

EVALUATION

Q : Is the following sentence plausible?

'ROBIN LEHNER LAUNCHED THE DESPERATION HEAVE'

A :

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (B), because Robin Lehner is a goaltender which is different from a player

Launching the desperation heave is part of hockey.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (B), because launching the desperation heave is for player, not goaltender

So the answer is yes.

# Is the above step of reasoning:

# (A) Correct

# (B) Incorrect

# The above step of reasoning is: (A), but the previous step is incorrect

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes using a fine-grained self-evaluation method to refine multi-step reasoning inference of the Large Language Models. The authors propose a novel promting approach that integrates self-evaluation guidance through stochastic beam search. Concretely, to guide the beam search process, they employ LLM self-evaluation as an automatic criterion to offer the guiding signal. And they incorporate temperature-controlled randomness to optimize the traditional beam search algorithm in order to balance the quality-diversity trade-off for better reasoning chains. Experimental results of their methods on various arithmetic, symbolic, and commonsense reasoning tasks show considerable improvements.\n\nSTRENGTHS:\n1. The authors propose a novel and general multi-step decoding method (a beam search variant) which employs LLM self-evaluation and temperature-controlled randomness. The method aims to improve LLM performance on reasoning tasks.\n2. Extensive experiments are conducted on a wide range of reasoning benchmarks and considerable improvements are achieved. Besides, comprehensive ablations and running examples are presented to further demonstrate the effectiveness of their approach.\n\nWEAKNESSES:\nThe paper should be more self-contained. For example, the notion of self-evaluation is first used in the introduction section without a concise explanation, which makes it a bit hard to follow. And the details of the evaluation benchmark are not introduced (arithmetic, symbolic, and commonsense reasoning tasks).\n\nQUESTIONS:\nAuthors should concisely introduce the notions when they first use it (e.g., self-evaluation) to make the paper more self-contained. Besides, the details of the different evaluation benchmarks and compared baseline methods should be concisely introduced in the experiment section as well.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a self-evaluation enhanced beam search for using LLMs on multi-step reasoning tasks. Specifically, in addition to standard beam search that only scores a beam based on the probability, the method further adjusts the score with a self-evaluation-based faithfulness score reflecting the correctness of intermediate reasoning steps. The score is obtained by LLMs using few-shot prompting and is calculated for each rollout. The paper applies this approach on several arithmetic reasoning tasks, symbolic reasoning tasks, and common sense reasoning tasks. The results suggest that the proposed beam search variant achieves better performance compared to greedy decoding. The paper also includes a series analysis on the effectiveness of calibration, sensitivity to hyper-parameters, etc.\n\nSTRENGTHS:\nThe proposed approach shows an application of self-calibration for multi-step reasoning tasks.\n\nThe experiments cover three tasks and multiple datasets. The paper also includes a series of analyses providing more details on the effectiveness of calibration for reasoning tasks.\n\nThe paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The most critical issue is that the comparison in this paper is misleading.\n\nThe paper draws the majority of attention to the comparison between the proposed beam-search-based approach against greedy decoding. It’s understood that both the greedy decoding and the beam search produce one single chain. But the proposed approach uses *beam search* and self-evaluation (which is also done by few-shot prompting). This is way more expensive than greedy decoding. \n\nSpecifically,  based on the hyper-parameter in Appendix A.3, the model uses beam size (k=5, and use n=16 rollouts for each beam). Assuming we evaluate a problem with steps T=3. In a rough estimation, this approach needs to call the API to generate 5 * 16 * (3 - 1) * 2 = 320 completions (there is a factor 2 because the approach also uses LLM to self-evaluate each rollout, and based on the appendix, the length of the evaluation prompt is at the same magnitude as the length of the few-shot prompt). Not that this estimation follows the way that OpenAI charges API calls, which charges both prompt and completion tokens, see https://openai.com/pricing. In contrast, greedy decoding only requires running 1 completion. There’s a distinct gap in the computation cost (320 vs 1 measured by API cost). Similarly, the computation cost in the multiple reasoning chains setup is also different.\n\nIn short, the experiments and the interpretation of results need substantial refactoring. At least the paper should compare the proposed approach against baselines that uses the same amount of computation instead of producing the same number of reasoning chains. As a result, the improvements in Table 1 are inflated (E.g., considering PAL with 8 samples for self-consistency decoding can get ~78% performance on GSM8K as shown in Figure 12 in PAL).\n\n\n2. At the same time, the core technical contribution of the proposed approach is a straightforward extension of beam search based on self-evaluation, it is hard to judge the empirical merits of the proposed approach without a fair, clear, and thorough comparison.\n\nQUESTIONS:\nCan authors provide the exact cost (measured by API cost, or tokens being scored) for PAL and the proposed approach in the single chain setting of Table 1?\n\n\nWhen calculating $P_{LM}$, is the probability of a reasoning step normalized by the number of tokens?\n\nLIMITATIONS:\nSome limitations are discussed at the end of the paper; no obvious negative societal impact as far as I know.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose using beam search to guide each reasoning step in chain-of-thought prompting. Functionally, the paper is quite similar to the concurrent \"Tree of Thoughts\" work from Yao et al. (2023).\n\nSTRENGTHS:\nOverall, it's a generally solid work. The experiments are reasonable and fairly comprehensive; the method, while not groundbreaking, is the natural next step of a lot of work in this area and is well-motivated; its figures are good and convey its message well; the results are generally positive - it would be very surprising to me if they weren't.\n\nWEAKNESSES:\nPrimary concerns:\n- Because the self-evaluation requires extra inference, it's not entirely obvious that all of the comparisons are fair, so it would be useful to see some discussion around this. There's also the overhead from being unable to cache preceding tokens, but that's an artifact of using an API. The paper doesn't seem to discuss details like the number of tokens generated or the number of beams used, which in general, make comparison hard. \n- The title is very confusing - this paper has little to do with \"decomposition\" since every step taken is sequential. This should really be fixed as it implies a very different message than the one actually made by the paper. Normally this wouldn't be a substantial issue, but in this case, it is especially confusing.\n- The writing could really use work - the phrasing is awkward and often nonstandard, like the first sentence of the abstract \"We endow Large Language Models (LLMs) with fine-grained self-evaluation to refine multi-step reasoning inference.\" In general, the paper reads fairly awkwardly and is unnecessarily wordy.\n\nOther concerns:\n- The novelty of the paper seems to be in the use of model-probability-guided beam search itself, but in my view, the paper somewhat oversells the novelty of using model self-evaluation for selection. Even within the code domain there's lots of work on this, such as \"Coder Reviewer Reranking for Code Generation\" by Zhang et al. (2022). Plus, there is work on evaluating uncertainty with prompting which would probably even be better suited for this, like the concurrent \"Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation\" by Kuhn et al. (2023). This is not to say that the paper is not novel, but the main novelty seems to be the use of self-guided beam search in general and not the \"Self-Evaluation as Faithfulness Constraint.\"\n- This method isn't really possible with recent closed language models that don't expose logprobs (as mentioned), and it's not obvious how to integrate this. I also don't really feel that it's fair to refer to this as faithfulness.\n\nQUESTIONS:\nHow do you ensure the comparison with the other models is fair? How many beams do you use? For k beams, is it more useful to sample 2*k chains of thought?\n\nLIMITATIONS:\nReasonable discussion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to improve the reasoning capability of large language models (LLMs) by decomposing the reasoning task into multiple steps and then performing self-evaluation guided beam search to find the optimal reasoning chain. On a set of arithmetic reasoning, symbolic reasoning, and commonsense reasoning tasks, the proposed method outperforms baselines in most cases. The paper also provided analysis regarding the quality-diversity trade-off in the generation of the reasoning chain.\n\nSTRENGTHS:\n1. This paper proposed a novel and effective stochastic beam search which searches for multi-step reasoning. The approach integrates the LLM probability distribution and the self-evaluation faithfulness score. \n2. The proposed method is evaluated on multiple datasets. The discussion and analyses are helpful.\n\nWEAKNESSES:\n1. The proposed method is very costly, as it requires prompting the LLM for multiple times (when searching for next steps and for self evaluation). Its effectiveness also depends on the LLM search space, e.g., when an LLM is very confident with their (incorrect) reasoning, the potential of locating the correct steps in beam search will become small.\n\n2. Some details are missing or require clarification: \n- How did the framework define one \"step\" in each reasoning dataset? For example, is every step a single statement for PAL and a complete sentence (i.e., ended with a period) for CoT?\n- Figure 4(a) is not referred to in the paper. It is unclear based on which dataset the two plots were drawn. The numbers on the plots also need more explanation. For example, in the left figure, why is PAL+SC so stable when changing the LM temperature (which I believe would end up with very different sets of samples in SC)? Similarly in the right figure, why is CoT+SC changing if it does not involve beam search at all (and hence should be agnostic to the change of the sampling temperature in Eq 4)? \n\n3. Baseline: how does the proposed method compare with self-refinement (Madaan et al., 2023), which, instead of searching for better steps during reasoning, iteratively self-refines the reasoning output?\n\nQUESTIONS:\nPlease address my concerns and questions in Weaknesses.\n\nLIMITATIONS:\nThe paper included a limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors propose a self-evaluation guided strategy to improve the rationale decoding process from CoT sampling. They also explore ways to balance the quality of evaluation with diversity of generation. Overall, their approach work well on existing datasets and improve few-shot accuracy on multiple math and commonsense datasets. Additionally, their approach leads to more robust reasoning.

STRENGTHS:
* The ability to guide the rationale generation process via self quality evaluation and temperature-controlled randomness is an effective solution towards improving the decoding time performance and consistency of CoT sampling process
* The work is timely in the community since CoT is a predominant way to generate rationales using LLMs. This approach can help the effort in improving robustness of these rationales

WEAKNESSES:
* Since the stepwise self-evaluation involves multiple calls to the base LLM, this adds an overall overhead both in terms of compute and cost. There is no discussion regarding this in the paper. This restricts the reader to understand the feasibility of such an approach. Please include some time and cost analysis with and without the self-evaluation based search. I understand that the authors point some of these in the limitations, but I strongly believe they are crucial to have in the paper.
* Can you consider results on more recent models that do provide logits (say GPT3) to showcase the utility of this approach even in the more recent models? This is optional, but a good to have analysis on a few datasets (say StrategyQA and GSM8K).

QUESTIONS:
* How are the different hyper-parameters ($\alpha, \lambda$, etc.) tuned? Or do you use some generic settings for all datasets?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07