SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxi Xie $^{1*}$  Kenji Kawaguchi $^{1}$  Yiran Zhao $^{1}$  James Xu Zhao $^{1}$
Min-Yen Kan $^{1\dagger}$  Junxian  $\mathrm{He}^{2\dagger}$  Michael Qizhe Xie $^{1\dagger}$
$^{1}$  National University of Singapore  $^{2}$  The Hong Kong University of Science and Technology
Breaking down a problem into intermediate steps has demonstrated impressive performance in Large Language Model (LLM) reasoning. However, the growth of the reasoning chain introduces uncertainty and error accumulation, making it challenging to elicit accurate final results. To tackle this challenge of uncertainty in multi-step reasoning, we introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of LLMs. We propose a decoding algorithm integrating the self-evaluation guidance via stochastic beam search. The self-evaluation guidance serves as a better-calibrated automatic criterion, facilitating an efficient search in the reasoning space and resulting in superior prediction quality. Stochastic beam search balances exploitation and exploration of the search space with temperature-controlled randomness. Our approach surpasses the corresponding Codex-backed baselines in few-shot accuracy by  $6.34\%$ ,  $9.56\%$ , and  $5.46\%$  on the GSM8K, AQuA, and StrategyQA benchmarks, respectively. Experiment results with Llama-2 on arithmetic reasoning demonstrate the efficiency of our method in outperforming the baseline methods with comparable computational budgets. Further analysis in multi-step reasoning finds our self-evaluation guidance pinpoints logic failures and leads to higher consistency and robustness. Our code is publicly available at https://guideddecoding.github.io/.
The remarkable empirical achievements of Large Language Models (LLMs) have recently ushered in a new era in machine reasoning through few-shot prompting techniques (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a; OpenAI, 2023). In particular, breaking down a problem into intermediate stages, or a reasoning chain, can significantly improve model performance on reasoning tasks (Cobbe et al., 2021). Various prompting approaches have been proposed to define these chains, such as scratchpads (Nye et al., 2021), chain-of-thought (CoT) (Wei et al., 2022b), least-to-most (Zhou et al., 2023), and program-aided language models (PAL) (Gao et al., 2023; Chen et al., 2022). However, as the complexity and length of reasoning chains increase with the difficulty of tasks, LLMs struggle with errors and imperfections that accumulate across multiple intermediate steps (Wu et al., 2016; Guo et al., 2018; Chen et al., 2022). Furthermore, the growing number of steps leads to an exponential growth in the search space for reasoning, making it exceedingly difficult to obtain accurate final outcomes.
Confronted with the challenges of uncertainty in multi-step chaining, several previous studies have worked on different aspects to alleviate the impact of reasoning errors. For instance, Wang et al. (2023) introduce self-consistency as a method to determine the final answer through majority voting using multiple sampled reasoning paths, while Li et al. (2022) investigate various prompts to diversify the sampling outcomes. Gao et al. (2023) and Chen et al. (2022) utilize Python programs to
*Correspondence to: Yuxi Xie (xieyuxi@u.nus.edu).
$\dagger$  Equal advising. Ordering is determined by dice rolling.
Figure 1: Self-Evaluation can calibrate the decoding direction in multi-step reasoning. We illustrate our method in the form of stepwise stochastic beam search with the beam size equal to 1. The scale of the self-evaluation score is visualized in the colormap. We adopt Program-Aided Language models (PAL) reasoning (Gao et al., 2023; Chen et al., 2022) for this math word problem.
achieve higher accuracy in mathematical computations. While these approaches have contributed to significant performance improvements in reasoning, the process of generating reasoning chains has been parameterized as a standard autoregressive process and intrinsically faces the challenge of sampling within an exponentially large search space.
Motivated by this challenge, we employ LLM self-evaluation (Kadavath et al., 2022) as a better-calibrated criterion to automatically guide the search in the reasoning space, drawing inspiration from prior works on utilizing LLMs for self-evaluation (Rae et al., 2021; Paul et al., 2023; Madaan et al., 2023; Shinn et al., 2023). We integrate the self-evaluation guidance for reasoning in a stepwise and generalizable manner. Specifically, we formulate the reasoning chain generation as a decoding process consisting of multiple intermediate steps. Unlike traditional text decoding where each step produces a single token, we consider each decoding step as a reasoning logic composed of a sequence of tokens. This framework enables us to employ beam search (Jurafsky and Martin, 2009; Graves, 2012) decoding tailored for intermediate steps and guide the beam searching process by controlling the error of each reasoning step to prevent potential error accumulation throughout the chaining. Figure 1 illustrates an example of decoding a chain of program-aided reasoning steps. Furthermore, we incorporate temperature-controlled randomness (Ackley et al., 1985; Kool et al., 2019; Meister et al., 2021) into the traditional (deterministic) beam search to balance the quality-diversity trade-off in searching for better reasoning chains. Our approach has resulted in respectable improvements across various arithmetic, symbolic, and commonsense reasoning tasks. For instance, by guiding the reasoning decoding process of the Codex model (Chen et al., 2021), we achieve accuracies of  $85.5\%$ ,  $64.2\%$ , and  $77.2\%$  on the GSM8K, AQuA, and StrategyQA benchmarks, compared to the vanilla reasoning-enhanced Codex performance of  $80.4\%$ ,  $58.6\%$ , and  $73.2\%$ , respectively. Our further analysis on Llama-2 (Touvron et al., 2023b) demonstrates the efficiency of our method in surpassing the self-consistency baseline under equivalent computational budgets.
2 Self-Evaluation Guided Stochastic Beam Search
Considering the input prompt and question  $Q$  represented as  $x$ , we formulate the answer distribution  $P(a \mid x)$  by decomposing it as a reasoning chain generation process  $P(R \mid x)$  and an answer generation process  $P(a \mid R, x)$ :
$$
P (a \mid x) = \mathbb {E} _ {R \sim P (R | x)} P (a \mid R, x), \tag {1}
$$
where  $R$  is the intermediate reasoning chain variable that is typically a text sequence.  $P(a \mid R, x) = \frac{\mathbb{1}_A(a)}{\max(|A|, 1)}$ , where  $A = \text{execute}(R)$  represents the set of predicted answer(s) interpreted from  $R$ , and  $\mathbb{1}_A$  is the indicator function of the subset  $A$ . In practice,  $|A| \geq 0$  can be 0 or larger than 1 when the reasoning  $R$  returns no valid answer or produces more than one possible answer, respectively.
Figure 2: Our framework of self-evaluation guided stochastic beam search for multi-step reasoning. The schema of the decoding process is on the left, where we keep  $k = 2$  candidates at each timestep, with the detailed illustration of timestep  $t$  at the bottom. Here "Gen" and "Self-Eval" represent the generation and evaluation LLMs, respectively. The corresponding prompt formulations are provided on the right, where the questions  $Q$ , reasoning steps  $R$ , and evaluation scripts are highlighted in orange, green, and yellow, respectively. Steps in light green ( $e.g., s^t$ ) are for models to generate or evaluate at the current timestep. Specifically, we follow Kadavath et al. (2022) to prompt the LLM evaluation by answering the multiple-choice question, i.e., the lines starting with #.
Prior research has modeled the reasoning chain generation  $P(R \mid x)$  by prompting LLMs to explicitly elaborate on the required intermediate steps  $R$ . Through setting different prompting schemes, the reasoning process  $P(R \mid x)$  can be modeled as chain-of-thought free-text reasoning (Kojima et al., 2022; Wei et al., 2022b), a two-stage question decomposition and answering pipeline (Zhou et al., 2023), or program-aided reasoning to generate a python program (Gao et al., 2023; Chen et al., 2022). While effective, previous work mostly uses a single sample of  $R$  from the LLMs to approximate the expectation in Eq. 1 – the generated reasoning chain is often unreliable and causes incorrect answers. To mitigate this issue, Wang et al. (2023) conduct majority voting to approximate the expectation via sampling and aggregating multiple reasoning chains. Li et al. (2022) take a further step to diversify the sampling and calibrate  $P(R \mid x)$  with a task-specific fine-tuned verifier. Another line of work focuses on improving  $P(a \mid R, x)$  instead. For example, Gao et al. (2023) and Chen et al. (2022) employ Python programs for more accurate calculations in math word problems.
In this work, we focus on improving  $P(R \mid x)$  to enhance the consistency of the sampled reasoning chains. To this end, we propose to explicitly break down the reasoning process into multiple steps, as shown in Figure 2, where each step yields a semantically integrated sequence of tokens, representing a single step within the overall reasoning chain. From this perspective, we can approach the task of enhancing  $P(R \mid x)$  as a decoding problem over the reasoning chains. Considering the exponentially large search space and the potential unreliability of LLM-produced chains in reasoning, we propose a constrained stochastic beam search decoding approach to improve the reasoning step by step and obtain high-quality reasoning with a limited number of samples. We detail our approach next.
2.1 Multi-step Reasoning via Stochastic Beam Search
In multi-step reasoning, a reasoning chain of  $T$  steps is sequentially generated through several timesteps as  $R = [s^1, s^2, \dots, s^T] = s^{1:T}$ , where  $s^t$  represents a sequence of tokens as the  $t$ -th step. Formally, the reasoning generation process  $P(R \mid x)$  can be factorized in an autoregressive manner:
$$
P (R = s ^ {1: T} \mid x) = \prod_ {t} P \left(s ^ {t} \mid x, s ^ {1: t - 1}\right), \tag {2}
$$
which resembles the typical token-level autoregressive distribution of language models. Stepwise reasoning allows us to formulate the process as a step-by-step decoding problem, where we can utilize widely-used strategies such as beam search for the generation. Different from the typical text
decoding process where each step consists of a single token, here we view a sequence of reasoning tokens as a single step. One of the most severe issues in LLM-based reasoning is the potential unreliability and inaccuracy of each reasoning step generated by the model. Furthermore, errors from individual steps may accumulate throughout the reasoning chain, exacerbating the problem. To address the issue, we define a constraint function  $\mathcal{C}(s^t,s^{1:t - 1})\in [0,1]$  within each reasoning step that outputs the LLM confidence in the correctness of the reasoning sequence  $s^t$  based on the previous context  $s^{1:t - 1}$ . Then, we present a constrained decoding approach that combines the language model probability and the correctness confidence as a new decoding objective function  $\mathcal{E}(s^{1:T})$ :
$$
\mathcal {E} \left(s ^ {1: T}\right) = \prod_ {t} P _ {\mathrm {L M} \mathcal {G}} ^ {\lambda} \left(s ^ {t} \mid x, s ^ {1: t - 1}\right) \mathcal {C} ^ {1 - \lambda} \left(s ^ {t}\right), \tag {3}
$$
where  $P_{\mathrm{LM}_{\mathcal{G}}}$  is the language model distribution  $^4$ $\lambda \in [0,1]$  is a weight hyperparameter to balance the LM score and the confidence score. We will detail the design of  $\mathcal{C}(s^t)$  in Section 2.2. Eq 3 follows an autoregressive factorization form, and thus traditional token-level decoding methods such as beam search can be applied here on the chain level. As it is desirable to obtain high-quality reasoning chains with limited samples that are scored high by  $\mathcal{E}(s^{1:T})$  , it is natural to utilize greedy or beam search decoding to approximate the reasoning sequences that maximize  $\mathcal{E}(s^{1:T})$
Additionally, multiple diverse reasoning chains could be aggregated to further improve the final accuracy, as suggested by Eq 1 and empirically confirmed by self-consistency reasoning (Wang et al., 2023). To this end, we propose a variant of stochastic beam search (Kool et al., 2019; Meister et al., 2021) to strike a tradeoff between exploration and exploitation. Concretely, for beam size  $k$ , at each reasoning step we draw  $n$  samples of  $s^t$  following  $P_{\mathrm{LM}_{\mathcal{G}}}(s^t \mid x, s^{1:t-1})$  for each beam, and we end up with  $nk$  chain hypotheses of  $s^{1:t}$  to form the candidate set  $S$ , then we perform beam pruning through sampling – we sample  $k$  reasoning beams without replacement, rather than finding the arg max  $k$ , following a distribution defined by the accumulated score:
$$
P _ {b e a m} \left(s ^ {1: t}\right) \propto \exp \left(\mathcal {E} \left(s ^ {1: t}\right) / \tau\right), \quad s ^ {1: t} \in \mathcal {S} \tag {4}
$$
where the temperature  $\tau$  is a hyperparameter to control the randomness in stochastic beam search; when  $\tau \rightarrow 0$ , stochastic beam search becomes the vanilla beam search algorithm. The reasoning beams  $s^{1:t}$  can be sampled efficiently since  $|S| = nk$  is a finite set. To enable fine-grained control of sampling randomness in decoding, we also introduce a hyperparameter  $\alpha \in [0,1]$  so that  $\tau$  can decay step by step as  $\tau \rightarrow \alpha \tau$ . By annealing  $\tau$  with  $\alpha$ , we can mitigate the error accumulation due to aggregated randomness throughout chaining, as discussed in Section 3.4.
By incorporating controllable randomness, we not only achieve a more reliable single reasoning chain generation by setting randomness to be small, but also leverage multiple diverse reasoning chains with larger variance. Next, we introduce our constraint function  $\mathcal{C}(s^t,s^{1:t - 1})$  that utilizes a self-evaluation scheme to improve the consistency of each reasoning step.
2.2 Self-Evaluation as Correctness Control
Inspired by the recent success of self-evaluation (Kadavath et al., 2022; Shinn et al., 2023; Madaan et al., 2023; Paul et al., 2023), a scheme to prompt LLMs to evaluate their own generation, we use LLMs to judge the correctness of  $s^t$  based on  $s^{1:t-1}$ . Specifically, the evaluation and generation models use the same backend LLM with different prompts, which consist of few-shot exemplars. We follow previous works of CoT (Wei et al., 2022b) or PAL (Gao et al., 2023) to formulate the generation prompts. To construct the in-context exemplars prompt $_{C}$  for the self-evaluation LLM  $\mathrm{LM}_{C}$ , we provide stepwise evaluation examples (as question answering with rationales) in each instance. Inspired by Kadavath et al. (2022), we design prompt $_{C}$  in the form of multiple-choice questioning (as shown in Figure 2) to better calibrate the model predictions, where we adopt the token-level probability of option A to represent the correctness score as:
$$
\mathcal {C} \left(s ^ {t}\right) = P _ {\mathrm {L M} _ {\mathcal {C}}} (\mathrm {A} \mid \operatorname {p r o m p t} _ {\mathcal {C}}, Q, s ^ {1: t}) \tag {5}
$$
3For ease of notation, we will use  $\mathcal{C}(s^t)$  throughout the paper when there is no confusion.
<sup>4</sup>We will denote the LM generation probability by  $\mathcal{P}$  throughout the paper for simplification.
In Appendix A.1, we justify the approximation error rate of Eq 4, which computes normalized probability on the subset  $S$  instead of on the entire set.
We present and analyze the results of our self-evaluation guided beam search with different LLM backbones on various reasoning benchmarks. Implementation details including prompt examples and hyperparameter setup can be found in Appendix A.3.
Benchmarks. We evaluate the effectiveness of our approach across three types of reasoning tasks: (1) Arithmetic Reasoning on five math word problem benchmarks, including GSM8K (Cobbe et al., 2021) on math word problems, AQuA (Ling et al., 2017) on algebraic word problems, SVAMP (Patel et al., 2021) on structure variations of math word problems, ASDiv (Miao et al., 2020) on diverse math word problems, and TabMWP (Lu et al., 2023) on tabular math word problems; (2) Symbolic Reasoning on BIG-Bench (Srivastava et al., 2022), which involves Date Understanding of context-based date inferring and Object Counting of enumerating and counting objects of different types; (3) Commonsense Reasoning on three benchmarks, including CommonsenseQA (Talmor et al., 2019) of commonsense questions that require prior world knowledge to answer, StrategyQA (Geva et al., 2021) of questions that require a multi-hop strategy to answer, and Sports Understanding from BIG-Bench (Srivastava et al., 2022) to determine whether a sports-related sentence is plausible.
Baselines. We consider two types of baselines: (1) Chain-of-Thought (CoT) (Wei et al., 2022b) prompting in free-text reasoning and (2) Program-Aided Language models (PAL) (Ling et al., 2017) and Program-of-Thought (PoT) (Chen et al., 2022) prompting in program-aided reasoning. We also include their self-consistency (Wang et al., 2023) variants for multiple-chain reasoning. For generation, we follow the few-shot exemplars of baselines. For self-evaluation, we manually create a set of few-shot exemplars based on the baseline outputs on corresponding training data. We formulate self-evaluation as a task of multiple-choice question answering, following Kadavath et al. (2022). For baselines, we represent the cost as the number of generated tokens. For the cost of our method, we also include the number of additional input tokens in self-evaluation for a fair comparison.
Backboned LLMs. We assess our approach on closed- and open-source LLMs using both PAL and CoT prompting. For closed-source LLMs, we choose Codex (code-davinci-002) (Chen et al., 2021) to report and compare the results on all datasets. We use Llama-2 (13B) (Touvron et al., 2023b) as our open-source LLM to conduct cost-performance analysis on different datasets.
Arithmetic and Symbolic Reasoning. Table 1 shows the results for arithmetic and symbolic reasoning. Our method achieves significant performance improvements on most benchmarks in both single-  $(\tau = 0)$  and multiple-chain scenarios, with PAL as the baseline. For arithmetic reasoning, we observe absolute increases in accuracy of  $5.3\%$ ,  $8.3\%$ , and  $0.7\%$  on GSM8K, AQuA, and SVAMP, respectively. One possible explanation for this discrepancy in improvements is the reduced diversity in LLM generations due to higher confidence in predictions, as evidenced by the relatively high performance on the tasks. This highlights the importance of incorporating controllable randomness into the candidate generations to expand the search space for self-evaluation guided decoding. We further explore the impact of generation diversity by varying the temperature  $\gamma$  in Section 3.4.
For symbolic reasoning, our approach also leads to consistent performance gains. However, when the baseline itself performs well on the task (e.g.,  $96.7\%$  on Object Counting), our approach may not yield substantial improvement. This can also be attributed to the constrained accessible search space for self-evaluation guidance to refine the generation distribution. This limit suggests the inherent deficiency in our LLM-based prompting method that it becomes increasingly challenging to calibrate the generation direction when the model  $\mathrm{LM}_{\mathcal{G}}$  is more confident in its predictions. In other words, the high baseline performance usually indicates lower diversity in the LLM generations even with a large temperature  $\gamma$ , resulting in a limited accessible search space for the model to find a better solution.
Commonsense Reasoning. Table 2 compares methods using CoT prompting in commonsense reasoning. Our approach shows consistent performance improvements across several tasks. For example, on StrategyQA, we achieve an accuracy of  $77.2\%$  compared with  $73.2\%$  of the baseline.
Table 1: Result comparison (accuracy %) on arithmetic and symbolic reasoning tasks. The best result is in **bold** and the lowest cost is in **green**. We report methods all with Codex backbone for a fair comparison. Similar to Huang et al. (2022), Diverse (Li et al., 2022) fine-tune task-specific verifiers to apply weights on samples in self-consistency (SC). Other fine-tuning methods include reward-based supervision (Uesato et al., 2022) and content-specific training (Lewkowycz et al., 2022). We also report the number of tokens (# Tokens) on GSM8K to compare the costs of different methods.
Approach	GSM8K	# Tokens	Arithmetic	Symbolic
AQuA	SVAMP	ASDiv	TabMWP	DATE	OBJECT
single reasoning chain
CoT	65.6	0.2k	45.3	74.8	76.9	65.2	64.8	73.0
PoT	71.6	-	54.1	85.2	-	73.2	-	-
PAL	72.0	0.3k	-	79.4	79.6	-	76.2	96.7
Ours-PAL	80.2	27.7k	55.9	89.6	84.9	79.1	78.6	96.8
multiple reasoning chains
CoT, SC	78.0	5.3k	52.0	86.8	-	75.4	-	-
CoT, Diverse	82.3	-	-	87.0	88.7	-	-	-
PoT, SC	80.0	-	58.6	89.1	-	81.8	-	-
PAL, SC	80.4	7.4k	-	-	-	-	-	-
Ours-PAL	85.5	550.0k	64.2	90.3	85.8	80.9	-	-
Table 2: Result comparison (accuracy %) on commonsense reasoning tasks, with Codex backbone. Here we only report results in the single reasoning chain scenario following Wei et al. (2022b). We report # Tokens on StrategyQA for cost comparison.
Approach	StrategyQA	# Tokens	CommonsenseQA	Sports
CoT	73.2	0.06k	77.9	98.5
Ours-CoT	77.2	11.6k	78.6	98.4
Human	87.0	-	88.9	-
Likewise, the performance of our approach is constrained by the low diversity of LLM generations on Sporting Understanding, as we observe on Object Counting in symbolic reasoning.
Computational Cost Overhead. Despite the fact that our approach achieves significant improvement on various benchmarks, we observe an overhead of computational cost compared with the corresponding baselines. For example, the single-chain version of our approach using PAL costs about 3 times more than the self-consistency baseline on GSM8K. As detailed in Appendix A.3, this is due to a relatively large hyperparameter – the number of rollouts per beam  $n$  – which we set as 16 for better performance. To strike a balance between performance and cost and present a complete picture, we adopt  $n = 2$  and conduct cost-performance analysis on our approach in Section 3.3.
3.3 Cost Analysis
Table 3 compares the baseline and our approach under comparable computational budgets (measured in # Tokens). Our method consistently outperforms self-consistency on the arithmetic reasoning tasks even when benchmarked for relatively less computational cost. For example, we achieve  $46.1\%$  on GSM8K with a cost of  $12.6k$  tokens, compared with the accuracy of  $41.8\%$  of self-consistency which costs  $13.9k$  tokens. Figure 4 further illustrates the cost-efficiency of our approach on GSM8K using different prompting methods under various levels of costs. Our approach significantly outperforms the corresponding equal-cost baseline especially when the computational budget increases, indicating the improvement in the performance upper bound brought by our method.
However, our approach lags behind the CoT baseline on commonsense reasoning. This implies the limitation of our method when applied to shorter reasoning chains, i.e., decreasing the number
Table 3: Cost (# Tokens) and result (accuracy %) comparison on arithmetic and commonsense reasoning tasks. We base our experiments on Llama-2 (13B) since Codex is not available. We show the results of the baseline and our method both in the multiple-chain scenario for a fair comparison. Here we use PAL and CoT prompting for arithmetic and commonsense reasoning, respectively.
Approach	GSM8K	Arithmetic (PAL)	Commonsense (CoT)
AQuA	SVAMP	ASDiv	TabMWP	StrategyQA	CommonsenseQA
Baseline	41.8	30.7	71.2	66.2	43.7	71.0	74.4
# Tokens	13.9k	6.6k	5.9k	2.7k	1.9k	2.7k	1.2k
Ours	46.1	31.5	74.6	67.7	49.6	70.6	74.0
# Tokens	12.6k	6.0k	5.0k	2.5k	1.2k	2.6k	1.2k
(a) PAL Prompting Methods on GSM8K
(b) CoT Prompting Methods on GSM8K