SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Accuracy curves on GSM8K of different methods with the change of the cost. We conduct the performance comparison using both PAL and CoT prompting with Llama-2 (13B) backbone.
(a) Score distribution of PAL baseline predictions on GSM8K.
(b) Score distribution of CoT baseline predictions on StrategyQA.
Figure 5: Distributions of the self-evaluation score and its components (i.e., generation confidence  $\mathcal{P}$  and correctness confidence  $\mathcal{C}$ ) on correct/incorrect baseline predictions. We highlight the median scores of the positive and negative cases using lines of the same colors respectively.
of intermediate steps weakens the effect of stepwise self-evaluation in beam search in reducing error accumulation. On the other hand, self-consistency can directly improve performance through instance-level aggregation without additional cost for self-evaluation. We analyze how our method benefits longer reasoning chains on different tasks in Section 3.4.
We now provide a detailed analysis of why our method achieves significant gains.
Generation and Self-evaluation Calibration. We investigate the distributions of generation confidence (i.e., the LM probability  $\mathcal{P}$ ) and correctness confidence  $\mathcal{C}$  in our self-evaluation score  $\varepsilon$ . By
Table 4: Absolute accuracy (in %) increases on instances of different complexity determined by the length of reasoning chains (represented as # Steps).
# Steps	GSM8K
# Ins.	PAL	Ours	ΔAccu.
< 7	437	85.8	91.3	+5.49
∈ (7,9]	524	74.8	82.6	+7.82
≥ 9	358	72.9	82.6	+9.78
StrategyQA
# Steps	# Ins.	CoT	Ours	ΔAccu.
< 4	637	84.6	84.9	+0.31
∈ [4,5)	1,301	78.6	79.1	+0.46
≥ 5	351	68.4	71.8	+3.42
(a) effect of beam size
Figure 6: Accuracy curves and distributions of our approach on GSM8K with different hyperparameter settings: (a) Changes in performance (Llama-2 backboned) when the beam size  $k$  varies. Methods of the same  $k$  have equal computational costs; (b) Accuracy distributions (Codex backboned) with different generation temperature  $\gamma$  and sampling temperature  $\tau$  (with decay ratio  $\alpha$ ).
(b) effect of generation and sampling diversity
comparing the score distributions for correct and wrong predictions, we aim to gain an intuitive understanding of whether these confidence scores are reliable. Figure 5 shows different score distributions on correct and wrong baseline predictions. The difference in distribution between the two prediction sets is substantial for arithmetic reasoning, but negligible for commonsense reasoning. Notably, in both instances, correctness confidence is more discriminatory than generation confidence.
To achieve a balance between these two confidence scores, we utilize a tunable hyperparameter  $\lambda$ , setting  $\lambda = 0.5$  for all datasets. Nevertheless, varying its value can lead to distinct outcomes. For instance, when setting  $\lambda$  to  $1(\mathcal{E} = \mathcal{C})$  or  $0(\mathcal{E} = \mathcal{P})$ , the performance on GSM8K decreases from  $80.2\%$  to  $74.5\%$  and  $77.1\%$ , respectively. This indicates that both scores play a crucial role in our final performance. A more comprehensive analysis of  $\lambda$  can be found in Appendix A.2.
Reasoning Complexity. We investigate if our approach is more beneficial for instances needing more reasoning steps. Table 4 shows that performance gains (in absolute accuracy % increase) increase as reasoning chains become longer on both GSM8K and StrategyQA. Notably, the improvement on StrategyQA primarily comes from improvements in longer reasoning chains, showcasing the effectiveness of our method in navigating lengthy and intricate reasoning chains.
Hyperparameters in Stochastic Beam Search. We examine the significance of hyperparameters associated with stochastic beam search, including the beam size  $k$  and the temperatures  $\gamma$  and  $\tau$  controlling the generation and sampling diversity, respectively.
Figure 6a shows the trend of performance improvement with the increase of beam size  $k$ . Notably, our beam search approach inherently enables majority voting on the final beam without additional cost, resulting in a more significant performance improvement in the multiple-chain reasoning when the beam size is larger (e.g.,  $42.8\%$  compared with  $35.7\%$  when  $k = 10$ ).
For generation and sampling diversity, it is clear that more diversity resulting from higher temperatures generally leads to a decline in performance when only considering a single reasoning chain. However, diversity significantly benefits majority voting on multiple reasoning chains<sup>6</sup>. This benefit comes
In this study, we did not explore higher generation temperatures (i.e.,  $\gamma > 1.0$ ) since this hyperparameter is limited to 1.0 in the OpenAI API.
$\left|Q_{1}\right|$  Grace weighs 125 pounds. Alex weighs 2 pounds less than 4 times what Grace weighs. What are their combined weights in pounds? [Ground-Truth  $a_{1}^{*}]$  623.0   
 $\left[\mathrm{Predicted~}a_{11}\right]623.0$ $[R_{11}]$  in Python   
C P E grace_weight = 125   
C P E alex_weight = 4 \* grace_weight - 2   
C P E answer = grace_weight + alex_weight   
 $\left[\mathrm{Predicted~}a_{12}\right]627.0$ $[R_{12}]$  in Python   
C P E grace_weight = 125   
C P E alex_weight = 2   
C P E weight-multiplier = 4   
C P E alex_total = alex_weight + weight-multiplier \* grace_weight   
C P E answer = grace_weight + alex_total
$[Q_{2}]$  Mariah and grandma used 1/4 and 1/2, respectively, from 364  
yards in a skein of yarn. How many yards of yarn did they use in total?  
[Ground-Truth  $a_{2}^{*}$ ] 273.0  
 $[Predicted a_{21}]$  273.0  
 $[R_{21}]$  in Python  
\[
\begin{aligned}
& \text{C P E Yards_per_skein} = 364 \\
& \text{C P E M ariah_yards} = 1 / 4 * \text{yards_per_skein} \\
& \text{C P E M grandma_yards} = 1 / 2 * \text{yards_per_skein} \\
& \text{C P E M answer} = \text{mariah_yards} + \text{grandma_yards}
\end{aligned}
\]  
\[
\begin{aligned}
& \text{[Predicted a}_{22}\text{]} 273.0 \\
& \text{[R}_{22}\text{]} \text{in Python}
\end{aligned}
\]  
\[
\begin{aligned}
& \text{C P E Yarn_marah} = 1 / 4 \\
& \text{C P E Yarn_granda} = 1 / 2 \\
& \text{C P E Yards_per_skein} = 364 \\
& \text{C P E Ytotal_yards} = \text{yarn_marah} + \text{yarn_granda} \\
& \text{C P E Yards_used} = \text{total_yards} * \text{yards_per_skein}
\end{aligned}
\]
(a) Examples of self-evaluation score distribution of different predictions on the GSM8K dataset.
$[Q_3]$  Did Columbus obtain his funding from the rulers of the Portuguese Empire?  
[Ground-Truth  $a_{31}^{*}$ ] no  
[Explanation] King Ferdinand and Queen Isabella funded Columbus' voyage. TI were the joint rulers of the Iberian Peninsula, which included Spain but exclude Portugal. King John II of Portugal rejected Columbus' request for funding.  
[Predicted  $a_{21}$ ] no  
 $[R_{31}]$  in Chain-of-Thought  
C P E Columbus obtained his funding from the rulers of the Spanish Empire.  
C P E Thus, Columbus didn't obtain his funding ... Portuguese Empire.  
C P E So the answer is no.  
[Predicted  $a_{22}$ ] no  
 $[R_{32}]$  in Chain-of-Thought  
C P E Columbus was Italian and the Portuguese Empire was Portuguese.  
C P E Thus, Columbus didn't obtain funding ... Portuguese Empire.  
C P E So the answer is no.
$\left|Q_4\right|$  Is Freya a combination of Athena and Aphrodite?  
 $\left[\text{Ground-Truth } a_{4}^{*}\right]$  is yes  
 $\left[\text{Explanation}\right)$  Athena was the Greek goddess of war. Aphrodite was the Greek goddess of love. Freya was the Norse goddess of war, love, and fertility.  
 $\left[\text{Predicted } a_{41}\right]$  no.  
 $\left[R_{41}\right]$  in Chain-of-Thought  
 $\begin{array}{c|cc}C & P & \mathcal{E}\text{ Freya is a Norse goddess.}\\ C & P & \mathcal{E}\text{ Athena and Aphrodite are Greek goddesses.}\\ C & P & \mathcal{E}\text{ Thus, Freya is not a combination of Athena and Aphrodite.}\\ C & P & \mathcal{E}\text{ So the answer is no.} \end{array}$ $\left[\text{Predicted } a_{42}\right]$  yes  
 $\left[R_{42}\right]$  in Chain-of-Thought  
 $\begin{array}{c|cc}C & P & \mathcal{E}\text{ Freya is the goddess of love, beauty and fertility.}\\ C & P & \mathcal{E}\text{ You can compare these to the Greek gods Athena and Aphrodite.}\\ C & P & \mathcal{E}\text{ It is not clear how their powers exactly compare.}\\ C & P & \mathcal{E}\text{ However, it seems that Freya must have attributes that match both.}\\ C & P & \mathcal{E}\text{ So the answer is yes.} \end{array}$
(b) Examples of self-evaluation score distribution of different predictions on the StrategyQA dataset. We also provide explanations corresponding to the ground-truth answers for reference.
Figure 7: Comparisons among predictions of high and low self-evaluation scores on arithmetic (7a for GSM8K) and commonsense (7b for StrategyQA) reasoning tasks. Scores from low to high are visualized from orange (0.0), yellow (0.4), to green (1.0). Here  $\mathcal{C}, \mathcal{P}$ , and  $\mathcal{E}$  represent the evaluation confidence, the generation confidence, and their combination as the final score, respectively.
from the improved coverage of the plausible generations and the ensembling effect. Nevertheless, one can adjust the sampling-related parameters (i.e.,  $\tau$  and  $\alpha$ ) to incorporate more randomness into the generations. In practice, we find that a moderate temperature decay (e.g.,  $\alpha = 0.5$ ) results in improved performance. We conduct further analysis of the effect of sampling diversity in Appendix A.2.
Qualitative Analysis. We examine particular instances to investigate the behavior of correctness confidence scores  $\mathcal{C}$  and generation probabilities  $\mathcal{P}$  in different scenarios. From the comparison shown in Figure 7, we have the following main observations:
- In general, the correctness confidence is more effective at identifying logical errors, taking into account the accumulated mistakes from prior steps, while the generation probability focuses more on text perplexity as the confidence of the generation LLM.
- When comparing arithmetic and commonsense tasks, LLMs exhibit greater confidence in dealing with structured and objective reasoning chains such as problems in GSM8K, for both generation and self-evaluation, as opposed to reasoning chains in StrategyQA.
- Reasoning chains that appear logically plausible can achieve high correctness confidence scores but still result in incorrect answers, as demonstrated in  $R_{41}$  in Figure 7b. Moreover, the correctness confidence can be influenced by minor details (e.g., imperfect variable naming in PAL reasoning) and assign low scores regardless of the correctness of the final answers as shown in  $R_{22}$  in Figure 7a.
- Incoherence due to a sudden jump in reasoning (e.g.,  $R_{32}$  in Figure 7b) can lead to low correctness confidence. Additionally, the correctness confidence tends to be lower when the generation LLM makes a probability statement with less certainty, such as "it seems" as illustrated by  $R_{42}$  in Figure 7b.
Reasoning Formulation. Several studies have attempted to better formulate the reasoning problem. One approach is to generate rationales to enhance model interpretability (Zhou et al., 2020; Wegreffer and Marasovic, 2021; Wegreffer et al., 2021). Recently, the focus has shifted towards decomposing the reasoning process into intermediate steps before reaching the final answer (Wei et al., 2022b; Zhou et al., 2023; Gao et al., 2023; Chen et al., 2022). Various decomposition techniques have been explored, such as question reduction (Zhou et al., 2023; Yang et al., 2022), iterative prompting (Wang et al., 2022), and chaining the steps (Wu et al., 2022). While incorporating intermediate reasoning steps has resulted in substantial performance improvements, errors or imperfections can accumulate, especially when the chains become longer (Wu et al., 2016; Guo et al., 2018). As such, we utilize LLM self-evaluation as a stepwise criterion to improve the chaining process.
LLM Self-Evaluation. Recent research on LLM calibration shows that current LLMs' probabilistic predictions correspond well with actual token occurrence frequencies, leading to well-calibrated predictions for specific tasks (Rae et al., 2021; Kadavath et al., 2022; Guo et al., 2017; Kadavath et al., 2022; Jiang et al., 2021; Kuhn et al., 2023). Notably, scaling model size plays a crucial role in enhancing calibration (Rae et al., 2021; Wei et al., 2022a). As LLMs exhibit good calibration, an increasing number of studies focus on prompting LLMs to perform self-evaluation as a means of verification (Zhang et al., 2023; Shinn et al., 2023; Madaan et al., 2023; Paul et al., 2023). Self-evaluation provides an effective and efficient assessment method without requiring task-specific verifier fine-tuning, which typically involves additional annotations (Li et al., 2022). In contrast to existing works that refine generation results through instance-level self-evaluation, our approach applies self-evaluation results as a stepwise criterion to calibrate generation at a finer granularity. By focusing on step-by-step self-evaluation, our method enables fine-grained guided decoding, addressing the challenges associated with complex or lengthy reasoning.
Decoding Strategies. A tradeoff typically exists between diversity and quality. Deterministic decoding methods such as greedy decoding and beam search (Jurafsky and Martin, 2009; Graves, 2012) often produce high-quality results but lack diversity (Stahlberg and Byrne, 2019; Meister et al., 2020). Temperature sampling (Ackley et al., 1985), top- $k$  sampling (Fan et al., 2018), and top- $p$  sampling (Holtzman et al., 2020) are various techniques used to enhance diversity. The recent work of tree-of-thought (Yao et al., 2023) explores different search algorithms such as breadth-first and depth-first searches tailored for different tasks. Differently, we propose a unified framework of stochastic beam search (Caccia et al., 2020; Kool et al., 2019; Meister et al., 2021), which combines beam search and temperature sampling to balance the quality-diversity trade-off in multi-step reasoning.
We have introduced a multi-step decoding method that calibrates reasoning with stepwise self-evaluation guidance via stochastic beam search for current large language models. The empirical success of our method across a broad range of tasks, from arithmetic and symbolic to commonsense reasoning, demonstrates its robustness and generalizability in various application areas. The significant performance gains of our method on long reasoning chains also highlight its applicability to other multi-step tasks, such as multi-hop question answering and more complex scenarios involving multi-modal understanding, reasoning, and planning. In future work, we will investigate how to utilize external tools to further enhance the calibration and explore its generalizability on other multi-step scenarios to deal with more complex information such as external knowledge and multimodilities.
Potential Impacts and Limitations
We propose self-evaluation guided stochastic beam search to facilitate multi-step reasoning. However, our approach, based on stepwise self-evaluation guidance, has certain limitations. It requires access to LLM logits to calculate the self-evaluation score, restricting its applicability to more powerful LLMs, such as GPT-4, which do not provide token likelihoods. Plus, multi-step decoding inherently causes additional costs from candidate sampling and self-evaluation. For optimal balance between efficiency and cost, our approach is best applied to longer reasoning chains, where the cumulative effect of calibration across multiple steps can improve the overall performance more significantly.
The computational work for this article was partially performed on resources of the National Supercomputing Centre (NSCC), Singapore<sup>7</sup>. We would like to thank Prof. Hwee Tou Ng for his insightful discussions that enhanced the depth and quality of our study.
<sup>7</sup>https://www.nssc.sg/
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang. 2020. Towards interpretable natural language understanding with explanations as latent variables. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
A.1 Theoretical Analysis of Eq. 4
In Eq. 4, we use  $S$  sampled from the language model  $\mathrm{LM}_{\mathcal{G}}$  generations. This is an approximation for sampling from the infinite set of all possible chaining paths. And the finite set  $S$  is constructed based on the generation LM  $P_{\mathrm{LM}_{\mathcal{G}}}$ , which is different from our target distribution as shown in Eq. 4.
Specifically, denote the infinite set of all possible generated completions till the  $t$ -th step as  $\mathcal{S}^*$ , we approximate sampling from  $P_{beam}^{*}(s^{1:t}) = \frac{\exp(\mathcal{E}(s^{1:t}) / \tau)}{\sum_{s^{1:t} \in \mathcal{S}^{*}} \exp(\mathcal{E}(s^{1:t}) / \tau)}$  via  $P_{beam}(s^{1:t}) = \frac{\exp(\mathcal{E}(s^{1:t}) / \tau)}{\sum_{s^{1:t} \in \mathcal{S}} \exp(\mathcal{E}(s^{1:t}) / \tau)}$ , where  $\mathcal{S}$  is the approximation of  $\mathcal{S}^*$  with  $|\mathcal{S}| = nk = M \leq |\mathcal{S}^*|$ .
Define the upper bound  $\bar{c}$  and the lower bound  $\underline{c}$  on each  $\exp (\mathcal{E}(s^{1:t}) / \tau)$  as  $\bar{c} \geq \exp (\mathcal{E}(s^{1:t}) / \tau) \geq \underline{c}$  for all  $s^{1:t} \in S^*$ . Define the ratio as  $r = \bar{c} / \underline{c}$ . Note that  $\underline{c} \geq 1$  since  $\mathcal{E}(s^{1:t}) / \tau \geq 0$ . Thus, we can take  $r \leq \bar{c}$ .
We now give the following proposition which shows that  $|P_{beam}^{*}(s^{1:t}) - P_{beam}(s^{1:t})|$  decreases at the rate of  $\mathcal{O}\left(\frac{1 - M / |\mathcal{S}^{*}|}{M}\right)$  toward 0 as  $M$  increases. Note that as  $M$  increases toward  $|\mathcal{S}^{*}|$ , the numerator  $1 - M / |\mathcal{S}^{*}|$  decreases toward 0 while the factor for the denominator  $\frac{1}{M}$  also decreases.
Proposition 1. For any  $s^{1:t}$ , the difference between  $P_{beam}^* (s^{1:t})$  and  $P_{beam}(s^{1:t})$  is bounded by
$$
\left| P _ {b e a m} ^ {*} \left(s ^ {1: t}\right) - P _ {b e a m} \left(s ^ {1: t}\right) \right| \leq r ^ {2} \left(\frac {1 - M / \left| S ^ {*} \right|}{M}\right) \tag {6}
$$
Proof. We now prove the second statement by analyzing the absolute difference:
$$
\begin{array}{l} \left| P _ {\text {b e a m}} ^ {*} \left(s ^ {1: t}\right) - P _ {\text {b e a m}} \left(s ^ {1: t}\right) \right| (7) \\ = \left| \frac {\exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right)}{\sum_ {s ^ {1 : t} \in \mathcal {S} ^ {*}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right)} - \frac {\exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right)}{\sum_ {s ^ {1 : t} \in \mathcal {S}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right)} \right| (8) \\ = \frac {\exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right) \left| \sum_ {s ^ {1 : t} \in \mathcal {S} ^ {*}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right) - \sum_ {s ^ {1 : t} \in \mathcal {S}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right) \right|}{\left(\sum_ {s ^ {1 : t} \in \mathcal {S}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right) \sum_ {s ^ {1 : t} \in \mathcal {S} ^ {*}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right)\right)} (9) \\ = \frac {\exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right) \left| \sum_ {s ^ {1 : t} \in \mathcal {S} ^ {*} \backslash \mathcal {S}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right) \right|}{\left(\sum_ {s ^ {1 : t} \in \mathcal {S}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right)\right) \sum_ {s ^ {1 : t} \in \mathcal {S} ^ {*}} \exp \left(\mathcal {E} \left(s ^ {1 : t}\right) / \tau\right)} (10) \\ \end{array}
$$
Since  $\exp (\mathcal{E}(s^{1:t}) / \tau)$  is nonnegative, using the upper bound on each  $\exp (\mathcal{E}(s^{1:t}) / \tau)$ , we have:
$$
\left| P _ {b e a m} ^ {*} \left(s ^ {1: t}\right) - P _ {b e a m} \left(s ^ {1: t}\right) \right| \leq \frac {\bar {c} ^ {2} \left(\left| \mathcal {S} ^ {*} \right| - M\right)}{\left(\sum_ {s ^ {1: t} \in \mathcal {S}} \exp \left(\mathcal {E} \left(s ^ {1: t}\right) / \tau\right)\right) \sum_ {s ^ {1: t} \in \mathcal {S} ^ {*}} \exp \left(\mathcal {E} \left(s ^ {1: t}\right) / \tau\right)} \tag {11}
$$
Similarly, using the lower bound on each exp  $(\mathcal{E}(s^{1:t}) / \tau)$
$$
\left| P _ {\text {b e a m}} ^ {*} \left(s ^ {1: t}\right) - P _ {\text {b e a m}} \left(s ^ {1: t}\right) \right| \leq \frac {\bar {c} ^ {2} \left(\left| \mathcal {S} ^ {*} \right| - M\right)}{\underline {{c}} ^ {2} \left| \mathcal {S} ^ {*} \right| M} = r ^ {2} \left(\frac {1 - M / \left| \mathcal {S} ^ {*} \right|}{M}\right) \tag {12}
$$
A.2 Extended Experiments
Figure 8: Accuracy curves with different sampling diversity. The two plots show the changes in performance on GSM8K when the sampling temperature  $\tau$  and its decay ratio  $\alpha$  vary, respectively.
Sampling Diversity. In accordance with Figure 6b, we observe similar results when ablating the sampling hyperparameters  $\tau$  and  $\alpha$  for the single reasoning chain case, as shown in Figure 8. Increasing  $\tau$  and  $\alpha$  generally adds more diversity to the decoding process, but excessive randomness negatively impacts the performance of the single-chain decoding. Generally, a moderate temperature decay results in improved performance. Therefore, we set  $\alpha = 0.5$  throughout our experiments for simplicity and only tune  $\tau$  for randomness control.
(a)  $\lambda$ -AUC curves of  $\mathcal{E}_{\lambda}$  on GSM8K (PAL).
(b)  $\lambda$ -AUC curves of  $\mathcal{E}_{\lambda}$  on StrategyQA (CoT).
Figure 9: The change of AUC scores with different values of  $\lambda$  in  $\mathcal{E}_{\lambda}$ . We calculate the AUC score as how  $\mathcal{E}_{\lambda}$  can successfully determine whether the corresponding predicted reasoning chain can produce the ground-truth answer. The predictions here are from the baseline methods (i.e., CoT & PAL) with different LM temperatures  $\gamma$ , as represented by curves of different colors.
(a) GSM8K (PAL prompting)
Figure 10: Comparison between predictions of high v.s. low self-evaluation scores on instance-level accuracy.
(b) StrategyQA (CoT prompting)
More Analysis on Self-Evaluation. Recall that we use a combination of generation confidence and faithfulness score as  $\mathcal{E}_{\lambda} = \mathcal{C}^{\lambda}\cdot \mathcal{P}^{(1 - \lambda)}$ , with  $\lambda \in [0,1]$ . In our experiments, we set  $\lambda = 0.5$  for all tasks for simplicity. However, we investigate its effects here since, intuitively, it is an important hyperparameter for distinguishing correct / incorrect predictions and might require different values for various reasoning tasks and datasets. Its effect is also coupled with the language model temperature  $\gamma$ .
Figure 9 demonstrates how  $\lambda$  functions on arithmetic (GSM8K) and commonsense (StrategyQA). In general, we observe that the performance remains relatively stable with different choices of  $\lambda$  on different datasets, although fine-tuning this hyperparameter might lead to further improvements. This stability suggests that the choice of  $\lambda$  is not overly sensitive across various reasoning tasks and datasets, but exploring its optimal value for specific tasks could potentially lead to even better performances.