Reasoning in Flux: Enhancing Large Language Models Reasoning through Uncertainty-aware Adaptive Guidance (2024)

Abstract: Machine reasoning, which involves solving complex problems through step-by-step deduction and analysis, is a crucial indicator of the capabilities of Large Language Models (LLMs). However, as the complexity of tasks escalates, LLMs often encounter increasing errors in their multi-step reasoning process. This study delves into the underlying factors contributing to these reasoning errors and seeks to leverage uncertainty to refine them. Specifically, we introduce Uncertainty-aware Adaptive Guidance (UAG), a novel approach for guiding LLM reasoning onto an accurate and reliable trajectory. UAG first identifies and evaluates uncertainty signals within each step of the reasoning chain. Upon detecting a significant increase in uncertainty, UAG intervenes by retracting to a previously reliable state and then introduces certified reasoning clues for refinement. By dynamically adjusting the reasoning process, UAG offers a plug-and-play solution for improving LLMs’ performance in complex reasoning. Extensive experiments across various reasoning tasks demonstrate that UAG not only enhances the reasoning abilities of LLMs but also consistently outperforms several strong baselines with minimal computational overhead. Further analysis reveals that UAG is notably effective in identifying and diminishing reasoning errors.

heading: 1 Introduction
text: The impressive advancements of Large Language Models (LLMs) have recently brought about a new era in machine reasoning (Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2023; Jiang et al., 2024; Sun et al., 2024c, inter alia). For challenging scenarios, decomposing a problem into a series of intermediate steps has been shown to significantly improve the performance of LLMs (Cobbe et al., 2021; Yu et al., 2023; Sun et al., 2024a).
† Corresponding Authors
However, as the difficulty of tasks escalates, the reasoning chains inevitably become more complex and lengthy (Zhang et al., 2023b). This poses a challenge for LLMs in managing the accumulation of errors across multiple intermediate steps (Chen et al., 2022; Chu et al., 2023).
To mitigate the aforementioned issues, existing studies have focused on addressing the challenges from the perspective of alleviating uncertainty. For example, self-consistency decoding (Wang et al., 2023) samples multiple reasoning chains and employs a majority voting mechanism to mitigate the inherent randomness. Tree-of-thought (Yao et al., 2023) enables the exploration and evaluation of coherent thought units that serve as intermediate steps in problem-solving. Furthermore, Xie et al. (2023) conceptualize reasoning as a beam search, incorporating step-wise evaluation in the decoding process. While these methods have shown promise in enhancing reasoning performance, their manipulation of the reasoning process is conducted after the gen-
2401
eration of individual intermediate steps, lacking the ability to make fine-grained, flexible adjustments with each step to steer the model effectively. This oversight has long been neglected in the study of reasoning chains, yet it unveils a new pathway to improve the reasoning capabilities of LLMs.
Delving into individual reasoning steps, our observation reveals that the model can exhibit signs of uncertainty when faced with potential errors spontaneously. Figure 1 illustrates two distinct reasoning chains generated by LLaMA-2 (Touvron et al., 2023b), where a gradual shift towards red hues indicates an increase in the model’s uncertainty. This increase in uncertainty is particularly notable when the model is in the midst of an incorrect reasoning step. For instance, the model erroneously interprets that 13 eggs were “baked” instead of “left” in one flaw, which leads to a cascade of subsequent errors.
Inspired by this, we propose a novel approach: UAG, which integrates clues throughout the reasoning process based on the model’s fine-grained uncertainty. UAG first guides the model in an autoregressive manner to conduct reasoning. Upon detecting a significant rise in uncertainty within a step, the erroneous step will be removed, and targeted reasoning clues will be incorporated. Prior research (Cao, 2023; Diao et al., 2023) indicate that reasoning exemplars imbued with rich clues can substantially assist models in completing complex reasoning tasks. From a Bayesian perspective, we infer the necessity of selecting examples based on their relevance and originality. Furthermore, we cluster these example samples to minimize the search space and reduce computational costs. Our empirical studies on various datasets, including mathematical, commonsense, and symbolic reasoning, demonstrate that UAG significantly enhances model performance on complex reasoning tasks. Moreover, our method is plug-and-play, applicable to various open-source LLMs such as LLaMA (Touvron et al., 2023a) and Mistral (Jiang et al., 2023).
Our primary contributions are as follows:
• We conduct a pioneering study that explores the underlying causes of errors in LLM reasoning, focusing on the role of uncertainty.
• We introduce the Uncertainty-aware Adaptive Guidance (UAG), a novel technique that leverages model uncertainty to evaluate and enhance the reliability of each reasoning step.
• Our experimental evaluations, conducted across a diverse range of reasoning tasks,
show that UAG significantly outperforms a series of strong baselines.

heading: 2 Related Work

heading: 2.1 Demonstration Guidance.
text: Recent advancements have emphasized the significance of effective exemplar selection to guide model reasoning (Zhang et al., 2022; Shum et al., 2023; Paranjape et al., 2023; Su et al., 2023). Pioneering this field, Zhang et al. (2023c) introduce AutoCoT, a method that automates the creation of exemplars by sampling a diverse array of problems and autonomously generating corresponding reasoning chains (Wei et al., 2022; Kojima et al., 2022). This approach eliminates the need for manually crafting task-specific examples. Additionally, some strategies leverage the intrinsic knowledge of LLMs to enhance the accuracy and factuality of reasoning processes through exemplar extraction (Wang et al., 2024).
In parallel, Diao et al. (2023) investigate the application of active learning for selecting informative exemplars, which utilizes the model’s inherent uncertainties. Ye and Durrett (2023) focus on assessing the validity of reasoning chains within exemplars by evaluating their log-likelihood and performance accuracy on novel instances. Further extending the utility of LLMs in reasoning, Li and Qiu (2023) introduce memory-of-thought, a novel approach that retrieves pre-established, highconfidence thought processes to aid in current reasoning tasks. The potential for LLMs to utilize their self-generated reasoning chains for continuous self-improvement has been demonstrated in recent studies (Huang et al., 2023; Zheng et al., 2023; Lu et al., 2024; Madaan et al., 2023).

heading: 2.2 Decomposition and Validation.
text: Zhou et al. (2023) address the challenges that LLMs encounter in complex reasoning tasks by advocating for the decomposition of these tasks into simpler sub-questions. This concept aligns with the modular decomposition strategy introduced by Khot et al. (2023), which aims to optimize the handling of individual subtasks effectively. Building upon these foundations, Yao et al. (2023) innovate further by integrating a verification process within the decomposition framework. They conceptualize reasoning as a tree search, allowing LLMs to traverse various decision branches and perform both forward and backward exploration at each node.
Similarly, Besta et al. (2024) propose modeling the reasoning process as a graph structure, which offers a more dynamic framework for synthesizing LLM thoughts. Further advancing evaluation techniques, Yin et al. (2024) introduce a two-stage evaluation framework that incorporates local scoring and global evaluation to enhance LLM decisionmaking. Recent developments also include collaborative efforts, where problems are distributed among multiple LLMs for resolution (Yin et al., 2023a). Despite the innovative nature of these methods, they inherently increase computational demands, a consequence of the extensive decomposition (Hao et al., 2023; Zhang et al., 2023a; Han et al., 2023; Zhang et al., 2024), exploration (Liu et al., 2023), and verification (Sel et al., 2023).

heading: 2.3 Decoding Enhancement.
text: In the realm of decoding strategies, Wang et al. (2023) introduce self-consistency decoding, a significant shift from traditional greedy decoding. This method enhances reasoning capabilities by generating multiple reasoning paths and selecting the most consistent answer, effectively reducing the randomness associated with single-sample decoding. Complementing this, Fu et al. (2023b) advocate for guiding LLMs through more complex reasoning processes by using exemplars that feature increased reasoning complexity.
Furthering this line, Xie et al. (2023) propose a model akin to beam search that incorporates a selfevaluation mechanism to refine the decoding process. Building on this, Li et al. (2023a) introduce contrastive decoding, which helps LLMs avoid basic errors common in smaller models by utilizing model comparisons. This significantly enhances the reasoning capabilities of LLMs, as echoed in the work of O’Brien and Lewis (2023). Additionally, Chuang et al. (2023) explore intra-model dynamics by contrasting outputs from later versus earlier layers, aiming to cultivate more factually accurate reasoning within LLMs. Despite these advancements, these methodologies primarily rely on the internal representations within models (Sun et al., 2023; Stechly et al., 2023), often neglecting the integration of external reasoning cues.

heading: 3 Preliminary
text: This section outlines the foundational concepts that underpin our UAG method. Given a problem Q, a LLM generates an answer A, constructing it to-
ken by token through a probabilistic approach as described below:
P (A∣Q) = ∣A∣∏ i=1
PM(ai∣Q, a<i), (1) where PM(ai∣Q, a<i) represents the probability of generating the i-th token of the answer, given the problem Q and the sequence of previously generated tokens a<i.
CoT (Wei et al., 2022) involves supplementing the problem Q with several demonstrations D that include detailed reasoning processes. This methodology guides the model to first generate the rationale R, followed by the answer A.
P (R,A∣D,Q) = P (A∣D,Q,R)P (R∣D,Q), (2)
Applying Bayesian theorem (Bayes and Price, 1763) allows us to further refine our understanding:
P (R,A∣D,Q) = P (D∣Q,R,A)P (R,A∣Q)P (Q) P (D,Q)
= P (D∣Q,R,A)P (R,A∣Q) P (D∣Q) ,
(3)
where a low P (R,A∣Q) indicates the model’s difficulty in generating the desired rationale R and answer A without additional context. Our goal is to enhance the probability of accurately generating both the rationale and the answer by improving P (R,A∣D,Q). According to Bayes’ Theorem, this requires increasing P (D∣Q,R,A) while decreasing P (D∣Q).
Given that the answer A typically comprises fewer tokens than the rationale R, our primary focus is on optimizing the impact of the reasoning process R. To this end, we define the following criteria:
• Relevance: P (D∣Q,R,A) indicates that the reasoning within D aligns closely with our expected reasoning process, emphasizing the relevance of the exemplified reasoning. • Originality: P (D∣Q) suggests that the reasoning within D introduces novel concepts or steps unknown to the model (Yin et al., 2023b), highlighting the originality of the exemplar’s reasoning process.

heading: 4 Uncertainty-aware Adaptive Guidance
text: Uncertainty-aware Adaptive Guidance (UAG) aims to enhance the reasoning capabilities of LLMs by incorporating uncertainty awareness and adaptability. As depicted in Figure 2, our method progresses through three interconnected phases: Uncertainty Identification, Adaptive Reasoning Adjustment, and Demonstration Clustering. Each phase plays a crucial role in refining the LLM’s reasoning process, which we detail in the subsequent sections.

heading: 4.1 Uncertainty Identification
text: In complex reasoning tasks, LLMs generate reasoning chains R sequentially, one token at a time. This generative process can be formally described as follows:
P (R∣Q) = ∏ t PM(rt∣Q, r<t), (4) where P (R∣Q) represents the probability of generating the reasoning chain R given a question Q, and PM(rt∣Q, r<t) denotes the probability assigned by the model M to the t-th token, given the question and the preceding tokens r<t.
A significant challenge in LLM reasoning is the unreliability and inaccuracy of generated reasoning chains, often resulting from cumulative errors in specific reasoning steps (Xie et al., 2023; Zhang et al., 2023b; Wang et al., 2024; Hao et al., 2023).
The uncertainty during the decoding process typically reflects the model’s confidence level or lack thereof (Manakul et al., 2023; Iter et al., 2023).
We define the uncertainty of generating the t-th token as: H(rt) = − logP (rt∣r<t), (5) where p(rt∣r<t) is the probability that the LLM assigns to the t-th token, given the previous tokens. To assess changes in uncertainty, we utilize the following difference function: ∆H(rt) = H(rt) −H(rt−1), (6) where ∆H(rt) quantifies the uncertainty gap for the t-th token relative to the t−1-th token. This metric is crucial for highlighting fluctuations in model confidence between decoding steps. Specifically, a positive ∆H(rt) indicates rising uncertainty, signaling challenging reasoning steps or potential mistakes. Conversely, a negative value indicates increasing reliability in the sequence.
If the increase in ∆H(rt) exceeds a predefined threshold θ, formally expressed as: if ∆H(rt) > θ, (7) this condition suggests a significant rise in uncertainty, indicating a potential need for intervention, such as introducing additional context or implementing a corrective mechanism to steer the reasoning chain toward a more reliable trajectory.

heading: 4.2 Adaptive Reasoning Adjustment
text: When faced with increased uncertainty, UAG aims to rectify errors by eliminating erroneous reasoning steps and introducing supplementary reasoning clues, as aligned with the relevance and originality criteria defined in Section 3.
For refinement, we first backtrack within the reasoning chain to the last coherent step, denoted as rm, where r≤m represents the most recently completed reasoning step (as illustrated in Figure 2). To mitigate the uncertainty in the reasoning process, guided by Eq 3, our objective is to carefully select a demonstration to serve as external insights. We aim to increase P (D∣Q,R,A) and decrease P (D∣Q), where D comprises {Qd,Rd,Ad}.
Given that the model has not yet completed its reasoning, we utilize the existing reasoning process r≤m to calculate the relevance score SR as follows:
SR = logP (D∣Q, r≤m) = logP (Qd,Rd,Ad∣Q, r≤m) (8)
A lower probability suggests higher originality; hence, we calculate the originality score through the negative log-likelihood of the probability:
SO = − logP (D∣Q) = − logP (Qd,Rd,Ad∣Q) (9)
The selection score S is then computed as the weighted average of the two scores:
S = λ1SR + λ2SO, (10)
where λ1 and λ2 are the weights assigned to the relevance and originality scores, respectively.
Following this, we rank the demonstrations according to the selection score S , ordered from highest to lowest as {S1,S2,S3, . . . }, corresponding to{D1,D2,D3, . . . }. We retain the initial reasoning process r≤m and sequentially append each Di in front of the problem Q. Our objective is to identify a Di such that, upon generating up to the next step, the uncertainty remains consistently below a predefined threshold θ: ∃Di ∶ ∀k,∆H(rm+k) ≤ θ, (11) where k is the index of the tokens generated after appending Di. This ensures that the enhanced reasoning process effectively mitigates uncertainty and increases the reliability of the reasoning chain.

heading: 4.3 Optimizing Demonstration Selection Through Clustering
text: Given the reasoning process for each question Q, it becomes necessary to retrieve demonstrations D when the uncertainty exceeds the predefined threshold θ. This could lead to a high computational overhead due to the extensive retrieval of demonstrations. To address this, we propose a clustering approach for organizing the demonstration set {D}, aimed at selecting representative exemplars efficiently.
Initially, we compute a vector representation for each demonstration Di using a sophisticated text embedding model, text-embedding-3-large. Subsequently, these vectorized demonstrations are grouped into k distinct clusters using the k-means clustering algorithm. This method efficiently groups similar reasoning processes, enhancing the efficiency of demonstration selection.
We structure each cluster Cj by sorting its demonstrations according to their proximity to the cluster’s centroid, prioritizing those closest as they are most representative of the cluster’s reasoning pattern:
Cj = [Dj1,Dj2, . . .], Cj ∈ K-Means({D}), (12)
where each cluster Cj comprises demonstrations[Dj1,Dj2, . . .] that exhibit similar reasoning traits, determined by the K-means clustering of the demonstration set {D}.
This clustering-based approach not only streamlines the demonstration selection process but also ensures that the model has access to a diverse yet concise set of reasoning examples. It effectively balances the need for a broad variety of reasoning examples with computational efficiency, ensuring that the reasoning process is both accurate and practical.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: In this section, we delineate and scrutinize the performance of our proposed UAG, utilizing a variety of LLM backbones across a series of reasoning benchmarks. To evaluate the efficacy of UAG, we primarily employ accuracy as the performance metric. Furthermore, we undertake an analysis of
https://openai.com/index/new-embedding-models-andapi-updates
report the average number of generated tokens ( #Tokens ) to compare the computational efficiency of each method.
the computational costs among different methods, quantified by the number of tokens generated during the reasoning process. Implementation details and configurations of the various approaches can be found in Appendix B.
Benchmarks. Our evaluation encompasses three distinct categories of reasoning tasks, ensuring a comprehensive analysis of UAG’s versatility and effectiveness:
• Arithmetic Reasoning: This category includes GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), SingleEq (Koncel-Kedziorski et al., 2016), AddSub (Hosseini et al., 2014), SVAMP (Patel et al., 2021), and AQUA (Ling et al., 2017), which encompass a range of arithmetic problem-solving tasks.
• Commonsense Reasoning: We employ StrategyQA (Geva et al., 2021), CommonsenseQA (CSQA; Talmor et al., 2019), BoolQ (Clark et al., 2019), and AI2 Reasoning Challenge (ARC-c;
Clark et al., 2018) to gauge the model’s ability to understand and apply commonsense knowledge.
• Symbolic Reasoning: This involves datasets derived from BigBench (bench authors, 2023; Suzgun et al., 2023), specifically Date Understanding, Penguins in a Table, Colored Objects, and Object Counting, which test the model’s skill in abstract and symbolic thought processes.
Following Zhang et al. (2023c), our experiments are conducted in a “test question only” scenario, where we lack access to correct answers and must independently construct exemplars. Detailed descriptions and statistics of these benchmarks are provided in the appendix A.
Baselines. For an intuitive and comprehensive performance comparison, we incorporate three primary categories of baselines: (1) Zero-shot CoT (Kojima et al., 2022) for reasoning without exemplars; (2) CoT (Wei et al., 2022) for exemplarguided chain-of-thought prompting; and (3) Com-
plexCoT (Fu et al., 2023b) for complexity-based prompting. Furthermore, we also employ selfconsistency decoding (Wang et al., 2023) as a strong baseline with multiple reasoning chains for comparison. In our analysis, UAG’s performance is contrasted not only with these baselines but also with a variety of exemplar selection (Zhang et al., 2023c) and decoding enhancement techniques (O’Brien and Lewis, 2023; Chuang et al., 2023) in Appendix C.4. For generation, we adhere to the few-shot exemplars of baselines and use the number of generated tokens as a metric to assess the computational cost of each method.
Backbones. We derive embeddings for clustering through text-embedding-3-large. In our evaluation, we utilize open-source models LLaMA2 (Touvron et al., 2023b) and Mistral (Jiang et al., 2023), applying various prompting techniques such as Zero-Shot CoT (ZS-CoT) (Kojima et al., 2022), CoT (Wei et al., 2022), and ComplexCoT (Fu et al., 2023b). Section 5.3 details our examination of the scalability of our approach across various model sizes, specifically using the 7B, 13B, and 70B parameter configurations of LLaMA-2. Additionally, we integrate a Mixture of Experts model, Mistral8x7B (Jiang et al., 2024) to further diversify our evaluations.

heading: 5.2 Main Results
text: Arithmetic Reasoning. In Table 1, we present the results of arithmetic reasoning tasks. Our method manifests substantial performance enhancement across most benchmarks, in both single and multiple chain scenarios. Notably, we observe absolute increments in accuracy of 7.81%, 2.78%, and 3.14% on GSM8K, AddSub, and AQuA benchmarks, respectively, when compared to CoT approach (Wei et al., 2022). This disparity in performance gains can be attributed to UAG’s strategy of constraining the reasoning search space by strategically eliminating uncertainty, as reflected in its superior performance across tasks. This underscores the efficacy of introducing controllable randomness in the UAG decoding process to expand the search space. A more extensive comparison of UAG against a broader spectrum of methods is detailed in Appendix C, where we delve deeper into the underlying factors contributing to UAG’s enhanced performance.

heading: Commonsense and Symbolic Reasoning. As
text: illustrated in Figure 3a and Figure 3b, UAG con-
sistently outperforms existing approaches across a variety of tasks. For instance, on the BoolQ dataset, our approach achieves an accuracy rate of 62.26%, significantly surpassing the 58.07% accuracy of the baseline method. Particularly noteworthy is our performance on the StrategyQA and BoolQ datasets, where we observe that the Zero-Shot CoT (Kojima et al., 2022) struggled in guiding the model to generate precise and accurate answers, leading to significant performance declines.
Computational Cost. Despite the significant improvements our approach yields across various benchmarks, a potential concern arises regarding the computational overhead. Upon examining Figures 3a and 3b, we note that UAG, when applied to commonsense and symbolic reasoning tasks, does not incur substantial performance costs in comparison to CoT. For instance, in the Date Understanding dataset, UAG demonstrates only a 20% increase in overhead relative to CoT, yet it achieves an enhancement of over 3% in performance. Furthermore, as illustrated in Table 1, in arithmetic reasoning, UAG’s computational demand is comparable to ComplexCoT and is even more efficient than the exemplar-free Zero-Shot CoT approach. Notably, UAG’s mechanism of refining reasoning based on finished reasoning steps ensures that the additional computational overhead is minimal.

heading: 5.3 Further Analysis
text: Performance on various models. In Figure 4, we compare the performance of UAG against other baselines across different LLMs. UAG consistently outperforms these baselines. Intriguingly, CoT does not always outperform Zero-Shot-CoT (Kojima et al., 2022), e.g., Zero-Shot-CoT outperforms CoT on Mistral-7B, and this advantage is more pronounced on Mistral-8x7B. Furthermore, an increased number of exemplars does not unconditionally enhance performance; in certain cases, such as with Mistral, better results are achieved even with-
out exemplars (Li et al., 2023b). In this scenario, Mistral is even better without exemplar. This phenomenon underscores UAG’s strategic advantage in introducing the most appropriate exemplars ondemand, which substantially contributes to the observed performance improvements of our method.

heading: Importance of Relevance and Originality. In
text: Section 3, we introduce the concepts of relevance and originality, with exemplar selection criteria detailed in Eq 10. An ablation study, as depicted in Figure 5, evaluates the impact of these factors. We modify the experiment by omitting the condition in Eq 11 and employing random selection for comparison. The results reveal that excluding originality leads to an average performance drop of 5.73%, as the model then relies solely on question relevance, often being misled by errors in analogous questions (Zhang et al., 2023c). Conversely, eliminating relevance also results in significant performance decline, rendering the exemplar selection process ineffective. A further analysis of the weights λ1 and λ2 is elaborated in Appendix C.1.
Comparison to Existing Demonstration Selection Method. In Figure 6, we conduct an analysis of our UAG method against another representative demonstration selection strategy utilizing clustering, Auto-CoT (Zhang et al., 2023c), across various reasoning datasets. This comparison, under the same experimental setting, reveals a notable performance enhancement by UAG over Auto-CoT in each dataset. These findings indicate the superior-
ity of UAG, which dynamically selects appropriate demonstrations based on uncertainty during the reasoning process, as opposed to Auto-CoT’s static pre-selection approach.

heading: 6 Conclusion
text: In this paper, we start by observing the causes behind LLM reasoning errors from the perspective of uncertainty, paving the way for nuanced and adaptive modifications in the reasoning process. Driven by this insight, we introduce the Uncertainty-aware Adaptive Guidance (UAG). UAG strategically mitigates reasoning errors by identifying and eliminating uncertainties with reasoning steps, concurrently leveraging exemplars chosen for their relevance and originality to dispel uncertainty and steer the subsequent reasoning trajectory. Comprehensive experimental evaluations across a spectrum of reasoning tasks demonstrate that UAG not only surpasses several strong baselines but also exhibits adaptability to a diverse array of models, underscoring its efficacy and wide applicability in enhancing LLM reasoning capabilities. Further analysis showcases the remarkable versatility of UAG, which can function as a plug-and-play module and efficiently identify and eliminate reasoning errors.

heading: Ethics Statement
text: The development and deployment of Large Language Models (LLMs), such as those described in this paper, necessitate careful consideration of various ethical concerns. We outline several key areas of focus: Data Privacy and Security. Our approach, involving the enhancement of LLMs for reasoning tasks, does not require the collection or processing of personal data. The prompts and methods used are devoid of personal information, aligning with privacy preservation principles. Impact on Workforce. The automation capabilities of LLMs might affect employment in certain sectors. It is important to consider the broader societal implications and support the workforce in adapting to these technological changes. Environmental Considerations. The computational resources required for training and running LLMs have environmental impacts. We advocate for the use of sustainable practices and the exploration of energy-efficient models. In conducting this research, we have adhered to ethical guidelines and ensured compliance with
the licensing requirements of the datasets used, as detailed in Table 2. Our commitment to ethical research extends beyond legal compliance, encompassing a broader responsibility to the societal implications of our work.

heading: Limitations
text: While our Uncertainty-aware Adaptive Guidance (UAG) method demonstrates significant improvements in reasoning tasks, it is important to acknowledge certain limitations that point towards areas for future development:
Applicability to Closed-Source Models. A notable constraint of our method is its limited applicability to closed-source models. Models such as ChatGPT and Claude, which do not provide access to the probability distribution of tokens, pose a challenge to the implementation of UAG. This limitation restricts the versatility of our approach, as it cannot be directly applied to these commercially closed-source models.

heading: Generalization to Broader Generative Tasks.
text: While our research has focused predominantly on reasoning tasks, the potential of leveraging uncertainty in LLMs extends to a wider spectrum of generative applications (Manakul et al., 2023), such as code generation (Sun et al., 2024b). This includes using uncertainty as a metric for evaluating model hallucination (Ji et al., 2023), generation quality, and other aspects of generative performance. However, our current scope has not encompassed these areas, and we recognize this as an opportunity for future research to expand the applicability and utility of UAG in these domains.

heading: Acknowledgement
text: We extend our gratitude to the members of the FudanNLP group for their insightful suggestions and thought-provoking discussions that greatly enhanced this work. We also sincerely appreciate the anonymous reviewers and area chairs for their constructive feedback, which was instrumental in advancing the quality of our study. This work was supported by the National Natural Science Foundation of China (No. 62236004). The computations in this research were performed using the CFFF platform of Fudan University.

heading: A Statistics and Details of Datasets
text: For our experimental analysis, we carefully selected a diverse set of 14 datasets, encompassing the domains of arithmetic reasoning, commonsense reasoning, and symbolic reasoning. In Table 2, we comprehensively detail each dataset, including its source, type of answers it contains, the count of Chain-of-Thought (CoT) (Wei et al., 2022) prompt exemplars used, the size of the test sample, and the applicable licenses.
B Implementation Details
Baseline Implementation. In our main experiments, we employ three baselines: Zero-Shot CoT (Kojima et al., 2022), CoT (Wei et al., 2022), and ComplexCoT (Fu et al., 2023b). For ZeroShot CoT, we append “Let’s think step by step” after each question. For both CoT and ComplexCoT, we adhere to their original prompting exemplars. To maintain consistency, we standardize the prompt format in ComplexCoT to match CoT, replacing “Question:” and “Answer:” with “Q:” and “A:”, respectively. The details of prompts used are listed in Table 2. While CoT and ComplexCoT have the same number of prompts, each example in ComplexCoT encompasses more intermediate steps. The experimental outcomes for CD and DoLA were sourced from Chuang et al. (2023) and O’Brien and Lewis (2023). For comparison, we use the LLaMA-1 model (Touvron et al., 2023a) and sample 20 reasoning chains for selfconsistency (Wang et al., 2023), aligning with their settings.
Generation Settings In our experimental setup, we adapt the generation temperature τ for different tasks and baseline models. For the LLaMA model, Zero-Shot-CoT shows optimal performance at lower temperatures, specifically within τ ∈[0.1, 0.5]. This contrasts with CoT and ComplexCoT, which perform better at higher temperatures
(τ > 0.5), consistent with Xie et al. (2023). For the Mistral model, a temperature control within τ ∈ [0.3, 0.7] yields more favorable results. This variance in optimal temperature settings can be attributed to differences in model architecture and the distribution of the training data.
In scenarios involving multiple reasoning paths, we set the sampling temperature to 0.7 and set the number of reasoning chains to 5 to explore the reasoning space thoroughly. A majority voting mechanism is employed post-answer generation for the final decision (Wang et al., 2023). We note that some special tokens, such as the “muffins” in Figure 1, exhibits high uncertainty, likely due to infrequent usage, but did not consider these as reasoning errors if their uncertainty exceeded the threshold θ.
For different datasets, we follow the Auto-CoT setup (Zhang et al., 2023c) with varying cluster numbers; for mathematical reasoning, k = 8, while for symbolic reasoning, k = 4 was uniform. In commonsense reasoning, the cluster count was set to 7 for CSQA and StrategyQA, and 5 for BoolQ and ARC-c. To ensure precise answer extraction and mitigate evaluation errors, as discussed in Section 5.3, an exemplar was incorporated in the initial stages for Mistral-7B, LLaMA models.
Hardware utilization included a single RTX4090 for running LLaMA-7B and Mistral-7B models, two RTX4090s for LLaMA-13B, and two A100s for LLaMA-70B and Mistral-8x7B. In the multiple reasoning chains scenario, employing multiplesampling effectively reduces the randomness associated with a single run. The UAG method is implemented using PyTorch and Transformers, with Copilot and ChatGPT assisting in code writing and debugging.
Hyper parameter. We set the respective weights of relevance and originality λ1 and λ2 to 0.5 and 0.5, respectively, and the ablation experiments for
this setting are shown in 5.3, which we analyze in more detail in C.1. In the main experiment, we set the threshold θ to 16, and we analyzed the different threshold selections in C.2. In the clustering phase, we use text-embedding-3-large to obtain the corresponding embedding. We use NLL loss to react to the uncertainty of the model and use the model’s respective tokenizer to get the number of generated tokens. Benefiting from the uncertainty assessment, we are able to first complete inference on a batch of confident samples. These samples are used as examples to bootstrap questions that are interrupted due to high uncertainty, similar to active learning, allowing our approach to be applied in test question only scenarios.

heading: C Extended Analysis

heading: C.1 Further Analysis of Relevance and Originality
text: In Figure 7, we examine the influence of relevance and originality weights on performance, utilizing the Mistral-7B model. We maintain the constraint λ1 +λ2 = 1, where λ1 varies from 0 to 1, denoting

heading: 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0
text: an increasing emphasis on relevance. Our findings indicate that an increment in λ1 corresponds with a gradual improvement in model performance, underscoring the positive role of relevance in enhancing model reasoning. However, beyond λ1 > 0.6, there is a notable decline in performance, suggesting an overreliance on correlation and the detrimental impact of reasoning errors from similar samples. The model exhibits optimal performance when λ1 = λ2 = 0.5, striking a balance between relevance and originality. This balanced setting is adopted for our subsequent experiments, reflecting its effectiveness in optimizing model performance.

heading: C.2 Threshold θ
text: In Figure 8 and Figure 9, we delve into the influence of threshold θ on the performance in the GSM8K and StrategyQA datasets. The histograms display the distribution of samples requiring exemplar introduction at various thresholds. The accompanying curves illustrate the corresponding accuracy at each threshold level. It is observed that the model’s accuracy escalates with an in-
crease in θ, peaking before a rapid decline post θ > 16. To understand this phenomenon, we examine the frequency of samples meeting the criteria of Eq 7 at varying thresholds, depicted through a histogram. This analysis reveals a decrease in the number of samples satisfying Eq 7 as θ ascends. A lower θ entails a larger subset of samples undergoing reasoning adjustment, which, given that many samples are correctly inferred initially, could inadvertently introduce errors. Conversely, a higher θ may fail to identify samples with reasoning errors, gradually like an exemplar-free Zero-Shot-CoT approach (Kojima et al., 2022) and resulting in a marked degradation in performance.

heading: C.3 Correlation between Reasoning Errors and Model Uncertainty
text: In Figure 1, we observe a correlation between reasoning errors and model uncertainty, a phenomenon previously substantiated by various studies (Yuan et al., 2021; Fu et al., 2023a; Manakul et al., 2023). To delve deeper, we analyze 100 error-containing samples from the GSM8K and
StrategyQA datasets, identified through labeled correct answers, as illustrate in Figure 10 and Figure ??. These samples are processed using both ChatGPT and our UAG method for error localization and correction. Employing the reasoning process generated by Mistral-7B, we utilized gpt-3.5-turbo-1106, following the method outlined in Xie et al. (2023). ChatGPT exhibits a substantial error identification rate, correctly pinpointing 71% and 83% of errors in the datasets and successfully amending 57% and 69%, respectively. UAG demonstrates a comparable proficiency, correctly identifying 60% and 81% of the errors and effecting corrections in 43% and 55% of the cases. Notably, UAG capitalizes on the model’s inherent uncertainty for judgment, obviating the need for reevaluation. This approach not only significantly reduces computational overhead but also remains versatile across models of varying parameter sizes.

heading: C.4 Comparison to Decoding Enhancement Method
text: In Section 2, we discuss a variety of reasoning enhancement methods. Due to experimental constraints, it was impractical to compare our approach with all notable methods. Consequently, we select two representative decoding enhancement methods for comparison: Contrastive Decoding (CD) (Li et al., 2023a) and Decoding by Contrasting Layers (DoLA) (Chuang et al., 2023). Utilizing the LLaMA-1 backbone (Touvron et al., 2023a), with baseline results sourced from Chuang et al. (2023), Figure 11 and Figure 13 illustrates that UAG consistently outperforms the DoLA method across all model sizes on both the GSM8K and StrategyQA datasets. O’Brien and Lewis (2023) highlight that meticulous hyperparameter selection can notably enhance the performance of CD methods in reasoning tasks. In Figure 14, our comparison with these carefully tuned CD results reveals that UAG not only demonstrates comparable performance but also exhibits a distinct edge in commonsense reasoning tasks. This comparison underscores UAG’s ability to achieve significant performance improvements across a broader array of tasks.

heading: Future Work
text: However, our current scope has not encompassed these areas, and we recognize this as an opportunity for future research to expand the applicability and utility of UAG in these domains