Venue: NAACL 2025

# Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for

Efficient LLM Sampling

Guangya Wan\({}^{1*}\), Yuqi Wu\({}^{2*}\), Jie Chen\({}^{2,\dagger}\), Sheng Li\({}^{1,\dagger}\)

\({}^{1}\)School of Data Science, University of Virginia

\({}^{2}\)Department of Electrical and Computer Engineering, University of Alberta

{wxr9et,shengli}@virginia.edu, {yuqi14,jc65}@ualberta.ca

These authors contributed equally to this work.Corresponding authors.Code available at: [https://github.com/wan19990901/RASC/tree/Submission_Code](https://github.com/wan19990901/RASC/tree/Submission_Code)

###### Abstract

Self-Consistency mitigates hallucinations in Large Language Models (LLMs) by sampling multiple reasoning paths, but it lacks a systematic approach to determine the optimal number of samples or select the most faithful rationale. To address this limitation, we introduce Reasoning-Aware Self-Consistency (RASC), a novel framework that enhances sampling efficiency and reasoning faithfulness by dynamically evaluating both outputs and rationales. RASC assesses the quality of reasoning and the consistency of answers for each generated sample, using these assessments to guide early stopping decisions and rationale selection.The framework employs criteria-based stopping and weighted majority voting, enabling more informed choices on when to halt sampling and which rationale to select. Our comprehensive experiments across diverse question-answering datasets demonstrate that RASC outperforms existing methods, reducing sample usage by approximately 70% while maintaining accuracy. Moreover, RASC facilitates the selection of high-fidelity rationales, thereby improving the faithfulness of LLM outputs. Our approach effectively addresses the efficiency-accuracy trade-off in LLM reasoning tasks, offering a new perspective for more nuanced, faithful, and effective utilization of LLMs in resource-constrained environments. 1

Footnote 1: footnotemark:

+
Footnote †: footnotemark:

+
Footnote †: footnotemark:

## 1 Introduction

Large Language Models (LLMs) have demonstrated remarkable complex reasoning capabilities in various domains [38, 3], particularly when employing chain-of-thought (CoT) prompting [26] which allows LLMs to generate reasoning paths (RPs) before answers. Building upon this, Wang et al. wangyang2023learning introduced Self-Consistency (SC), a decoding strategy that significantly enhances reasoning performance by sampling multiple RPs and marginalizing over these samples. This approach has substantially improved LLM's performance across various reasoning domains [38, 39]. However, SC's effectiveness is tied to the number of reasoning paths sampled, creating a critical trade-off between performance and computational cost. While the original paper show that sampling more paths (e.g., 40) generally leads to better performance, there's no systematic method to determine the optimal number of samples for any given task. This ambiguity often results in over-sampling, leading to excessive computational demands [10]. Thus, we need more efficient methods that can maintain SC's reasoning benefits while reducing its computational burden.

Recent research has introduced early-stopping mechanisms to address the computational costs of Self-Consistency. Adaptive Consistency (AC) [1] employs incremental sampling, terminating when a clear majority emerges based on a predefined probability distribution, while Early Stopping Self-Consistency (ESC) [10] segments the preset sample size into windows and stops sampling when answers within a window achieve uniformity. Although these methods effectively reduce computational demands, they fail to address two critical limitations in the original self-consistency approach: (1) they treat all samples equally for sampling decisions, basing the stopping criteria solely on the consistency of final answers, with this uniform treatment extending to the use of majority voting; (2) these approaches lack of a formal mechanism to distinguish and prioritize the most faithful and high-quality rationale among the selected samples, which can be critical in domains that require precise explanations such as healthcare and science [37, 11]

et al., 2022; Wu et al., 2024).

To address these limitations while preserving the benefits of self-consistency, we propose Reasoning-Aware Self-Consistency (RASC), a framework that enhances the original SC approach. As illustrated in Fig. 1, our method assesses both the rationales across samples and the consistency of final answers, which enables more nuanced and efficient sampling. RASC is motivated by the principle that higher-quality rationale often leads to better LLM reasoning Zelikman et al. (2022). By leveraging this insight, RASC prioritizes robust reasoning in decision-making and enables early stopping when a consistent pattern of high-quality RP and consistency of answers is observed. This approach not only improves efficiency but also allows RASC to identify the most faithful and high-quality chains of thought among selected samples. Our approach significantly outperforms existing methods across 10 datasets spanning Commonsense, Mathematical, and Symbolic reasoning, reducing sample usage by 60%-80% while maintaining or improving accuracy. Furthermore, experiments demonstrate RASC's ability to select high-fidelity chain-of-thought reasoning, as well as its robustness across various datasets, prompting strategies, and multiple base LLM models, highlighting its adaptability to different efficiency-accuracy trade-offs and application needs. This combination of efficient sampling, high-fidelity reasoning selection, and broad applicability positions RASC as a versatile and powerful tool for enhancing the reasoning capabilities of large language models. In summary, our key contributions are as follows:

* **Analysis of Sampling-Based LLM Reasoning Techniques**: We present the first systematic evaluation of current SC-based sampling methods for LLM reasoning, offering new insights into their limitations, including the disregard for reasoning quality, uniform sample weighting, and the inability to identify the most faithful rationale.
* **Reasoning-Aware Self-Consistency (RASC) Framework**: We propose a novel framework that accounts for both rationale quality and answer consistency to dynamically select the number of samples and the best rationale for sampling-based LLM reasoning .

Figure 1: Comparison between Self Consistency and Reasoning-Aware Self Consistency (RASC) methods for a reasoning problem. The top half illustrates the standard Self Consistency approach, generating multiple reasoning paths and using majority voting for the final answer. The bottom half demonstrates RASC’s improvements: it assigns scores based on the qualities of both answers and reasoning paths, effectively handling incorrect or irrelevant responses. RASC’s resampling and stop condition mechanisms optimize sampling, enhancing efficiency and maintaining accuracy. Moreover, RASC enables the selection of the most faithful rationale among generated samples, improving the quality and reliability of the reasoning process in complex tasks



* **Robust Evaluation**: We showed RASC outperforms existing techniques in efficiency and explanation quality across various settings.

## 2 Related Work

### Self-Consistency and Sampling Methods

Self-Consistency (SC) Wang et al. (2023) is a powerful decoding strategy that mitigates LLM hallucinations by sampling multiple reasoning paths and deriving final answers through majority voting. Recent studies have demonstrated SC's critical importance and versatility across different domains: Huang et al. (2024) showed that SC outperforms alternative approaches like multi-agent debate for reasoning tasks, while Chen et al. (2024) and Wang et al. (2024b) successfully extended SC to open-ended generation tasks including reasoning domains and text summarization. However, despite its effectiveness, SC incurs significant computational costs at inference time due to its requirement for multiple sampling iterations. To address this limitation, researchers have proposed several adaptive approaches: Adaptive Consistency (AC) Aggarwal et al. (2023) employs incremental sampling with a Dirichlet-based stopping criterion, while Early-Stopping Self-Consistency (ESC) Li et al. (2024) segments sampling into sequential windows and stops when answers achieve uniformity. Although these methods effectively reduce computational demands, they primarily focus on checking the consistency of answers but overlook sample quality variations and the reasoning process's importance. Our work extends these approaches by addressing such limitations, aiming to enhance both the computational efficiency and faithfulness of self-consistency techniques.

### Chain-of-Thought Prompting and Reasoning Process

Chain-of-Thought (CoT) prompting Wei et al. (2022) has significantly enhanced LLM performance through step-by-step reasoning, with variants Kojima et al. (2022); Zhou et al. (2023) further refining this approach. Recent work has sparked the idea that higher-quality intermediate reasoning paths lead to better final answers Zelikman et al. (2022); Zhang et al. (2024); Khan et al. (2024), concerns about the faithfulness of CoT-generated explanations have also emerged, particularly the risk of hallucination snowballing Zhang et al. (2024). This has prompted a research theme on how to evaluate the intermedaite reasoning paths Lighthman et al. (2024); Radhakrishnan et al. (2023); Golovneva et al. (2023) and developing methods for reasoning path correction for better reasoning Wan et al. (2024); Miao et al. (2024); Weng et al. (2023).Building on these advancements, our work introduces RASC, which adapts CoT evaluation frameworks to the self-consistency setting to dynamically enhancing the reliability and faithfulness of SC-based reasoning.

## 3 Reasoning-Aware Self-Consistency

Self-Consistency (SC) and its variants enhance question-answering accuracy in Large Language Models (LLMs) by generating multiple answers, but often operlook reasoning processes, leading to inefficient sampling. We propose Reasoning-Aware Self-Consistency (RASC), a framework that optimizes sampling efficiency while maintaining accuracy by evaluating both reasoning paths and answers. RASC introduces a sufficiency scoring function that combines reasoning quality and answer consistency features, mapped to a continuous score via an optimized classifier. The framework employs a dynamic sampling process with a buffer of high-quality samples, stopping when a predefined capacity is reached. Finally, RASC uses weighted majority voting based on the scores to select the final answer and most faithful reasoning path. This approach integrates reasoning evaluation into the self-consistency process, offering a more faithful and computationally efficient method for complex question-answering tasks in LLMs.

### Reasoning and Answer Level Features

We formalize our sufficiency scoring approach by defining a function \(F:X\rightarrow[0,1]\), where \(X\) represents the space of all possible reasoning-answer pairs. This score is a composite measure of reasoning quality and answer consistency, indicating whether it's sufficient to terminate sampling. For a given reasoning-answer pair \(x=(r,a)\in X\), we approximate \(F(x)\) through a composition of functions: \(F(x)\approx f_{\theta^{*}}(\phi(x))\). Here, \(\phi:X\rightarrow\mathbb{R}^{n}\) is a feature extraction function mapping the reasoning-answer pair to an \(n\)-dimensional real-valued vector, and \(f_{\theta^{*}}:\mathbb{R}^{n}\rightarrow[0,1]\) is an optimized classifier that outputs a continuous sufficiency score given the \(n\)-dimensional input value. To capture both reasoning quality and answer consistency, we design \(\phi\) to extract two sets of features:

**Reasoning Quality Features:** These features as 

sess the quality of the RPs generated through CoT prompting. As shown in Table 1, these include measures such as RP-Length, Num-of-Steps, Step-Relevance, and Question-Relevance. These features capture the coherence, relevance, and depth of the reasoning process. In this study, the relevance was calculated as Jaccard Similarity.

**Answer Consistency Features:** These features, including Local-Consistency and Global-Consistency as shown in Table 1, evaluate the consistency of answers in the generated samples.

The feature extraction function \(\phi\) now combines both Reasoning Quality and Answer Consistency features into a single vector: \(\phi(x_{i})=[\text{Local Consistency}(x_{i}),...,\text{Global Relevance}( x_{i})]\)

**Feature Set Applicability and Extensibility:** Our feature set provides a strong foundation for evaluating reasoning quality and answer consistency. We designed \(\phi\) with a modular architecture to allow future integration of additional features as our understanding evolves. Importantly, these features are intentionally lightweight, avoiding computationally intensive extractors to optimize the cost of SC while maintaining effectiveness.

### Sampling and Decision Process

**Sufficiency Score Computation**: We learn a scoring function \(f:\mathbb{R}^{n}\rightarrow[0,1]\) from a family of parameterized functions \(\mathcal{F}\), where each function is denoted as \(f_{\theta}\) for parameters \(\theta\). Given a dataset \(\mathcal{D}=(x_{i},y_{i})_{i=1}^{M}\), where \(x_{i}=\phi(r_{i},a_{i})\) is the feature vector of the \(i\)-th reasoning-answer pair \((r_{i},a_{i})\), and \(y_{i}\in 0,1\) indicates the correctness of the answer, we obtain the optimal parameters \(\theta^{*}\) by minimizing:

\[\theta^{*}=\arg\min_{\theta}\left(\frac{1}{M}\sum_{i=1}^{M}\mathcal{L}(f_{ \theta}(x_{i}),y_{i})\right) \tag{1}\]

Where \(\mathcal{L}\) represents the loss function (e.g., cross-entropy).

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Feature** & **Description** & **Calculation** \\ \hline
**Answer-Level Features** & & \\ \hline
**Local-Consistency** & Check if the current answer is the same as the prior answer. & \(LC=1\) if \(a_{t}=a_{t-1}\), \\ (\(a_{t}\leftrightarrow a_{t-1}\)) & \multirow{2}{*}{Check if the current answer is within the previous answers.} & otherwise \(LC=0\) \\
**Global-Consistency** & Check if the current answer is within the previous answers. & \(GC=1\) if \(a_{t}\in A\), \\ (\(a_{t}\leftrightarrow a_{1},...,a_{t-1}\)) & & \multicolumn{1}{c}{otherwise \(GC=0\)} \\
**Parsing-Error \((a_{t})\)** & Ans: error \(\rightarrow\) Parsing-Error = 1; Ans: \(2\to\) Parsing-Error = 0 & \(PE=1\) if the answer is an error, otherwise \\  & & \multicolumn{1}{c}{\(PE=0\)} \\ \hline
**Reasoning-Level Features** & & \\ \hline
**RP-Length** & Reasoning path string length. & \(\text{RP-Length}=\text{len}(RP)\) \\
**Num-of-Steps (\(r_{t}\))** & Number of steps in reasoning paths. & Num-of-Steps \(=\) \\
**Step-Relevance (\(r_{t}\))** & The coherence between the reasoning steps (number of overlapping words). & \(SR=|\frac{A|AB|}{|AB|}\) where \(A=\text{vocab in step}_{t-1}\), \(B=\text{vocab in step}_{t}\) \\
**Question-Relevance** & The similarity between the input question \(Q\) and the reasoning (\(r_{t}\leftrightarrow Q\)) & \(QR=|\frac{A|OB|}{|A:UB|}\) where \(A=\text{vocab in question}\), \\
**Error-Admitting (\(r_{t}\))** & The LLM acknowledges making mistakes during the response. & \(EA=1\) if ERROR else \(EA=0\) \\
**Local-Relevance** & The similarity between the current and previous generated (\(r_{t}\leftrightarrow r_{t-1}\)) & \(LR=|\frac{A|DB|}{|AQ|B|}\) where \(A=\text{vocab in $RP_{t-1}$}\), \\
**Global-Relevance** & The similarity between the concatenation of all previous sample (\(r_{t}\leftrightarrow r_{1},..,r_{t-1}\)) & \(RS\) and the current sample RP. & \(GR=|\frac{A|OB|}{|AQ|B|}\) where \(A=\text{vocab in $RP_{t-1}$}\), \\  & & \multicolumn{1}{c}{\(B=\text{vocab in $RP_{t}$}\)} \\ \end{tabular}
\end{table}
Table 1: Answer-Level and Reasoning-Level Features for RASC. These features were designed based on our preliminary analysis (Appendix A.8) and insights from previous literature (Jin et al., 2024; Li et al., 2024; Zhang et al., 2023; Huang et al., 2025; Hosking et al., 2024; Bang, 2023; Tu et al., 2020; Golovneva et al., 2023). They capture various aspects of reasoning quality and answer consistency. Symbols in parentheses indicate relationships between samples: *-Consistency targets exact matches, while -Relevance considers vocabulary overlap. This comprehensive set of features enables RASC to make informed decisions about sampling sufficiency. Refer to Table 20 in appendix for specific examples.

 



**Stop Condition**: For a sequence of \(K\) sampled reasoning-answer pairs \(x_{i=1}^{K}\), we compute sufficiency scores \(SS_{i}=f_{\theta}(\phi(x_{i}))\) using the optimized function \(f_{\theta}\). We maintain a buffer \(\mathcal{B}\) of high-quality pairs with sufficiency scores above a predefined threshold \(T\):

\[\mathcal{B}=(r_{i},a_{i})\mid SS_{i}\geq T \tag{2}\]

The sampling process terminates when \(|\mathcal{B}|\geq N\), where \(N\) is a predefined capacity.

**Key Trade-off Parameters**: \(N\) and \(T\) are crucial for balancing sample reduction and accuracy. \(N\) determines the number of high-quality samples required before stopping, while \(T\) sets the threshold for considering a sample as high-quality. Adjusting these parameters allows RASC to prioritize either greater sample reduction or higher accuracy.

**Weighted Sampling for Answer and Rationale**: Upon reaching the stop condition, RASC determines the final answer and reasoning path through weighted majority voting using the sufficiency scores of pairs in \(\mathcal{B}\):

\[\mathrm{(Ans,RP)} =\left(\operatorname*{arg\,max}_{a\in\mathcal{A}}\left(\sum_{(r_{ i},a_{i})\in\mathcal{B},a_{i}=a}SS_{i}\right),\right.\] \[\operatorname*{arg\,max}_{r_{j};(r_{j},\text{Ans})\in\mathcal{ B}}SS_{j}\bigg{)} \tag{3}\]

where \(\mathcal{A}\) represents the set of possible answers, \(\mathcal{B}\) is the buffer of high-quality reasoning-answer pairs, and \(SS_{i}\) is the sufficiency score for the \(i\)-th pair. This approach ensures that higher-quality reasoning-answer pairs have greater influence on the final decision. The answer (Ans) with the highest weighted sum of sufficiency scores is selected, and then the most faithful reasoning path (RP) is chosen as the one with the highest sufficiency score among those supporting the selected answer. For complete details, refer to Algorithm 1 and Appendix C for theoretical justification.

```
1:Input: Query \(Q\), Threshold \(T\), Buffer Size \(N\), Scoring Function \(F\), Base LLM \(\mathcal{M}\)
2:Output: Final Answer \(A_{best}\), Best Reasoning \(\text{Apth}~{}RP_{best}\)
3:\(B\leftarrow\{\}\) #Initialize empty buffer
4:while\(|B|<N\)do
5:\((A_{i},RP_{i})\leftarrow\mathcal{M}(Q)\)
6:\(SS_{i}\gets F(A_{i},RP_{i})\)
7:if\(SS_{i}\geq T\)then
8:\(B\gets B\cup\{(A_{i},RP_{i},SS_{i})\}\)
9:endif
10:endwhile
11:\(\mathcal{A}\leftarrow\{A|(A,RP,SS)\in B\}\)
12:\(A_{best}\leftarrow\operatorname*{arg\,max}_{A\in\mathcal{A}}\sum_{(A^{\prime}, RP,SS)\in B:A^{\prime}=A}SS\)
13:\(RP_{best}\leftarrow\operatorname*{arg\,max}_{RP}\{SS|(A_{best},RP,SS)\in B\}\)
14:return\(A_{best},RP_{best}\)
```

**Algorithm 1** RASC Early Stopping

## 4 Experiments

### Experimental Setup

#### Baseline and Data:

We compare RASC against three established baseline methods: SC Wang et al. (2023), AC Aggarwal et al. (2023), and ESC Li et al. (2024b). Following the previous work, our main evaluation dataset comprises 16,725 samples spanning commonsense reasoning (Date Understanding, CommonsenseQA), symbolic reasoning (boolean expressions, disambiguation, last letters), and mathematical reasoning GSM8K, ASDIV, MathQA, SVAMP). To assess generalizability on unseen data, we use out-of-distribution datasets including BigBench (bench authors, 2023) and StrategyQA Geva et al. (2021), comprising an additional 10,824 examples that require implicit reasoning steps across various topics. See the Appendix for the configurations of the baseline methods (A.4) and the data (A.6).

Figure 2: Reasoning-Aware quality features. Different colour corresponds to different feature visualizations. Ans: Output Answer. The upper dashed box is the reasoning-level features, and the lower dashed box is the answer-level features. The blue and purple arrows represent relative relations between different samples (consistency and relevance).





**Implementation Details:** Our experiments utilize LLMA2-7B (Meta, 2024), GPT3.5-turbo/GPT4 (OpenAI, 2024), and Vicuna-13B (Chiang et al., 2023). For parameters of our algorithm, we make buffer size N being 5 with a quality threshold T of 0.5 based on experimental results in Figure 3. We use customized metrics that measure the trade-off between efficiency and accuracy to select the best hyper-parameters. We use Logistic Regression as a lightweight sufficiency scoring model, which shows a comparable correlation with human judgments and efficiency on running time. We keep the parameters learned from training set and apply it on testset to prevent overfitting. LLM inference is applied with a temperature of 0.5. See Appendix A for further details on experimental setup, including computational resources, model configurations, and prompts. Additional results, including analysis of various base models and Pearson correlation of other NLG metrics with human judgement, can be found in Appendix B.

### Main Results

**RASC Excels Across Diverse Reasoning Tasks and LLMs:** Table 2 showcases RASC's performance across a wide spectrum of reasoning tasks and base Language Models (LLMs). Our comprehensive analysis reveals that RASC consistently achieves superior sample efficiency while maintaining similar accuracy across mathematical, commonsense, and symbolic reasoning tasks. This superior performance and efficiency are statistically significant across multiple benchmarks and models, as detailed in Appendix B.2. The robustness of performance of RASC is further highlighted by its consistent effectiveness across diverse LLMs, from closed-source models like GPT-3.5 Turbo and GPT-4 to open-source alternatives such as Vicuna-13B and Llama2-7B.

**RASC Excls in Computational Efficiency:** Table 3 empirically highlights RASC's improvement in computational efficiency over other methods. While SC incurs minimum additional computational overhead, it requires the longest inference time. AC and ESC, on the other hand, introduce relatively small overhead due t

\begin{table}
\begin{tabular}{l|c c c|c c c|c c c} \hline \multirow{2}{*}{Model} & \multirow{2}{*}{Method} & \multicolumn{8}{c|}{Benchmark Category} \\ \cline{3-9}  & & \multicolumn{3}{c|}{Mathematical Reasoning} & \multicolumn{3}{c|}{Commonsense Reasoning} & \multicolumn{3}{c}{Symbolic Reasoning} \\ \cline{3-9}  & & \# of Gen & Acc & Gain/Sample & \# of Gen & Acc & Gain/Sample & \# of Gen & Acc & Gain/Sample \\ \hline \multirow{6}{*}{GPT-4} & CoT & 1 & 82.1\% & - & 1 & 83.2\% & - & 1 & 92.5\% & - \\  & SC & 40 & 87.5\% & 0.139\% & 40 & 88.0\% & 0.123\% & 40 & 97.3\% & 0.123\% \\  & ESC & 6.86 (-52.9\%) & 87.3\% & 0.88\% & 7.93 (-80.2\%) & 88.5\% & 0.767\% & 5.54 (-86.2\%) & 97.3\% & 1.056\% \\  & AC & 5.89 (-85.3\%) & 87.3\% & 1.056\% & 6.15 (-84.6\%) & 87.2\% & 0.779\% & 4.43 (-88.9\%) & 97.3\% & 1.395\% \\  & **RASC** & **4.59** (-88.5\%) & 87.5\% & **1.503\%** & **4.74** (-88.1\%) & 88.3\% & **1.367\%** & **4.19** (-89.5\%) & 97.3\% & **1.500\%** \\ \hline \multirow{6}{*}{GPT-3.5 Turbo} & CoT & 1 & 63.5\% & - & 1 & 71.2\% & - & 1 & 80.1\% & - \\  & SC & 40 & 69.1\% & 0.144\% & 40 & 16.1\% & 0.126\% & 40 & 85.3\% & 0.134\% \\  & ESC & 14.94 (-62.7\%) & 69.3\% & 0.417\% & 108.6 (-72.9\%) & 76.4\% & 0.527\% & 7.55\% (-81.1\%) & 86.0\% & 0.903\% \\  & AC & 12.49 (-68.6\%) & 67.6\% & 0.357\% & 8.34 (-79.2\%) & 75.0\% & 0.51\% & 6.39 (-84.0\%) & 84.7\% & 0.853\% \\  & **RASC** & **6.62** (-83.5\%) & 69.4\% & **1.054\%** & **4.95** (-87.6\%) & 76.1\% & **1.238\%** & **4.36** (-89.1\%) & 85.3\% & **1.547\%** \\ \hline \multirow{6}{*}{Vicuna-13B} & CoT & 1 & 36.2\% & - & 1 & 50.1\% & - & 1 & 41.5\% & - \\  & SSC & 40 & 41.7\% & 0.141\% & 40 & 55.9\% & 0.149\% & 40 & 47.3\% & 0.149\% \\  &ESC & 20.29 (-49.3\%) & 04.5\% & 0.223\% & 20.89 (-47.8\%) & 50.5\% & 0.020\% & 23.05 (-42.4\%) & 44.0\% & 0.113\% \\  & AC & 17.44 (-56.4\%) & 42.3\% & 0.371\% & 19.92 (-50.2\%) & 55.5\% & 0.306\% & 25.77 (-35.6\%) & 49.3\% & 0.314\% \\  & **RASC** & **8.24** (-79.4\%) & 42.0\% & **0.801\%** & **7.83** (-80.4\%) & 55.0\% & **0.718\%** & **8.71** (-78.2\%) & 45.3\% & **0.494\%** \\ \hline \multirow{6}{*}{Llama2-7B} & CoT & 1 & 15.5\% & - & 1 & 63.2\% & - & 1 & 19.5\% & - \\  & SC & 40 & 23.0\% & 0.115\% & 40 & 68.9\% & 0.146\% & 40 & 24.2\% & 0.121\% \\ \cline{1-1}  & ESC & 27.25 (-31.9\%) & 23.5\% & 0.190\% & 8.85 (-77.9\%) & 68.9\% & 0.726\% & 31.03 (-22.4\%) & 24.8\% & 0.177\% \\ \cline{1-1}  & AC & 23.91 (-40.2\%) & 21.5\% & 0.131\% & 7.97 (-80.1\%) & 68.3\% & 0.735\% & 24.62 (-38.5\%) & 24.2\% & 0.199\% \\ \cline{1-1}  & **RASC** & **10.71** (-73.2\%) & 23.2\% & **0.844\%** & **5.11** (-87.2\%) & 68.9\% & **1.390\%** & **1.390** (-67.3\%) & 24.8\% & **0.439\%** \\ \hline \end{tabular}
\end{table}
Table 2: Performance of Different Reasoning Methods on Mathematical, Commonsense, and Symbolic Reasoning Benchmarks, including Chain-of-Thought (CoT), Self-Consistency (SC), Early-Stopping Self-Consistency (ESC), Adaptive Consistency (AC), and Reasoning-Aware Self-Consistency (RASC). Metrics include Number of Samples Generated ( # of Gen), Accuracy (Acc), and Gained Accuracy per Sample relative to CoT.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline
**Method** & **Accuracy** & **Inference** & **Non-Inference** & **Total Time (s)** \\  & **(\%)** & **Time (s)** & **Time (s)** \\ \hline SC & 90.9 & 398.9 & 0.00 & 398.9 \\ ESC & 91.0 & 67.8 & 0.04 & 67.9 \\ AC & 90.6 & 54.9 & 0.06 & 55.0 \\
**RASC** & **91.0** & **45.1** & 2.05 & **47.2** \\ \hline \hline \end{tabular} Note: Values are averages per question over all data.

\end{table}
Table 3: Performance and Time Analysis of Methods Using GPT-4. Analysis of other models and costs can be found in Appendix B.3

calculations. RASC, although introducing the highest non-inference processing time for feature extraction, substantially reduces the overall inference time, which typically constitutes the largest portion of the total computational cost. Notably, RASC reduces inference time by approximately 80% compared to SC, while maintaining similar accuracy, and outperforming other efficient methods. We thus claim that as long as the non-inference overhead remains relatively smaller than the savings in inference time, RASC will consistently prove more computationally efficient. These gains hold across other evaluated models and is especially significant when inference speed is slower, as demonstrated in Table 14 in the Appendix.

### Analysis

**Robust Performance and Tunable Efficiency:** Fig. 3 shows RASC's performance across various threshold (\(T\)) and buffer size (\(N\)) configurations. RASC demonstrates a unique combination of performance stability and customizable efficiency. Accuracy improves consistently as \(T\) increases from 0.1 to 0.5, with the most significant gains between \(T=0.1\) and \(T=0.3\). Increasing \(N\) from 3 to 7 generally enhances accuracy, albeit with diminishing returns. This stability across hyperparameter values showcases RASC's robustness. RASC allows preference-based tuning: lower \(T\) and \(N\) values prioritize efficiency with minimal accuracy loss, ideal for resource-constrained scenarios, while higher values maximize accuracy at increased computational cost. This dual characteristic of robustness and adaptability enables RASC to optimize for specific accuracy or efficiency requirements across various computational environments.

**Performance Analysis on Tasks with Varying

Figure 3: Effects of varying \(N\) and \(T\) on performance tradeoffs, illustrating how changes in these parameters impact both accuracy (RASC Correctness) and the average number of samples generated.

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline Feature Set & \multicolumn{2}{c|}{Mathematical Reasoning} & \multicolumn{2}{c|}{Symbolic Reasoning} & \multicolumn{2}{c}{CommonSense Reasoning} \\  & \# Gen & Acc (\%) & \# Gen & Acc (\%) & \# Gen & Acc (\%) \\ \hline Answer-level only & 10.86 & 55.5 & 7.75 & 58.0 & 6.65 & 74.9 \\ Reasoning-level only & 5.50 & 50.4 & 4.30 & 55.9 & 3.08 & 72.4 \\ Combined Features & **8.20** & **55.8** & **6.28** & **58.1** & **5.63** & **75.0** \\ \hline \end{tabular} Results are averaged across all datasets and models. # Gen: Average number of generated samples. Acc: Accuracy.

\end{table}
Table 4: Performance comparison of different feature sets across reasoning tasks. Answer-level features include final answer characteristics. Reasoning-level features capture the quality of the reasoning process. Combined Features incorporate both answer-level and reasoning-level features. See Table 1 for more feature details.

 



**Difficulty:** To further validate RASC's effectiveness across different difficulty levels, we conducted experiments using the MMLU benchmark, examining performance on mathematical topics of varying complexity: Elementary Mathematics, College Mathematics, and Abstract Algebra.

The results reveal two key patterns. For harder tasks (Abstract Algebra), RASC requires more samples (8.54 vs 6.15 for College Math) to achieve confident stopping, reflecting the reasoning variations in evaluating complex reasoning and result in better performance gains. However, for easier tasks (Elementary Mathematics), RASC achieves extreme sampling efficiency due to rapid recognition of correct reasoning patterns although with a slight less performance improvement. Notably, while harder tasks require more samples, they still maintain significantly reduced sampling compared to SC , demonstrating RASC's ability to balance sampling efficiency with task complexity.

**Individual Impact of Reasoning-Level and Answer-Level Features:** Our comprehensive ablation studies reveal the crucial interplay between reasoning-level and answer-level features across various reasoning tasks. Table 4 demonstrates that _integrating both feature types consistently outperforms models using either feature set alone regarding the trade-off between accuracy and number of samples generated, validating our hypothesis of their complementary nature_. While answer-level features provide a strong baseline, as established in previous work, our novel incorporation of reasoning-level features, providing additional insights form the intermediate reasoning, significantly enhances the model's capabilities in distinguishing better samples, thus reducing the number of generations needed. This synergistic combination enables a more holistic capture of the reasoning process, highlighting the importance of considering both the reasoning path and the final answer in evaluating and optimizing language model outputs.

**Selection of High-Fidelity Chains of Thought:** Table 6 demonstrates RASC's superiority over SelfConsistency (SC) in rationale selection across traditional metrics and manual evaluation, with a 0.7-point increase in Human-Eval Score on a 5-point scale. Qualitative analysis in Appendix also supports this: for a travel distance question (Table 9, Sample 1), RASC provides a clear, correct explanation (human scores: 5) while SC introduces ambiguity (scores: 2). This analysis shows _RASC's superior capability in selecting faithful and high-quality Chains of Thought (CoTs)_, consistently producing more accurate, coherent, and relevant explanations for improved problem-solving.

### Robustness and Generalizability

To check RASC's robustness to different input

\begin{table}
\begin{tabular}{l c c l} \hline \hline
**Metric** & **RASC** & **SC** & **Diff.** \\ \hline BARTScore Yuan et al. (2021) & 0.61 & 0.39 & +0.22 \\ CTC Deng et al. (2021) & 0.55 & 0.45 & +0.10 \\ BLWRT Sellam et al. (2020) & 0.58 & 0.42 & +0.16 \\ Human-Eval Scorea  & 4.7 & 4.0 & +0.70 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of RASC and SC on CoT Faithfulness. This evaluation used 200 randomly selected questions from our training set. For RASC, we picked the CoT with the highest sufficiency score, and for SC, we selected a random CoT sample that produced the correct final answer. The evaluation assessed BARTScore, CTC, BLURT, and human-evaluated faithfulness scores. For BATRScore, CTC, and BLURT, we manually created golden CoTs as reference.

\begin{table}
\begin{tabular}{l c c} \hline \hline Method & Zero-shot & Few-shot & Least-to-Most \\ \hline SC & 69.0 / 40.0 & 75.0 / 40.0 & 85.0 / 40.0 \\ ESC & 67.0 / 9.3 & 75.0 / 9.5 & 85.0 / 8.8 \\ AC & 69.0 / 8.8 & 74.0 / 8.4 & 85.0 / 7.0 \\
**RASC** & **69.0 / 5.1** & **75.0 / 5.3** & 85.0 / **5.0** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Impact of Different Prompting Strategies on Performance on 100 Random Question from GSM8K dataset, a subset data from Mathematical Reasoning, with GPT-3.5 Turbo (a/b: a = accuracy (%), b = avg. samples)

\begin{table}
\begin{tabular}{l|c c c|c c c|c c} \hline \hline  & \multicolumn{3}{c|}{Enumerancy} & \multicolumn{3}{c|}{College} & \multicolumn{3}{c}{Abrens} \\ \cline{2-9} Model & \multicolumn{3}{c|}{Madmars} & \multicolumn{3}{c|}{Multiness} & \multicolumn{3}{c|}{Algebra} \\ \hline \multirow{2}{*}{GCN} & \(\mathbf{O}\mathbf{S}\) & Acc & Consaptive & Area & Cross-Sample & Action & ContainSuptive \\ \hline C++ & 1 & 95.8 & - & 1 & 75.4 & - & 1 & 71.8 & - \\ SC & 40.7 & 97.8 & 0.011 & 80.4 & 40.2 & - & 40.0 & 0.12 \\ ERC & 43.7 & 97.6 & 0.588 & 80.31 & 85.8 & 11.14 & 22.65 & 75.8 & 0.42 \\ ASC & 38.5 & 97.3 & 0.600 & 75.66 & 83.7 & 1.32 & 10.28 & 75.7 & 0.540 \\ RASC & **38.7** & **97.3** & **69.6** & **45.7** & **83.5** & **32.6** & **88.4** & **72.53** & **66.01** \\ \hline \hline \end{tabular} Note. Categories arranged in order of increasing difficulty (Elementary \(\rightarrow\) College \(\rightarrow\) Abstract)

\end{table}
Table 5: Performance on MMLU Matn Categories (GPT-4)

prompts, we evaluated RASC's performance using three distinct CoT prompting strategies: zero-shot (Kojima et al., 2022), few-shot (Wei et al., 2022), and least-to-most (Zhou et al., 2023). As Table.7 shows, RASC consistently outperforms other methods across all prompting strategies, demonstrating higher accuracy and lower sample (5 steps on average) requirements. This consistency highlights RASC's adaptability to various ways of eliciting reasoning from language models.

**Cross-Domain Adaptability and Performance Scaling:** To evaluate RASC's generalizability beyond our training distribution for reasoning evaluations, we tested it on two external datasets, including BigBench (bench authors, 2023) and StrategyQA (Geva et al., 2021) comprising 10,824 examples requiring implicit reasoning steps across various topics. Using GPT3.5 turbo as the base model, RASC demonstrated robust performance on these unseen, complex datasets. As illustrated in Figure 6 in Appendix, _RASC achieved the highest accuracy while significantly reducing the average number of samples generated compared to other methods_. Notably, RASC outperformed SC in accuracy (0.581 vs 0.579) while using only 14.5% of the samples (5.8 vs 40.0). It also surpassed both ESC and AC in both accuracy and efficiency. This performance suggests RASC's resilience across varied data distributions, highlighting its ability to generalize effectively to novel reasoning tasks without compromising accuracy or computational efficiency.

## 5 Conclusion

Through our investigation of sampling-based LLM reasoning, we discovered that the quality of intermediate reasoning paths serves as a crucial early indicator for efficient sampling decisions. Building on this insight, we developed RASC, a framework that strategically incorporates reasoning path evaluation into the sampling process. Our approach not only achieved a 60-80% reduction in required samples while maintaining accuracy. By introducing principled methods for assessing reasoning quality and selecting the most faithful rationale from generated samples, RASC provides a systematic solution to the challenge of balancing computational efficiency with reasoning reliability in LLMs. The framework's success across diverse reasoning tasks demonstrates that focusing on the quality of intermediate reasoning, rather than just final answers, offers a promising direction for developing more efficient and trustworthy LLM reasoning systems.

## 6 Limitations

While RASC demonstrates significant improvements in efficiency and faithfulness across various reasoning tasks, it is important to acknowledge some limitations and areas for future research:

* **Feature Limitations:** While RASC significantly reduces LLM calls, its reliance on manually designed features introduces computational overhead and potential limitations. The feature extraction process adds processing time that, although generally outweighed by savings from reduced LLM calls, could impact performance in scenarios requiring rapid responses or when handling large batches of simple queries. Moreover, these hand-crafted features, while empirically effective, may not capture all aspects of high-quality reasoning across different domains. Future work could explore adaptive strategies that dynamically adjust feature computation based on runtime constraints, investigate lightweight feature approximations, and leverage unsupervised or transfer learning approaches to automatically identify more comprehensive reasoning quality features.
* **Faithfulness Evaluation:** While we have shown improvements in faithfulness of the selected chain of thought reasoning, the evaluation relies on automated metrics and limited human evaluation. A more extensive human evaluation could provide stronger evidence on our work.

## References

* Aggarwal et al. (2023) Pranjal Aggarwal, Aman Madaan, Yiming Yang, et al. 2023. Let's sample step by step: Adaptive-consistency for efficient reasoning and coding with llms. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12375-12396.
* Ahn et al. (2024a) Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024a. Large language models for mathematical reasoning: Progresses and challenges. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop_, pages 225-237, St. Julian's, Malta. Association for Computational Linguistics.

 

Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024b. Large language models for mathematical reasoning: Progresses and challenges. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics_.
* Anthropic (2024) Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Available online. Accessed: 2024-06-01.
* Bang (2023) Fu Bang. 2023. Gptcache: An open-source semantic cache for llm applications enabling faster answers and cost savings. In _Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)_, pages 212-218.
* BIG bench authors (2023) BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_.
* Chen et al. (2024) Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2024. Universal self-consistency for large language models. In _ICML 2024 Workshop on In-Context Learning_.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chadgt quality.
* Deng et al. (2021) Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7580-7605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristothe use a laptop? a question answering benchmark with implicit reasoning strategies. _Transactions of the Association for Computational Linguistics_, 9:346-361.
* Goloveva et al. (2023) Olga Goloveva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In _The Eleventh International Conference on Learning Representations_.
* Honovich et al. (2022) Or Honorich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kulkinasy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRULE: Re-evaluating factual consistency evaluation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3905-3920, Seattle, United States. Association for Computational Linguistics.
* Hosking et al. (2024) Tom Hosking, Phil Blunsom, and Max Bartolo. 2024. Human feedback is not gold standard. In _The Twelfth International Conference on Learning Representations_.
* Huang et al. (2024) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In _The Twelfth International Conference on Learning Representations_.
* Huang et al. (2025) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. _ACM Trans. Inf. Syst._, 43(2).
* Jin et al. (2024) Mingyu Jin, Qinghai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. The impact of reasoning step length on large language models. In _Findings of the Association for Computational Linguistics: ACL 2024_, pages 1830-1842, Bangko, Thailand. Association for Computational Linguistics.
* Khan et al. (2024) Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kahitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, and Tim Rocktaschel. 2024. Debating with more persuasive llms leads to more truthful answers. In _Proceedings of the International Conference on Machine Learning (ICML)_.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Li et al. (2024a) Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. 2024a. Dissecting human and LLM preferences. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1790-1811, Bangko, Thailand. Association for Computational Linguistics.
* Li et al. (2024b) Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. 2024b. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning. In _The Twelfth International Conference on Learning Representations_.
* Lightman et al. (2024) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In _The Twelfth International Conference on Learning Representations_ 

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In _Advances in Neural Information Processing Systems_.
* Meta (2024)Meta. 2024. Meetllama3. Blog post. Available: [https://llama.meta.com/llama3/](https://llama.meta.com/llama3/).
* Miao et al. (2024) Ning Miao, Yee Whye Teh, and Tom Rainforth. 2024. Selfcheck: Using LLMs to zero-shot check their own step-by-step reasoning. In _The Twelfth International Conference on Learning Representations_.
* OpenAI (2024)OpenAI. 2024. Introducing gpt-3.5 turbo. Blog post. Available: [https://platform.openai.com/docs/models/gpt-3-5-turbo](https://platform.openai.com/docs/models/gpt-3-5-turbo).
* Radhakrishnan et al. (2023) Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Komile Lukosuite, Newton Cheng, Nicholas Joseph, Nicholas Schaefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandrasekaran, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. 2023. Question decomposition improves the faithfulness of model-generated reasoning. _Preprint_, arXiv:2307.11768.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In _Proceedings of ACL_.
* Tan et al. (2023)Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family. In _International Semantic Web Conference_, pages 348-367. Springer.
* Tu et al. (2020) Zhengkai Tu, Wei Yang, Zihang Fu, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2020. Approximate nearest neighbor search and lightweight dense vector reranking in multi-stage retrieval architectures. In _Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval_, pages 97-100.
* Wan et al. (2024a) Guangya Wan, Yuqi Wu, Jie Chen, and Sheng Li. 2024a. Cot rerailer: Enhancing the reliability of large language models in complex reasoning tasks through error detection and correction. _Preprint_, arXiv:2408.13940.
* Wan et al. (2024b) Guangya Wan, Yuqi Wu, Mengxuan Hu, Zhixuan Chu, and Sheng Li. 2024b. Bridging causal discovery and large language models: A comprehensive survey of integrative approaches and future directions. _Preprint_, arXiv:2402.11068.
* Wang et al. (2024c) Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeizHui Liu, Qi Li, and Jian Li. 2024c. Prompt engineering in consistency and reliability with the evidence-based guideline for llms. _npj Digital Medicine_, 7(1):41.
* Wang et al. (2024d) Xinglin Wang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Boyuan Pan, Heda Wang, Yao Hu, and Kan Li. 2024d. Integrate the essence and eliminate the dross: Fine-grained self-consistency for free-form language generation. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11782-11794.
* Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thoughst prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Weng et al. (2023) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shenping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. Large language models are better reasoners with self-verification. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Wu et al. (2024)Yuqi Wu, Kaining Mao, Yanbo Zhang, and Jie Chen. 2024. Callm: Enhancing clinical interview analysis through data augmentation with large language models. _IEEE Journal of Biomedical and Health Informatics_, 28(12):7531-7542.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In _Advances in Neural Information Processing Systems_, volume 34, pages 27263-27277. Curran Associates, Inc.
* Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. STar: Bootstrap reasoning with reasoning. In _Advances in Neural Information Processing Systems_.
* Zhang et al. (2024a) Dan Zhang, Singing Zoobian, Ziniu Hu, Yusong Yue, Yukiao Dong, and Jie Tang. 2024a. ReST-MCTS*: LLM self-training via process reward guided tree search. In _The Thirty-eighth Annual Conference on Neural Information Processing Systems_.
* Zhang et al. (2024b) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah Smith. 2024b. How language model hallucinations can snowball. In _Proceedings of the 41st International Conference on Machine Learning (ICML)_, Honolulu, Hawaii. PMLR.
* Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the ai ocean: a survey on hallucination in large language models. _arXiv preprint arXiv:2309.01219_.
* Zhou et al. (2023) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H.

 

Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In _The Eleventh International Conference on Learning Representations_.

## Appendix A Appendix: Experimental Details

### Base Models

The language models used in our experiments have varying numbers of parameters:

* LLAMA2-7B: This model has 8 billion parameters [14].
* GPT3.5-turbo: The exact number of parameters for this model is not publicly disclosed by OpenAI [6].
* Claude-3-Haiku: The number of parameters for this model is not publicly available from Anthropic [9].
* Vicuna-13B: This model has 13 billion parameters [10].

### Computational Resources and Infrastructure

Our experiments were conducted using various models and computing resources. Below, we detail the computational budget for each model and the specifications of our computing infrastructure.

**Computational Budget**

* LLAMA2-7B: We consumed approximately 100 GPU hours for generating all the collected data, running the model, generating CoT samples, and processing the results.
* GPT3.5-turbo: Approximately $500 was spent on API calls to OpenAI, covering all experiments including data generation, CoT samples, and result processing.
* Claude-3-Haiku: The total cost for API calls to Anthropic was approximately $300.
* Vicuna-13B: We utilized approximately 150 GPU hours for experiments, including experimental running, inference on CoT samples, and data processing.

**Computing Infrastructure** Our experiments were conducted on a computing infrastructure equipped with the following hardware:

* GPU: NVIDIA GeForce RTX 3070 Ti
* CPU: 16 cores 11th Generation Intel Core i7 Processors
* RAM: 64GB DDR4
* Storage: 1TB NVMe SSD

This infrastructure provided the necessary computational power and storage capacity to efficiently run our experiments across various models and datasets.

### Model Configurations

* **Temperature**: For all LLM calls, we used a temperature setting of 0.7 to balance creativity and coherence in the generated outputs.
* **Max Tokens**: The maximum number of tokens for each response was set to 1024 for consistency across models.
* **Top-p (nucleus sampling)**: We used a top-p value of 0.95 for all model calls to ensure diverse yet relevant outputs.

### Algorithm Configurations

* **ESC Algorithm**: We used a window size of 5 for the Entropy-based Stopping Criteria (ESC) algorithm.
* **Adaptive Consistency Algorithm**: We used the default settings with stopping_criteria=BetaStoppingCriteria(0.95) and max_gens = 40.

### Used Packages

In this study, several online available Python packages are used to conduct experiments and analysis:

* NLTK: For calculating Jaccard Similarity, Ngram, tokenizer
* statistics: For computing logistic regression
* PyTorch: For using pre-trained LLM
* pandas: For data manipulation
* json: Loading and saving JSON data
* sklearn: For supervised learning models training and evaluation
* adaptive_consistency: For implementing adaptive consistency algorithm 

* Levenshtein: For computing Levenshtein distance
* transformers: For Hugging Face pre-trained model usage
* LangChain: For LLM API usage and answer parser

### Datasets

To assess the generalizability and robustness of our proposed Reasoning-Aware Self-Consistency (RASC) method, we utilize a diverse range of datasets across multiple reasoning categories. Our evaluation leverages Question Answering datasets spanning three main categories:

**Mathematical Reasoning:** GSM8K, ASDIV, MathQA, SVAMP

**Commonsense Reasoning:** Date Understanding, CommonsenseQA

**Symbolic Reasoning:** Boolean Expressions, Disambiguation, Last Letters

These datasets provide a comprehensive foundation for evaluating our method across various problem domains and task types.

**Out-of-Domain (OOD) Data:** To rigorously test the adaptability and robustness of our RASC method, we incorporate out-of-domain (OOD) data, specifically StrategyQA Geva et al. (2021) and BigBench bench authors (2023). StrategyQA is a question answering benchmark designed for multi-hop reasoning, featuring:

* 2,780 examples requiring implicit reasoning steps.
* Diverse, short questions covering a wide range of strategies.
* Annotations including reasoning step decomposition and supporting evidence.
* A novel data collection procedure to ensure challenging and diverse questions.

BigBench is a comprehensive benchmark consisting of:

* 204 tasks contributed by 450 authors across 132 institutions.
* Diverse topics including linguistics, math, common-sense reasoning, and social bias.
* Tasks designed to be beyond the capabilities of current language models.
* Evaluations across various model scales and architectures.

By using these datasets, we aim to evaluate RASC's performance on questions that require complex, multi-step reasoning strategies not explicitly stated in the question. This allows us to verify the performance of our method on unseen data that demands sophisticated reasoning capabilities, providing insights into its real-world applicability and resilience across different computational environments and language model architectures. Using GPT3.5 turbo as the base model, RASC demonstrated robust performance on these unseen, complex datasets, comprising a total of 10,824 examples requiring implicit reasoning steps across various topics.

### Data Processing

We used a custom data preprocessing pipeline implemented in Python to clean and prepare the input data for our experiments:

* For text normalization, we employed NLTK's word_tokenize and WordNetLemmatizer.
* We used the json_parser module to parse the results and extract quality features from the model outputs.
* Our preprocessing pipeline includes steps for handling missing data, removing duplicates, and standardizing text formats.

For more detailed information on our data processing techniques, please refer to our GitHub repository [https://anonymous.4open.science/r/SC_conf-2D4B/README.md](https://anonymous.4open.science/r/SC_conf-2D4B/README.md).

### Common LLM Errors and Feature Selection

#### a.8.1 Common LLM Errors

As a crucial part of the solution, a faithful reasoning path is even more important than simply getting the correct answer to the question. Therefore, it is crucial for sampling methods, such as self-consistency, to consider the content quality of the reasoning path itself. To better understand the types of errors that LLM often makes during the response, we utilize common mathematical reasoning and commonsense reasoning datasets to systematically summarize the common errors. We start with the common errors observed by Golovneva et al. (2023) and form an initial set 

of potential quality features defined in Table 20. After human evaluation and statistical analysis, the most frequent mistakes that a bad reasoning path has include inconsistency with the question, lack of step coherence (lack of a logical flow), calculation mistakes, and hallucinations. In addition, we observe that LLMs are more likely to hallucinate when the generated context gets significantly longer. In addition, we observe that the RP always leads to an incorrect answer when it 'admits' that a mistake has been made in any of the proposed steps. However, the purpose of this study is not to comprehensively analyzing common mistakes or design automated metrics, we purposely exclude features that require intensive computational cost, such as features that have to be extracted by utilizing external machine learning models (SenteceBert, Transformers, or other neural network-based feature extractors).

#### a.8.2 Answer-level Features

These features include measures such as consistency between consecutive answers (Local-Consistency), consistency with the most common previous answer (Global-Consistency), and detection of deviations from expected response formats (Parsing-Error).

#### a.8.3 Reasoning-level Features

Our novel contribution lies in introducing reasoning-level features, which capture the intrinsic qualities of each Reasoning Path (RP). These features evaluate various aspects of the RP, such as the length and number of reasoning steps (RP-Length, Num-of-Steps), the logical flow between steps (Step-Relevance), and the relevance to the input (Question-Relevance). We also include time depedent features that compare the current RP with previous ones (Local-Consistency, Global-Consistency) and detect self-acknowledged mistakes or uncertainties (Error-Admitting). Refer to Table 20 for more examples. While similar work like ROSCOE Golovneva et al. (2023) introduces sets of reasoning-level features, our approach extends this by incorporating time-dependent features that compare the current RP with previous ones, allowing us to evaluate the consistency and progression of reasoning across multiple steps

### Preliminary Feature Analysis

Table 8 presents the correlation coefficients between each feature and the Correctness variable, along with the p-values from t-tests.

Global-Consistency (0.403) and Local-Consistency (0.367) show the strongest positive correlations with Correctness, suggesting these similarity measures between the actual solution and generated content are most predictive of correctness. All features show statistically significant differences between correct and incorrect instances (p-values \(\approx 0\)). RP-Length, Error-Admitting, Question-Relevance, Num-of-Steps, and Step-Relevance have weak negative correlations, indicating that longer solutions or those with more steps are slightly less likely to be correct. Word clouds of internal mistakes (Figures 4 and 5) reveal that both LLAMA3 and GPT-3.5-TURBO models frequently acknowledge potential errors and limitations, using phrases like "There seemed to be a mistake" and "Not enough information". These findings suggest that improving the model's ability to generate solutions similar to the actual solution and encouraging concise responses could enhance performance, while the models' self-awareness of mistakes could be leveraged to improve reliability.

### Sample Generation Prompts

In this study, we explore three different prompting strategies: Chain of Thought (CoT), Few-Shot, and Least-to-Most. Each strategy aims to enhance the performance of Large Language Models (LLMs) in generating reasoning paths (RPs) for given questions.

\begin{table}
\begin{tabular}{|l|r|r|} \hline
**Feature** & **Correlation** & **P-value** \\ \hline Local-Consistency & 0.367 & 0.00 \\ Global-Consistency & 0.403 & 0.00 \\ RP-Length & -0.0861 & 0.00 \\ Num-of-Steps & -0.0476 & 0.00 \\ Step-Relevance & -0.0264 & 1.25e-169 \\ Question-Relevance & -0.0423 & 0.00 \\ Error-Admitting & -0.0492 & 0.00 \\ Local-Relevance & 0.0530 & 0.00 \\ Global-Relevance & 0.0850 & 0.00 \\ \hline \end{tabular}
\end{table}
Table 8: Correlation with Correctness and P-values (3 significant figures) 



#### a.10.1 Chain of Thought (CoT) Prompting

CoT prompting explicitly requires LLMs to reason through the question _step by step_. We utilize zero-shot prompting to generate RPs in our datasets. The prompt is defined as follows:

**System Message:**_You are a professional specialized in [subject]. You need to help me answer the given question. Notice that you need to solve the question step by step and as detailed as possible. Do not jump to the answer directly. You must follow the RP format instructions._

**Human Message:**_{question}_

#### a.10.2 Few-Shot Prompting

Few-shot prompting provides the LLM with examples of how to solve problems using step-by-step reasoning. This approach helps guide the model in generating structured and detailed responses. The prompt is structured as follows:

**System Message:**_You are a professional specialized in [subject]. Your task is to answer the given question using step-by-step reasoning. Follow the examples provided, breaking down your thought process into clear steps before providing the final answer._

**Human Message:**_Here are two examples of how to solve problems using step-by-step reasoning: [Example 1 and Example 2 are provided here, demonstrating step-by-step problem-solving] Now, please solve the following question using a similar step-by-step approach: [question]_

#### a.10.3 Least-to-Most Prompting

The Least-to-Most approach involves breaking down complex problems into simpler sub-problems, solving them in order of increasing complexity, and using those solutions to address the main question. This strategy is particularly useful for tackling intricate problems. The prompt is structured as follows:

**System Message:**_You are an expert problem solver specialized in [subject]. Your task is to break down and solve complex problems using the Least-to-Most approach. This means you'll divide the main problem into simpler sub-problems, solve them in order of increasing complexity, and use those solutions to address the main question._

**Human Message:**_Let's solve problems using the Least-to-Most approach. Here's an example: [An example demonstrating the Least-to-Most approach is provided here] Now, please use this Least-to-Most approach to solve the following problem. Break it down into simpler sub-problems, solve them in order, and then use those solutions to address the main question: [question]_

Each of these prompting strategies aims to elicit detailed, step-by-step reasoning from the LLM, enabling us to generate high-quality reasoning paths for analysis and comparison.

### Human Evaluation Criteria

**Annotator Profile:** The evaluation was conducted by a group of 10 individuals: 7 males and 3 females. The age range of the annotators was 23-30 years old, with a median age of 25. All annotators hold

Figure 4: Word Cloud of Internal Mistakes for LLAMA3

Figure 5: Word Cloud of Internal Mistakes for GPT-3.5-TURBO

technical degrees spanning mathematics, computer science, and statistics, ensuring their capability to comprehend the logic behind the generated rationales effectively. Their educational backgrounds are as follows:

* 1 Assistant Professors (1 in Computer Science)
* 6 PhD candidates (1 in Computer Science, 4 in Data Science, 1 in Statistics)
* 3 Master's students (1 in Data Science, 1 in Electrical Engineering, 1 in Computational Mathematics)

Geographically, the annotators represent two countries:

* 3 from the United States
* 7 from China

Professionally, 8 are currently in academia (1 assistant professors, 5 PhD candidates, 2 Master's students), while 2 work in industry with Master's degrees and 1-3 years of relevant work experience related to technology.

**Ethical Considerations and Consent:** Prior to participating in the evaluation process, all annotators were provided with information detailed the study's objectives, the nature of their participation, and how their data would be utilized. Specifically, annotators were informed that their ratings, demographic information, and any generated "golden" CoTs would be used for research purposes and potentially published anonymously. They were assured of the confidentiality of their personal information and the anonymization of their responses in any resulting publications. Annotators were also informed of their right to withdraw from the study at any point and request the removal of their data. The data retention policy, including the duration of data storage and the method of eventual deletion, was clearly outlined. Furthermore, annotators were provided with contact information for the research team to address any concerns or questions about their data usage.

**Evaluation Instruction:** Human evaluators were asked to rate the generated Chain of Thought (CoT) responses on two main criteria:

1. **Overall Quality**[1-5]: This criterion assesses whether the generated response answers the question in a well-justified manner. The scale is interpreted as follows: * 1: Incomprehensible and completely incorrect. The response is unintelligible or entirely unrelated to the question. * 2: Mostly incorrect with major flaws in reasoning. The response may be partially related to the question but contains significant errors or misunderstandings. * 3: Partially correct but with notable gaps or minor errors. The response addresses the question but lacks full justification or contains some inaccuracies. * 4: Mostly correct with minor imperfections. The response is well-justified and accurate, with only slight room for improvement. * 5: Clear, correct, and fully justified. The response comprehensively answers the question with sound reasoning and no errors.
2. **Coherency**[1-5]: This criterion evaluates whether the whole generated response makes sense, regardless of its correctness in addressing context. The scale is interpreted as follows: * 1: Completely incoherent. The response is a jumble of words or concepts with no logical flow or structure. * 2: Mostly incoherent with occasional comprehensible parts. The response has severe issues with logical flow, making it very difficult to follow. * 3: Partially coherent but with noticeable lapses in logic or structure. The response has a basic structure but contains confusing or disconnected elements. * 4: Mostly coherent with minor clarity issues. The response has a clear logical flow with only slight ambiguities or structural weaknesses. * 5: Perfectly coherent and easy to understand. The response has a clear, logical structure with smooth transitions between ideas.

The overall score for each evaluated response is calculated as the average of these two metrics.

In addition to manually evaluating the best CoT responses (based on our method) and a randomly chosen CoT from SC, evaluators were also tasked with creating a "golden" CoT when necessary so 

that we have a reference available to calculate the BARTScore Yuan et al. (2021). This process involved:

1. Reviewing a random CoT for human-like reasoning.
2. Assessing whether the flow of logic closely resembles what a human solver would do, considering: * Natural, step-by-step reasoning processes * Logical order of steps * Inclusion of relevant intermediate calculations or considerations * Appropriate level of detail for human-like problem-solving
3. Rewriting the CoT if it did not closely resemble human-like reasoning, ensuring: * Maintenance of the correct answer while improving the reasoning process * Structure reflecting human approach to problem-solving * Clear, concise, and easy-to-follow response This comprehensive evaluation process ensures a thorough assessment of the quality and coherency of generated CoT responses, as well as the creation of human-like golden standards when needed. See Table 9 for two examples.

### Evaluation Metric for Accuracy-Cost Trade-Off

To quantitatively assess the trade-off between accuracy and cost, we introduced a custom metric that balances both factors. This metric normalizes the accuracy and cost values and computes a weighted average to provide a single score representing the overall performance of a method.

Let acc denote the accuracy of a method, and cost denote the computational cost, measured as the number of samples generated. Additionally, let sc_acc and sc_cost represent the accuracy and cost of the Self-Consistency (SC) method, respectively. Let single_sample_acc and direct_cost denote the accuracy from using only the first sample's answer and the cost of the direct sampling method, respectively.

The metric is defined as:

\[{\text{metric}=0.5\times\text{acc\_factor}+0.5\times\text{cost\_factor}} \tag{4}\]

where acc_factor and cost_factor are normalized values of accuracy and cost, calculated as follows:

\[{\text{acc\_factor}=\begin{cases}1&\text{if acc $\geq$ sc\_acc}\\ 0&\text{if acc $\leq$ sam\_acc}\\ \frac{\text{acc\_acc\_sm\_acc}}{\text{sc\_acc\_acc\_single\_acc}}&\text{ otherwise}\end{cases}} \tag{5}\]

\[{\text{cost\_factor}=\begin{cases}1&\text{if cost $\leq$ dr\_cost}\\ 0&\text{if cost $\geq$ sc\_cost}\\ \frac{\text{sc\_cost\_cost}-\text{cost}}{\text{sc\_cost}-\text{dir\_cost}}& \text{otherwise}\end{cases}} \tag{6}\]

The acc_factor is normalized to be between 0 and 1, where 1 corresponds to an accuracy greater than or equal to the SC method, and 0 corresponds to an accuracy less than or equal to the accuracy using just the first sample's answer. Similarly, the cost_factor is normalized to be between 0 and 1, where 1 corresponds to a cost less than or equal to the direct sampling method, and 0 corresponds to a cost greater than or equal to the SC method. The metric function calculates the weighted average of acc_factor and cost_factor, giving equal importance to both accuracy and cost. A higher value of the metric indicates a better trade-off between accuracy and cost. This function enables us to compare different methods and configurations based on their ability to achieve high accuracy while minimizing computational cost.

### Reproducibility

To ensure reproducibility of our results:

* We set a fixed random seed (42) for all randomized processes.
* All experiments were run using Python 3.8.10 in a conda environment. The full list of dependencies and their versions can be found in the requirements.txt file in our repository.

## Appendix B Appendix: Additional Results

### Impact of Base Estimator:

The performance of different sufficiency scoring models, as summarized in Table 10, identifies Logistic Regression as the most effective model in 

terms of accuracy and sample utilization. This finding shows the importance of selecting an appropriate sufficiency scoring model to maximize the benefits of the RASC approach. These results are obtained by training the sufficiency scoring models on the training set and then fine-tuning the best set of hyperparameters (N and T) using the customized metrics. The final evaluation is performed on the test set to ensure the robustness of the model.

Among the sufficiency scoring models evaluated **Logistic Regression** achieves the highest accuracy of 46.0% while requiring the lowest average number of samples (5.87) compared to other models. This indicates that it effectively captures the rela

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **Accuracy (\%)** & **Num. Samples** \\ \hline Random & 41.9 & 9.77 \\ NB & 45.9 & 7.91 \\
**LR** & **46.0** & **7.87** \\ RF & 45.8 & 8.38 \\ HHEM & 42.4 & 9.92 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Performance Comparison of Sufficiency Scoring Models

\begin{table}
\begin{tabular}{p{11.3pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline Sample Question & Answer RASC Rationale & SC Rationale & Golden CoT & Ratings (Qua, Coh) \\ \cline{3-6}  & & & Best & SC \\ \hline
1 & Elvis starts driving from his house and travels east for 5 hours at 18mph, he covered 5 x 18 = 90 miles. When he traveled east for 8 hours at 18mph, he covered 8 x 18 = 144 meters. The flow far is he was driving at an average speed of 18mph for both parts of the journey, how far is he from his house now? & If Elvis drove west for 5 hours at 18mph, he he covered a distance of 8* 18 = 144 miles. Between the two distances, which is 144 of the journey, how far is he from his house now? & Elvis drove for 5 hours at 18mph, so he covered a distance of 8* 18 = 144 miles. The total distance he covered is 90 + 144 - 90 = 54 miles. The snow far is he from his house now? & Elvis drove for 5 hours at 18mph, so he covered a distance of 8* 18 = 144 miles. The total distance he covered is 90 + 144 - 90 = 54 miles. The answer is 54 miles. & Elys drove for 5 hours at 18mph, so he covered a distance of 8* 18 = 144 miles. The total distance from his house covered is 90 - 54 miles away. So, we need more information to determine the distance he is from his house.

\begin{table}
\begin{tabular}{c l l l l l} \hline \hline
**Model** & **Accuracy (\%)** & **Num. Samples** \\ \hline Random & 41.9 & 9.77 \\ NB & 45.9 & 7.91 \\
**LR** & **46.0** & **7.87** \\ RF & 45.8 & 8.38 \\ HHEM & 42.4 & 9.92 \\ \hline \hline \end{tabular}
\end{table}
Table 11: ’HHEM: Hallucination Detector Model based on microsoft/deberta-v3-base, initially trained on NLI data

\begin{table}
\begin{tabular}{p{11.3pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline Sample Question & Answer RASC Rationale & SC Rationale & Golden CoT & Ratings (Qua, Coh) \\ \cline{3-6}  & & & Best & SC \\ \hline
1 & Elvis starts driving from his house and travels west for 5 hours at 18mph, he covered 5 x 18 = 90 miles. When he travelled east for 8 hours at 18mph, he covered 8 x 18 = 144 meters. Speed of 18mph for both parts of the journey, how far is he from his house now? & Elvis drove for 5 hours at 18mph, he covered a distance of 8* 18 = 144 miles. Between the two distances, which is 144 of the journey, how far is he from his house now? & Elvis drove for 5 hours at 18mph, he covered a distance of 8* 18 = 144 miles. Between the two distances, which is 144 of the journey, how far is he from his house now? & Elvis drove for 5 hours at 18mph, he covered a distance of 8* 18 = 144 miles. The total distance he covered is 90 + 144 - 90 = 54 miles. The answer is 54 miles. & Elys drove for 5 hours at 18mph, he covered a distance of 8* 18 = 144 miles. The total distance he covered is 90 + 144 - 90 = 54 miles. The over, do not know the direction of his house. If he is now east of his house, he is 144 - 90 = 54 miles away. If he is 90 - 144 - 54 miles away. So, we need more information to determine the distance he is from his house.

 

tionship between the features extracted from the RP and the likelihood of hallucination, which allows RASC to make informed decisions during the weighted majority voting process.

The accuracy of Random model drops below the baseline of Self-Consistency. The poor performance of the Random model, which assigns sufficiency scores randomly, emphasizes the importance of using a well-designed scoring model in the RASC approach. Without a meaningful assessment of the generated content's quality and consistency, the weighted majority voting process cannot effectively distinguish between reliable and unreliable samples, leading to suboptimal results.

The HHEM Model, which is a DeBERTa-based hallucination detector sourced from Hugging Face Honovich et al. (2022), does not perform as well as the other models in this context. This suggests that relying solely on a pre-trained model without considering the specific characteristics and requirements of the RASC approach may not yield the best results. The superior performance of models like Logistic Regression (LR), Naive Bayes (NB), and Random Forest (RF), which utilize manual feature engineering tailored to the RASC framework, highlights the importance of crafting features that capture the nuances of the generated content as we discussed in the method section.

### Statistical Significance

To rigorously evaluate the performance of RASC, we conducted statistical significance experiments across multiple benchmarks using various language models. The comparison between RASC (Reasoning-Aware Self-Consistency) and SC (Self-Consistency) methods reveals interesting patterns across different reasoning types. In terms of correctness, Tables 18 show no statistically significant differences between RASC and SC across common sense, mathematical, and symbolic reasoning tasks. However, the steps comparison in Tables 19 demonstrates that RASC consistently requires significantly fewer steps than SC across all reasoning types, with p-values of 0 indicating strong statistical significance. This holds the same for AC and RASC comparison as demonstrated in Table 15 and Table 16 These results suggest that while RASC maintains comparable correctness to SC, it achieves this with substantially improved efficiency in terms of the number of steps required.

Note: SC stands for Self-Consistency, a baseline method for comparison.

### Additional Performance and Time Analysis

This section focuses on the performance and time analysis of the open-source Llama2 model.

Table 14 compares different methods using the Llama2 model. RASC achieves the highest accuracy (38.86%) while significantly reducing inference time. Compared to SC, RASC improves accuracy by 0.33 percentage points and reduces total time by 66.23

These results demonstrate RASC's effectiveness in enhancing both performance and time efficiency. While we don't provide direct cost analysis for closed-source models, the substantial reduction in inference time achieved by RASC would likely

\begin{table}
\begin{tabular}{l c} \hline
**Metric** & **Pearson Correlation** \\ \hline RASC Score & 0.68 \\ BARTScore & 0.52 \\ BLURT & 0.58 \\ CTC & 0.65 \\ \hline \end{tabular} Note: Higher correlation indicates closer alignment with human judgments. All correlations are statistically significant (p ¡ 0.01).

\end{table}
Table 13: Pearson Correlation of Different Metrics with Human Ratings

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Method** & **Accuracy** & **Inference** & **Non- Inference** & **Total Time (s)** \\  & **(\%)** & **Time (s)** & **Time (s)** & **Time (s)** \\ \hline SC & 38.7 & 560.00 & 0.01 & 560.01 \\ ESC & 39.1 & 313.32 & 0.40 & 313.72 \\ EC & 39.1 & 313.32 & 0.40 & 313.72 \\ ESC & 39.0 & **134.96** & 2.05 & **137.01** \\ \hline \hline \end{tabular} Note: All values are averages per 1 question.

\end{table}
Table 14: Performance and Time Analysis of Methods Using Llama2.

 

translate to significant cost savings in scenarios where API calls are charged based on usage time.

### Comparison with Existing NLG Metrics:

To evaluate the effectiveness of our sufficiency scoring method, we compared it against three popular automated metrics for natural language generation: BARTScore Yuan et al. (2021), BLURT Sellam et al. (2020), and CTC Deng et al. (2021). Using the same 100 samples from the faithfulness analysis, we calculated the Pearson correlation between these scores and human-annotated quality ratings.

As shown in Table 13, RASC's sufficiency scores demonstrate a higher correlation with human ratings (0.68) compared to BARTScore (0.52), BLURT (0.58), and CTC (0.65). This suggests that our scoring method more closely aligns with human judgments of CoT quality in reasoning tasks, validating its effectiveness in assessing the quality of generated reasoning paths. The substantial improvement over BARTScore and the slight edge over BLURt and CTC underscore RASC's capability to capture nuances in reasoning quality that align more closely with human evaluations.

### Test Data Results

To evaluate RASC's generalizability, Fig. 6 presents the performance comparison between RASC and other methods on this dataset.

## Appendix C Theoretical Analysis of Reasoning-Aware Self-Consistency

In this section, we provide a formal theoretical justification for the design of the Reasoning-Aware Self-Consistency (RASC) framework. Our analysis is grounded in information theory and demonstrates how RASC optimizes inference-time reasoning through a dual information gain framework that incorporates both answer consistency and reasoning path quality.

### Preliminaries and Notation

Let \(A\) denote the set of possible answers and let \(p(a)\) be the probability mass function over \(A\). The entropy of the answer distribution is given by:

\[H(A)=-\sum_{a\in A}p(a)\log p(a).\]

A higher entropy \(H(A)\) implies greater uncertainty in the answer distribution, which corresponds to lower consistency. The goal of self-consistency-based inference methods is to iteratively refine the answer distribution such that \(H(A)\) decreases over successive samples.

Now, let \(R\) denote the set of all possible reasoning paths. Given an answer \(a\in A\), we define the conditional entropy of reasoning paths given an answer as:

\[H(R|A)=-\sum_{a\in A}p(a)\sum_{r\in R}p(r|a)\log p(r|a).\]

This quantity measures the uncertainty in the reasoning paths that lead to a particular answer. Since RASC is designed to enhance the coherence and quality of reasoning paths, it aligns with the objective of reducing \(H(R|A)\) after an answer has been proposed.

\begin{table}
\begin{tabular}{l c l r r} \hline \hline
**Reasoning Type** & **Mean Difference** & **P-value** & **95\% CI** & **Significant** \\ \hline Common Sense & -3.4301 & ¡0.0001 & [-4.2164, -2.6439] & Yes \\ Mathematical & -7.4156 & ¡0.0001 & [-8.0905, -6.7408] & Yes \\ Symbolic & -7.7112 & ¡0.0001 & [-9.4171, -6.0052] & Yes \\ \hline \hline \end{tabular}
\end{table}
Table 16: Comparison of RACS and AC Methods on Steps

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Reasoning Type** & **Mean Difference** & **P-value** & **95\% CI** & **Significant** \\ \hline Common Sense & -0.0075 & 0.6697 & [-0.0565, 0.0414] & No \\ Mathematical & -0.004 & 0.745 & [-0.0384, 0.0383] & No \\ Symbolic & 0.0033 & 0.9044 & [-0.0738, 0.0805] & No \\ \hline \hline \end{tabular}
\end{table}
Table 15: Comparison of RACS and AC Methods on Correctness 



### Information Gain and Sampling Efficiency

The information gain from each sample is defined as the reduction in uncertainty of the answer distribution:

\[IG=H(A_{\text{prior}})-H(A_{\text{posterior}}).\]

For classical self-consistency methods (e.g., Self-Consistency (SC), Early Stopping Self-Consistency (ESC), Adaptive Consistency (AC)), the information gain is driven solely by answer consistency. That is, we define:

\[IG_{\text{ans}}=f(c_{i}),\]

where \(c_{i}\) is a consistency metric for the \(i\)-th sample, and \(f:\mathbb{R}\rightarrow\mathbb{R}\) is a function characterizing how consistency influences information gain. The specific form of \(f\) varies across methods: for instance, ESC applies a sliding window to assess convergence, whereas AC employs an adaptive sequence check.

In contrast, RASC incorporates reasoning path quality into the information gain model. Specifically, the information gain is given by:

\[IG_{\text{ans-RP}}=f(c_{i})+g(r_{i}),\]

where \(r_{i}\) represents the quality of the reasoning path associated with the \(i\)-th sample, and \(g:\mathbb{R}\rightarrow\mathbb{R}\) is a function capturing the contribution of reasoning coherence and correctness to overall information gain. The augmentation by \(g(r_{i})\) ensures that the refinement of both answer consistency and reasoning quality contributes to entropy reduction.

### Stopping Condition and Convergence

The sampling process in RASC is governed by an optimization criterion where sampling terminates when the cumulative information gain exceeds a predefined threshold \(T\):

\[\sum_{i=1}^{n}IG_{i}\geq T.\]

Given the dual contribution of answer consistency and reasoning coherence in RASC, the total entropy \(H(A)+H(R|A)\) decreases at a faster rate compared to answer-only self-consistency methods. That is, for a given number of samples \(n\), we have:

\[H_{\text{RASC}}(A,R|n)<H_{\text{SC}}(A|n),\]

where \(H_{\text{RASC}}(A,R|n)\) and \(H_{\text{SC}}(A|n)\) denote the total entropy under RASC and SC, respectively, after \(n\) samples. This inequality formalizes the efficiency of RASC, demonstrating that fewer samples are required to reach the stopping criterion.

### Implications and Conclusion

The theoretical analysis yields the following key conclusions:

1. **Reduction in Entropy:** Answer-level features contribute to minimizing \(H(A)\), while reasoning-level features contribute to minimize \(H(R|A)\). The combination ensures a more rapid decrease in overall uncertainty.
2. **Enhanced Information Gain:** The presence of \(g(r_{i})\) in RASC ensures that each sample contributes more effectively to uncertainty reduction, thereby accelerating convergence.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Reasoning Type** & **Mean Difference** & **P-value** & **95\% CI** & **Significant** \\ \hline Common Sense & -3.4301 & ¡0.0001 & [-4.2164, -2.6439] & Yes \\ Mathematical & -7.4156 & ¡0.0001 & [-8.0905, -6.7408] & Yes \\ Symbolic & -7.7112 & ¡0.0001 & [-9.4171, -6.0052] & Yes \\ \hline \hline \end{tabular}
\end{table}
Table 17: Comparison of RACS and AC Methods on Steps

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Reasoning Type** & **Mean Difference** & **P-value** & **95\% CI** & **Significant** \\ \hline Common Sense & -0.0075 & 0.6697 & [-0.0565, 0.0414] & No \\ Mathematical & -0.0068 & 0.5819 & [-0.0412, 0.0275] & No \\ Symbolic & 0.0067 & 0.8104 & [-0.0705, 0.0839] & No \\ \hline \hline \end{tabular}
\end{table}
Table 18: Comparison of RASC and SC Methods on Correctness

3. **Empirical Consistency:** Our results confirm that RASC requires fewer samples than ESC and AC while maintaining accuracy, implying that the expected information gain from reasoning evaluation \(g(r_{i})\) satisfies: \[\mathbb{E}[g(r_{i})]>0.\]
4. **Sufficiency Function Validity**: The sufficiency function used in RASC effectively captures reasoning quality, ensuring that the decision to stop sampling is based on a meaning

Figure 6: Performance of models on out-of-distribution datasets using GPT-3.5 Turbo, assessing method’s robustness across diverse reasoning tasks.

 

ful reduction in uncertainty.

While the current formulation of \(g(r_{i})\) has demonstrated empirical success, future research could explore optimizing its structure via feature selection or more sophisticated scoring functions.