Calibrating Reasoning in Language Models with Internal Consistency
Authors: Zhihui Xie
Jizhou Guo
Tong Yu
Shuai Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Large language models
reasoning
faithfulness
internal consistency
Primary area: natural_language_processing
TL;DR: We observe internal inconsistency in chain-of-thought reasoning and propose a new method to mitigate it for enhanced reasoning performance of language models.

Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought prompting that elicits verbalized reasoning. However, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales. In this work, we investigate reasoning in LLMs through the lens of internal representations, focusing on how these representations are influenced by generated rationales. Our preliminary analysis reveals that while generated rationales improve answer accuracy, inconsistencies emerge between the model’s internal representations in middle layers and those in final layers, potentially undermining the reliability of their reasoning processes. To address this, we propose internal consistency as a measure of the model’s confidence by examining the agreement of latent predictions decoded from intermediate layers. Extensive empirical studies across different models and datasets demonstrate that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Motivated by this, we propose a new approach to calibrate reasoning by up-weighting reasoning paths with high internal consistency, resulting in a significant boost in reasoning performance. Further analysis uncovers distinct patterns in attention and feed-forward modules across layers, providing insights into the emergence of internal inconsistency. In summary, our results demonstrate the potential of using internal representations for self-evaluation of LLMs.

Calibrating Reasoning in Language Models with Internal Consistency
Zhihui Xiet Jizhou Guo Tong Yu' Shuai Li
"Shanghai Jiao Tong University zhxieml @gmaal 1. com souaili@@sjtu wcu..eduuccoo cn
!Adobe Research The University of Hong Kong
Abstract
Large language models (LLMs) have demonstrated impressive capabilities in var- ious reasoning tasks, aided by techniques like chain-of- chain-of-thought prompting that elicits verbalized reasoning. However, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability robustly process and utilize generated rationales. In this work, we investigate reasoning in LLMs through the lens of internal representations, focusing on how these representations are influenced by generated rationales. Our preliminary analysis reveals that while generated rationales improve answer accuracy, inconsistencies emerge between the model's internal representations in middle layers and those in final layers, potentially undermining the reliability of their reasoning processes. To address this, propose internal consistency examining the agreement of latent predictions decoded from intermediate layers. Extensive empirical studies across different models and datasets demonstrate that internal consistency effectively distinguishes between correct and incorrect reason- ing paths. Motivated by this, we propose new approach calibrate reasoning by up-weighting reasoning paths with high internal consistency, resulting in a significant boost in reasoning performance. Further analysis uncovers distinct patterns in attention and feed-forward modules across layers, providing insights the emergence internal inconsistency. summary, our results demonstrate the potential of using internal representations for self-evaluation of LLMs.
measure of the model's confidence by
Introduction
Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought (CoT) prompting that elicits verbalized reasoning (Wei et al., 2022; Merrill and Sabharwal, 2024). This approach directs the model to articulate step-by-step rationales before answering question, simulating the reasoning process used by humans. With these verbalized rationales, it is expected that not only will the model's 's croblem-solving capabilities be enhanced, but also the interpretability of its predictions will improve. Understanding how LLMs reason is essential for aligning them with human values (Bai et al.. 2022; Li et al., 2024).
Despite the continued improvement in performance and the emergence of new capabilities, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales (Ye and Durrett, 2022: Turpin et al., 2023). One notable failure mode is unfaithful reasoning, where LLMs provide rationales that contradict their final predictions (Lyu et al., 2023; Lanham al., 2023). This makes difficult to determine the trustworthiness of their predictions, highlighting the need for effective calibration methods to assess the reliability of rationales.
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HUMAN
the average temperature in the Himalayas lower than in New York summer?
ASSISTANT
ASSISTANT
Wong
The Himalayas' elevation is much higher than York.
The Himalaya is closer to the equator compared New York. Regions with lower latitude tend hotter. So, the answer False
Density
Higher elevations typically
have cooler temperatures.
So, the answer is True
TTTFF..TT..F
FFTTF.FF..TTF
Latent Predictions
Predictions
Internal Consistency
Consisenny-=4THleyeesss5-5
Consisenss-=FFlLayyrrsss-
An illustration of internal consistency. Given a true-or-false question with the ground truth being true, we elicit latent predictions (i.e., predictions decoded from intermediate layers, represented by "T" for true and "F" for false) from the answer token of reasoning paths. By defining internal consistency the agreement of latent predictions with the final stated one, we observe high correlation between internal consistency and prediction accuracy, which aids calibrating reasoning. Note that the figure on the right is synthesized for illustration purposes, but the distributions reflect actual observations as shown in Figure 3.
Figure 1:
In this work, we present the first exploration of leveraging internal representations to calibrate reasoning in LLMs. Our study is based on the intuition that the inner workings LLMs contain latent structures that can assess the reliability of reasoning paths. To investigate this, we begin by probing the model's intermediate activations for answer correctness CoT reasoning. Our preliminary analysis reveals that CoT reasoning leads to inconsistencies between the model's internal representations middle layers and those in final layers, potentially indicating degree of uncertainty,
Based on this observation, we propose internal consistency (illustrated in Figure 1) as a measure of the model's confidence in the reasoning process. Inspired by previous works on eliciting latent predictions from intermediate layers (nostalgebraii 2020; Belrose et al., 2023), we examine internal consistency by assessing the agreement of latent predictions decoded from intermediate layers. Unlike methods that require additional training and human annotations (Ye and Durrett, 2022; Khalifa et al., 2023), internal consistency provides reliable and off-the-shelf self-evaluation for reasoning.
We conduct extensive empirical studies various reasoning tasks, including reading comprehension, symbolic reasoning, and logical reasoning. Our analysis across different models and datasets demon- strates that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Moreover, by up-weighting reasoning paths with high internal consistency, we achieve significant improvement in reasoning performance. These results highlight the potential of using internal representations for the self-evaluation of L.LMs.
This work makes three main contributions. 1) We identify the emergence of inconsistencies between intermediate and final layer representations in LLM reasoning. highlighting potential issue where CoT reasoning leads to higher uncertainty. 2) We propose internal consistency as a novel measure to evaluate the model's confidence and calibrate reasoning by up-weighting paths with high internal consistency. 3) We provide insights into the cause of internal inconsistency in reasoning by analyzing Transformer components (i.c., attention and feed-forward networks) across layers. We believe our results show promise in leveraging internal representations to enhance reasoning in LLMs.
2
Preliminaries
In this section, we present background information, notations, and preliminary analysis to provide context for our study.
2.1 Transformer Architecture
Our analysis focuses on the prevalent decoder-only Transformer architecture (Vaswani al., 2017: Radford et al., 2018). To set notation and context, we briefly describe the key components follows.

---

Given sequence x (#1...., of input tokens, the inference process of the Transformer begins by projecting these tokens into a sequence of representations hi.... h2 E Rd. This representation then updated by series of L residual blocks (Elhage et al., 2021), each consisting of multi- head self-attention (MHSA) layer followed by a feed-forward network (FFN) layer'. In each block é L - 1I. the representation of each token is updated as follows:
h ++1 h + FFN + MHSA
Finally, the next token distribution is produced by applying an unembedding on h
p(In+1 x) = Softmax Unembed(h")) (
As the dimension of h, remains constant across layers, view it dynamic distribution processed by the model (Geva et al., 2022), providing insight into how LLMs reason internally.
2.2 Preliminary Analysis of Internal Representations in CoT Reasoning
Building upon works that explore the inner workings of LLMs (Burns et al., 2023; Ferrando et al., 2024), our study starts with exploring the encoded information in internal representations during reasoning. In the context of CoT prompting a prevalent technique for eliciting reasoning capabilities of LLMs-we analyze these representations along two dimensions: horizontally across reasoning steps, and vertically across network layers.
Zoom-in Evaluation
We conduct preliminary experiments using the Llama- 2-7B model (Touvron et al., 2023) on the PrOntoQA dataset (Saparov and He, 2023), which challenges the model to determine whether a proposition is true based series of assumptions. We apply 4-shot CoT prompting and greedy decoding to generate rea- soning paths and predictions. To discern encoded information crucial for reasoning, we utilize linear probing (Alain and Bengio, 2016; Belinkov, 2022) to train classifiers on the activations with respect to the ground truth labels, aiming differentiate be- tween true and false propositions. Reasoning paths segmented into individual steps using the NLTK punkt sentence tokenizer (Bird et al., 2009), with a dedicated probe classifier for each layer at the last token of every reasoning step. Implementation details are provided in Appendix B.I.
0.65
Accuracy
10
0.60
Layer
20
0.55
Step
30
0.50
Step
01234567
10
30
Reasoning Step
Layer
CoT reasoning improves answer accuracy but exacerbates the inconsistency between hidden and stated reasoning. Left: Heatmap of linear probe accuracies at all rea- soning steps and intermediate layers. Right: A zoom-in on the results of the last two steps.
Figure 2:
Figure 2 presents an evaluation of the probe accuracy the validation set, including detailed zoom-in on the probe accuracy at the last two reasoning steps, which reveals two distinct patterns:
1. Improved accuracy through verbalized reasoning: Probe accuracy increases monotoni- cally as the model processes and verbalizes reasoning paths, indicating that LLMs extract and utilize task-solving information from their generated rationales.
2. Emergent inconsistency across layers: A closer examination at the last two reasoning steps reveals a notable inconsistency in probe accuracy between middle and later layers. This inconsistency suggests that while middle layers capture essential reasoning information, may not be fully utilized or maintained by the later layers, potentially impacting the model's overall reasoning performance.
This analysis provides initial understanding of how LLMs internally process and handle information during CoT reasoning. While LLMS effectively gather information from verbalized reasoning paths, there is noticeable tendency for these models underutilize the processed information at intermediate layers. This motivates us to further investigate into the internal representations of LLMs and explore methods to better harness them enhance reasoning capabilities.
'Modern Transformer models apply normalization (Ba et al., 2016; Zhang and Sennrich, 2019) before or after each layer. We omit the notation here for simplicity.
3

---

2.3 Calibration
Calibration refers to the alignment between a model's predicted probability estimates and their actual prediction correctness (Guo al., 2017; Geng al., 2023). Calibration is crucial for assessing the reliability and trustworthiness of LLMs, and can also help improve performance (Wang et al., 2022).
More formally, want to find measure Score such that for any two prompt-ansser-preccccioo triplets (X,, V:) and (x3+X8 N,), the following holds:
Score (xa: y:) - Score (xj, - p (ÿ: y. x2) p (8, - Yj xj)
In this work, we reveal the potential of using internal representations for calibration and introduce new measure based on internal consistency, as described Section 3.2. The new measure does not require additional training and is more accurate than logit-based methods.
3 Internal Consistency as Lens to Calibrate Reasoning
In Section 2.2, uncover patterns of internal inconsistency in reasoning with linear probes. While probe accuracy indicates the "extractability" of task-solving information, it does not necessarily show how LLMs process internal representations for generation (Alain and Bengio, 2016; Belinkov, 2022). Therefore, we introduce a new method to measure internal consistency in this section.
3.1 Interpreting Internal Representations
The internal reasoning process of Transformer towards its final prediction can be comprehended through iterative inference (Jastrzębski et al., 2017; Geva al., 2022). A direct method to inspect this internal process during generation to early exit from intermediate layers. Specifically, instead of applying unembedding on h, we can obtain the token distribution over the vocabulary from any intermediate layer using the logit lens (nostalgebraist, 2020):
pd (xn+1 x) = Softmax Unembed (h.))
Building on this interpretation, we can decode latent predictions from any layer é
a a+1 arg p'xxpp(x x).
(I)
x
While Equation provides measurement of latent predictions, we find that the decoded distributions are often miscalibrated, biasing towards specific answers (Zhao et al., 2021; Belrose et al., 2023). For instance, in the PrOntoQA task with Llama-2-7B, the penultimate layer consistently assigns over 90% probability to the answer "True," regardless of the context. To address this, we balance the latent predictions across the possible answers for each layer separately. See Appendix B.4 for more details.
3.2 Calibration Measurement with Internal Consistency
Built our findings in Section 2.2, we investigate consistency of internal representations indication of uncertainty. Intuitively speaking, the model's internal representations are highly inconsistent with those of later layers, it may not faithfully say as it thinks. Based these intuitions, we propose a simple metric called internal consistency to measure the agreement of latent predictions.
Internal Consistency During inference, we collect latent predictions at the answer tokens and measure how often latent predictions match the final prediction:
L-1 Σ
Internal (x,
1 1
t=1
This measure offers a straightforward method to gauge the level internal consistency, serving as useful indicator of the output's correctness (i.e., calibration), without requiring additional training.

---

Calibrating Reasoning with Internal Consistency As LLMs often require long reasoning paths to resolve complex tasks (Wei al., 2022). a process that introduces cumulative errors and uncertainty, we integrate this consistency measure into the generation process to enhance reasoning calibration. Specifically, we propose new decoding method that assigns weights to each sampled reasoning path based on their calculated consistency score at the final answer token, Paths that exhibit higher internal consistency-indiece
robust alignment between intermediate and final predictions greater weight in determining the final model output. This weighted strategy aims to prioritize reasoning paths that not only maintain self-consistency but are also more likely to converge on the correct answer, thus potentially enhancing the model'ss 's accuracy in complex reasoning tasks.
assigned
4 Experiments
To study internal consistency, we conduct extensive experiments to address the following questions: 1) To what degree does internal consistency correlates with reasoning performance? 2) Can we leverage internal consistency to boost reasoning? 3) How do different components of the Transformer architecture contribute to the emergence of internal inconsistency?
4.1
Setup
We evaluate internal consistency across spectrum of models and tasks. See Appendix B.2 for further details of the experimental setup.
Models Our experiments are conducted using two prominent series of open-source Transformer- based models: the Llama-2 series (Touvron et al., 2023) and the Mistral series (Jiang et al., 2023). Specifically, we evaluate both the 7B and 13B configurations of the Llama-2 series to understand how model scale impacts internal consistency. Additionally, for the Mistral series, we explore the 7B and the 8B versions, the latter of which incorporates mixture of experts layers (Jiang et al., 2024). These models were chosen due to their extensive use and distinct architectural features, enabling comprehensive evaluation of internal consistency.
Datasets We evaluate various datasets that span reading comprehension, symbolic reasoning, and logical reasoning tasks. We choose these datasets because they involve explicit reasoning processes with unambiguous single-token answers (i.e., "True" and "False""), " which facilitates our analysis. To standardize the evaluation process, each dataset is transformed into a frue-o-fflls QA format. For detailed description of the datasets and examples, please refer to Appendix B.2.
1. BoolQ (Clark et al., 2019): A reading comprehension dataset where each instance involves a yes/no question grounded a related passage.
2. CoinFlip (Wei et al., 2022): A dataset that challenges the model's symbolic reasoning abilities by presenting task where the model must determine the outcome of coin (heads or tails) after a series of flips.
3. PrOntoQA (Saparov and He, 2023): dataset designed for logical reasoning. Each question requires a 3-hop deduction to determine the truth value of proposition based set of assumptions with fictional concepts.
4. ProofWriter (Tafjord et al., 2020): A logical reasoning dataset that, in contrast to PrOntoQA, uses real concepts for all assumptions. Each question requires 3 hops of reasoning.
We balance the labels and each dataset has at least 500 samples for evaluation. Our method requires no training procedure. To evaluate reasoning performance, we use calibrated accuracy following Zhao et al. (2021); Burns et al. (2023), balancing predictions to be 50/50 across the two labels.
4.2 Internal Consistency is Good Calibration Measure
Our analysis begins by examining patterns of internal consistency across different layers of LLMs. As illustrated in Figure 6 (Appendix C), latent predictions exhibit variable convergence during inference; notably, there are significant increases in consistency in the middle and final layers, while the early layers are characterized by considerable noise. Additionally, these patterns vary across models and
5

---

Effect of Prompting
Distribution Discrepancy
Pattem DI Sccrepancy
Calibration Curve
3.0
Wrong
90
Wrong
80
2.0
84
60
3.5
20
Interna
40
1.0
60
0.5
50
20
Zerovshol
Fevesshut Few-shut
25
50
75
100
10
20
25
50
75
100
COT
Intemaa Consistancy (84)
Layer
Internal Consistency 196)
Figure 3: Internal consistency is reliable measure of prediction confidence in CoT reasoning.
From left to right: 1) the effect different prompting techniques on the model's internal consistency; 2) the distribution discrepancy of internal consistency between correct and incorrect model predictions; 3) pattern variations in agreement values (representing the ratio of data instances where the latent predictions match the final predictions) across layers; and 4) calibration plot with bins according to the model's internal consistency on the x-axis and the accuracy within each bin on the y-axis. Results averaged over all models and datasets. See Appendix C for the full results.
datasets. Smaller models, such as Llama-2- 2-7B, show high variability in latent predictions across layers, indicating less stability in their reasoning processes. In contrast, larger models like Mixtral 8x7B demonstrate more robust internal reasoning, as evidenced by more reliable convergence of latent predictions. For BoolQ, the inconsistency less pronounced compared to other datasets that require more complex symbolic and logical reasoning.
Furthermore, the correlation between internal consistency and model accuracy is particularly evident when distinguishing between correct and incorrect answers in CoT reasoning. As depicted in Figure 3, the impact of prompting techniques on internal consistency is significant, highlighting the emergence inconsistency brought by CoT prompting as discussed in Section 2.2. While few-shot demonstrations generally enhance consistency, CoT prompting decreases it. A closer examination of CoT reasoning reveals that reasoning paths leading to incorrect answers typically exhibit lower internal consistency. These results underscore the potential of internal consistency only diagnostic tool but also as a reliable measure for calibrating reasoning in LLMs.
4.3 Enhancing Reasoning with Internal Consistency
Our analysis reveals strong correlation between internal consistency and prediction accuracy, suggesting its potential mechanism for improving reasoning performance. As demonstrated in Section 4.2, CoT prompting, while effective, tends to decrease internal consistency. This decrease possibly attributable to the generated reasoning paths that incorporate incorrect rationales, which confuse the model to make inconsistent predictions across layers. To investigate the generality of this phenomenon and its potential solutions, also least-to-most (L2M) prompting (Zhou et al., 2022), which decomposes complex problems into simpler sub- sub-problems for reasoning. For a detailed description of our ceast-to-most decomposition approach, please refer to Appendix B.3.
To incorporate internal consistency, we build on the self geff-consistency (SC) approach (Wang et al., 2022), which utilizes ensemble of multiple sampled reasoning paths to increase accuracy. Our proposed method, which we term SC+ IC, integrates internal consistency to weight these reasoning paths accordingly. Specifically, if reasoning path demonstrates high internal consistency in its answer prediction, the probability of accepting its corresponding answer as the final answer increases. We accumulate the sum of internal consistency scores across all reasoning paths for cach answer and choose the answer with the highest sum. For basclines, we also compare with greedy decoding Greedy and logit-based approach SC+A A (Wang and Zhou, 2024) that selects the final answer based on a confidence measure A = p (5 x) p (5² x), where ý' and ÿ represent the top two tokens for answer prediction of the k-th path.
Table summarizes the results, which demonstrate that SC+IC consistently outperforms the others across different models and tasks, with improvements ranging between 1.8% to 4.9% across models. Figure plots the calibrated accuracy with respect to varying numbers of sampled paths. We observe clear improvements of SC+ IC over the baselines, highlighting the effectiveness of leveraging internal

---

Table 1: Internal consistency improves reasoning performance across diverse tasks. We report calibrated accuracy averaged across runs of 10 different random seeds. For self-consistency (SC) and its variants, results are obtained with 40 sampled reasoning paths.
BoolQ
CoinFlip
PrOntoQA CoT
ProofWriter CoT
Mean
CoT L2M
CoT
L2M
L2M
L2M
CoT
L2M
LLAMA-2-7B
Greedy
61.4
71.2
81.9
51.7
51.1
52.4
60.7
60.6
63.8
SC
70.0
73.2
76.0
89.2
49.0
50.0
52.2
66.8
61.8
69.8
SC+A
69.9
73.0
76.2
90.1
50.2
50.1
52.4
68.2
62.2
70.3
SC+IC
70.7
73.4
77.6
90.2
49.0
51.2
52.2
69.0
62.4
71.0
SC+IC (tune)
78.6
50.8
55.7
70.7
SC+IC (transfer)
71.5
73.5
78.7
93.9
50.8
55.7
53.9
70.7
63.8
73.4
LLAMA-2-13B
Greedy
78.5
71.3
78.9
90.5
53.6
54.0
60.8
78.1
67.9
73.5
SC
80.8
78.3
81.0
94.2
52.6
55.4
61.0
88.6
68.9
79.1
SC+A
81.2
78.9
81.8
94.2
55.0
57.6
61.6
87.9
69.9
79.6
SC+IC
81.4
78.7
84.0
93.2
55.4
57.0
61.8
88.6
70.6
79.4
SC+IC (tune)
81.4
80.4
86.2
96.5
54.5
56.6
62.9
89.6
71.2
80.8
SC+IC (transfer)
80.2
86.4
54.4
56.2
62.8
89.7
71.2
MISTRAL--BB
Greedy
74.0
72.7
89.8
99.1
55.3
53.4
65.6
73.3
71,2
74.6
76.8
78.9
95.2
99.4
58.0
51.4
68.6
84.0
74.6
78.4
SC+A
77.9
79.5
95.2
99.4
59.6
52.7
66.8
85.3
74.9
79.2
SC+IC
78.1
78.9
96.4
99.4
59.8
53.2
70.0
85.0
76.1
79.1
SC+IC (tune)
77.8
79.5
97.7
99.9
60.4
56.6
71.2
88.6
76.8
81.2
SC+IC (transfer)
77.8
79.4
97.7
99.9
60.5
56.5
71.1
88.7
76.8
81.1
MIXTRAL-8x7B
Greedy
78.8
78.2
98.8
99.4
57.4
57.9
72.3
79.4
76.8
78.7
SC
81.6
85.3
99.4
100.0
61.6
56.8
75.6
90.2
79.6
83.1
SC+A
82.6
85.6
99.4
100.0
63.4
57.9
75.0
90.6
80.1
83.6
SC+TC
81.7
85.6
99.4
100.0
63.4
56.4
78.0
90.6
80.6
SC+IC (tune)
81.8
86.1
99.9
100.0
63.8
59.3
78.8
92.1
81.1
84.4
SC+IC (transser)
81.8
86.1
100.0
100.0
63.8
59.5
78.9
92.1
81.1
84.4
consistency to enhance reasoning. This improvement is particularly notable tasks involving symbolic and logical reasoning, which depend heavily the correctness of reasoning paths. In addition, we observe that internal consistency effectively distinguish between correct and incorrect paths (c.g., in the PrOntoQA examples in Table 2, internal consistency helps filter out flawed rationales), validating our hypothesis on why internal consistency can enhance CoT reasoning.
4.4 Layer-weighted Aggregation Finds Transferable Patterns
While results of SC+IC have demonstrated the effectiveness of leveraging latent predictions for reasoning, the relative importance different layers remains unexplored. Recent studies have shown that certain intermediate layers specialize in specific types of reasoning (Yang et al., 2024). These observations suggest that treating all layers equally when computing internal consistency might be suboptimal. Motivated by these insights, we introduce aggregation strategy for internal consistency with two variants: SC+IC (tune) and SC+IC (transfer) Specifically, instead of assigning equal weights to each layer in Equation 2, we consider learned weight vector IR (where L is the number of layers) for aggregation. For SC+IC (tune), we optimize layer weights using 500 held-out samples per dataset. The weights are learned using Adam optimizer with learning rate of 0.01 for 1,000 iterations. SC+IC (transfer) evaluates cross-task generalization by applying weights tuned PrOntoQA to other datasets.
The results are presented in Table and Figure Despite introducing only parameters, this weighted approach yields substantial improvements. Notably, the strong performance of SC+IC

---

Reading Comprehension
Symbolic Reasoning
Logical Reasoning
78.0
62.0
Accuracy 060
77.0
88.0
SC
87.0
60.0
+
75.0
SC+IC (trensfer)
Number of Vates
Number of Votes
Number of Vetes
Figure 4: Internal consistency brings larger gains for complex reasoning tasks. The figure shows calibrated accuracy function of the number votes for three types of tasks: reading comprehension (BoolQ), symbolic reasoning (CoinFlip), and logical reasoning (PrOntoQA and ProofWriter). Results are averaged over all models and random seeds.
(transfer) across diverse tasks indicates that optimal layer importance patterns generalize well, demonstrating the broad applicability of internal consistency for reasoning tasks.
4.5 Investigating the Role of Transformer Components in Internal Inconsistency
Having observed that CoT prompting exaggerates internal inconsistency and that reasoning perfor- mance can be improved by we-weighting reasoning paths based on internal consistency, we now aim to understand why internal inconsistency emerges from CoT reasoning. Specifically, we explore how different network components contribute to the final predictions CoT' reasoning and, consequently, the calibration effect of internal consistency. One hypothesis is that this effect arises from better elicitation of information processed in the middle layers, which tends underutilized later layers. To investigate this, we conduct a detailed analysis focusing the roles of self-attention and feed-forward network (FFN) layers.
We analyze the average attention weights across all attention heads each layer, focusing on what the answer tokens attend to. Consistent with the CoT process, we segment the processed tokens into three parts: the context, the query, and the generated rationale. Further details on the methodology and calculations can be found Appendix B.5.
Self-attention Layers
FFN Layers For FFN layers, use probe vector trained the model's last hidden state of the answer token with respect to the model's output. We compute cosine similarities between this probe vector and the value vectors (Geva et al., 2022: Lee et al., 2024), each corresponding to a column
Table 2: Examples where internal consistency helps select correct reasoning paths for Mistral-7B PrOntoQA. We show the paths with the highest and lowest internal consistency. The question has been reformatted for better presentation. Incorrect steps are colored in gray, whereas wrong predictions are highlighted in red.
Question: Zumpuses not transparent. Fach zumpus is wumpus. Wumpuses red. Each wumpus a jompus. Impuses are amenable. Every jompus is spicy. Jompuses are tumpuses. Every tumpus floral. Tumpuses are yumpuses. Every yumpus not amenable, Every yumpus is vumpus. Each vumpus metallic. Every vumpus is a numpus. Numpuses are temperate. Numpuses are dumpuses. Dumpuses are not small. Dumpuses are rompuses. Polly is jompus. True or false: Polly is amenable.
Sampled Paths:
P1 (IC=:.875): Polly is jompus. Jompuses are tumpuses. Polly is a tumpus. Every tumpus is floral. Tumpuses yumpuses. Polly is yumpus. Every yumpus is amenable. Polly not amenable. False
P2 (1C=0.656): Polly is a jompus. Jompuses are impuses. Impuses are amenable. Polly amenable. True
Ground-truth Path: Polly is jompus. Jompuses tumpuses. Polly is tumpus. Tumpuses yumpuses. Polly is yumpus. Every yumpus is not amenable. Polly is not amenahle. False

---

Llama-2778 on PrOntoQA
Llama-2-78 on ProcfWitter
Mistral-7B PrOntoQA
Mistral-78 ProofWrteer
No Number of Top Valu Wedeors
6.3-
Query Bationale
0.5
40
02-
30
30
-00
0.3
20
20
0.2
6.1
10
0.1
0.1
5
1.0
20
Lay 30
100
20
Layer 30
Layer
Layar
Figure 5: The emergence of internal inconsistency in CoT reasoning could be attributed to the misalignment between layers with high attention weights on critical tokens and those promotes certain predictions. The histograms display attention weights for each part (context, query, and rationale) across self-attention layers, accompanied by gray line indicating the count of value vectors in FFN layers that achieve high cosine similarity to the model's 's final prediction.
in the last FFN matrix of the layer. Our analysis focuses on the vectors with the top 0.1 of cosine similarity values, which play a pivotal role in forming final predictions. For more details, please see Appendix B.6.
Results Figure presents our findings: 1) Self-attention layers in the middle layers show marked focus on the query and reasoning steps, a pattern consistent across both models and datasets; FFN layers in the later layers dominate the final model outputs, as indicated by the fact that value vectors with the highest cosine similarities the probe vector tend to cluster in the later layers. The layers with strong attention query and reasoning steps and those where high cosine similarity value vectors cluster do align, providing insight into the possible mechanism behind the emergence of internal inconsistency. This finding is consistent with Figure 2 and Figure 6, suggesting that the model may make correct predictions the middle layers but fails to fully utilize them in the later layers. To further understand the specific concepts these high-similarity value vectors represent and verify their importance in determining the final prediction, we project them onto the vocabulary space (nostalgebraist, 2020). Additional analysis are provided in Appendix B.6.3.
Related Work
Understanding the Inner Workings of Language Models The rapid progress in LLM devel- opment has necessitated simultaneous efforts to interpret the inner workings of advanced mod- els (Bricken al.. 2023; Ferrando et al., 2024). These works aim to provide an internal view of mechanisms to ensure safety and fairness (Burns et al., 2023; Zou et al., 2023; Li et al., 2024) and to further improve model inference (Schuster et al., 2022; Raposo et al., 2024). Some studies also examine CoT reasoning from internal perspective and propose methods to teach models to perform implicit CoT (Deng et al., 2023). Unlike these approaches, which require additional training for interpretation, our internal consistency measure offers an off-the-shelf solution to calibrate CoT reasoning, providing a practical and efficient tool for enhancing model reliability.
Calibration in Language Models 2015; Guo et al., 2017) train a parameterized classifier on validation data to adjust the final output of a neural network towards expected outcomes. In the context of LLMs, previous works apply trained parameterized models to the logits at the final layer (Zhao et al., 2021; Shen et al., 2024). In comparison, our study focuses on the phenomenon of internal inconsistency CoT reasoning and demonstrates that internal consistency is a reliable unsupervised calibration measure.
Traditional calibration methods (Platt et al., 1999; Naeini et al.,
Faithfulness of CoT Reasoning Although the reasoning capabilities of LLMs have been greatly enhanced with techniques like CoT reasoning (Wei al., 2022: Yao et al., 2022; Xia al., 2024), previous investigations into the faithfulness of CoT reasoning have shown that the model's generated rationales often are not consistent with their predictions (I-yu et al., 2023; Lanham al., 2023). These studies primarily focus the alignment between model's verbalized reasoning and its outcomes,

---

without delving into the model's internal reasoning processes. In contrast, our work provides new insights into how unfaithfulness emerges internally during CoT reasoning and proposes solutions to calibrate reasoning with internal consistency.
work is that of Halawi et al. (2023), which studies harmful imitation behaviors of LLMs and the underlying internal mechanisms. Unlike their study that focuses on few-shot learning with false demonstrations, we provide new insights the phenomenon that CoT reasoning leads to internal inconsistency and that we can calibrate reasoning with internal consistency.
Closest
6
Conclusions
This work explores the use of internal representations to enhance the reliability of reasoning in LLMs. By examining CoT reasoning, we identify that although generated rationales can improve answer accuracy, they also lead to inconsistencies between middle and final layer representations, potentially affecting reliability. To mitigate this, we introduce internal consistency as a confidence measure, which evaluates the alignment of latent predictions from intermediate layers. Our extensive empirical studies across multiple models and datasets demonstrate that internal consistency a robust indicator of correct reasoning paths, which we show can be further used for enhancing CoT reasoning by prioritizing paths with high internal consistency. Finally, our analysis on the patterns in attention and ferwadd networks across layers provides insights why internal inconsistency emerges.
This work has several limitations. First, we restrict our empirical study on decoder-only models. While techniques like decoder lens (Langedijk et al., 2023) is a promising way extend the concept of internal consistency to encoder-decoder models, we leave it for future work. Second, our analysis focuses on vanilla CoT prompting as the simplest approach for reasoning. whereas many other prompting techniques have been proposed (Gao et al., 2023; Yao et al., 2024). It would be interesting to further investigate internal consistency under these more complex scenarios.
References
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, 2024. URL https: // //ooppeeeviieee net forum? JNG Ph8Wh.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 2022.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378 35:30378-30392, 2022.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don't 't always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388 2023.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful chain-of-thought reasoning. arXiv preprint arXiv 13379, 2023.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernan- dez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of thhiugh reasoning. arXiv preprint arXiv:2307.137000 2023.
10

---

nostalgebraist.
Interpreting GPT: The logit lens. Al Alignment Forum, 2020. URL https://wwer al www.nngininniaaa.aeemmmeeeetteie forum. org/posts posts wossts//kkBBBwwwaassee...... interpreecing---tt-t-----ggggg
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXivv2303.088 12, 2023.
Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang, Grace: Discriminator-guidee chain-of-thought reasoning. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15299-15322, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield- Deddde Danny Hemandez,, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http://taansoommee....iiuipppppp /frra//wwr//////ieee html.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rX1::1607.6444 2016.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14660, 2022.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference Learning Representations, 2023. URL https: opwnreviewweeeeeeeeo wet.iieweeeee/eeeeeeoe forum ETK-
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. primer on the inner workings of transformer-basee language models. arXiv preprint 2rXiv:2405.00200 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fune-tuned chat models. arXiv preprint arXin::307.002888 2023.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of -fttuhgghttttt The Eleventh International Conference on Learning Representations, 2023.
URL https: http///ppeeeeeeeeeeeeeeeeeeeeeee net forumm id=oFWVBzXxtr VVB2XxRVV
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.0166 2016.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):007-219, 2022.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O' O'Reilly Media. Inc.", 2009.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of language model confidence estimation and calibration. arXiv preprint arXi::2311.08298, 2023.
11

---

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 1711 2022.
Stanistaw Jastrzębski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. arXiv preprint 1rXiv:/710.04777 2017.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning. pages 12697-12706. PMLR, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.044 04008888, 2024.
Christopher Clark, Kenton Lee, Ming-Woo Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1xXiv:995..1044 2019.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012. 2020.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 2022.
Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200, 2024.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837 2024.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967, 2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravee, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph. Zac Hatfield-Dodds Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. http:///wwnooommm.mcmccii//////p features/index.htm.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach ai transparency. arXiv preprint arx 2023.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing 3 15:16756-177777 2022.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:24044002258 2024.
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stu- art Shieber. Implici chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01440, 2023.
12

---

John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference artificial intelligence, volume 29, 2015.
Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. Thermometer: Towards universal calibration for large language models, arXiv preprint arXiv:2403.0888/, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. arXiv preprint arXiv:2210.03629, 2022.
Yu Xia, Rui Wang, Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. Beyond chain-of-thought. survey chain-of-x paradigms for llms. arXiv preprint arXiv:2404. 2024.
Danny Halawi. Staai-Stanisla Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. arXiv preprint arXiv:2307.09476, 2023.
Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem Zuidema, and Jaap Jumelet. Decoderlens: Layerwise interpretation of encoder-decoder transformers. arXiv preprint arXiv:2310.03666, 2023.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aide -added language models. In International Conference on Machine Learning, pages 10764- 10799. PMLR, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0377 2019.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22215 2022.
Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D. Goodman. Certified deductive reasoning with language models, 2023.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, arXiv preprint XrXiv: 2019.
13

---

A Broader Impacts
This work contributes to the understanding and enhancement of reasoning capabilities in LLMs through the lens of internal consistency. While our findings and methodologies offer significant advancements in the calibration and reliability of LLMs, there are potential negative societal impacts to consider. Enhanced reasoning abilities in LLMs could be misused to generate more convineing disinformation, exacerbating issues related to misinformation and manipulation. To mitigate these risks, is crucial implement robust monitoring and access control mechanisms. This includes gated release of models, ensuring that only authorized and ethical usage is permitted. Furthermore, developing and providing defenses against misuse, such as mechanisms for detecting and countering disinformation, is essential. By proactively addressing these potential negative impacts, we aim to responsibly advance the field while safeguarding societal interests.
B Implementation Details
We performed all experiments compute node with 8 Nvidia GPU cards and 512 GB of memory. To replicate our results, the total estimated computing time is under 100 GPU hours, even though the entire research project required more computing power than the experiments detailed in the paper.
B.1 Preliminary Analysis
We conduct linear probing on activations to investigate how information is processed throughout layers and reasoning steps during CoT reasoning. Specifically, for each reasoning step and intermediate layer, we train a logistic regression classifier on the corresponding hidden states of all data instances. We split the dataset randomly by 80%/20% into training and validation subsets. Following Radford et al. (2021), we use the Scikit-learn package (Pedregosa et al., 2011) and determine the L2 regularization strength À using hyperparameter sweep over the range between 10 and 10 for logistic regression.
B.2 Main Experiments
Models Our implementation based on the Huggingface's Transformer library (Wolf et al., 2019). We use the Llama series. For the Mistral series, we evaluate mistralla.MMsstra mistrrala watrraaaaaiiiiiia /ttp//iiiatl11x11 v0. The Llama and Mistral series models released under the custom license and apache-2.0 license, respectively.
and peeta-llama//ilmmee-222
for
and
Dataset Construction
The construction processes for the test datasets are as follows:
We use the original validation ser', which has 2033 positive and 1237 negative samples. We balance the classes by randomly sampling 1237 samples from the positive ones, resulting in a subset of 2474 samples. BoolQ is released under the Creative Commons Share-Alike 3.0 license.
BoolQ:
2. CoinFlip:
Since the original dataset used in Wei et al. (2022) is not publicly available, we use the reproduced one" created by Kojima al. (2022) which contains 500 instances.
3. PrOntoQA: We use the official script provided by Saparov and He (2023) to generate 500 instances. Each instance requires 3 hops of reasoning and is based on fictional concept names. PrOntoQA is released under the Apache-2.0 license.
4. ProofWriter:
We use the subset" provided by Poesia et al. (2023) and randomly sample 500 instances. ProofWriter is released under the CC BY license.
http:///tttt..ooooieirreeeeeeeer coin hah__ttipp/ccch_Elii..oooor
httppp///...aaaoaaaacacc/caaaaaa prootwitee//rroowiiteer
14

---

Prompt Format We use the prompt format { (context) \nO (query) \nA for all datasets and models. For few-shot prompting, in-context examples are prepended using the following format (taking 2-shot prompting as an example): (icl example 1 (n)nn (nnnn(iii (icl example 2) 2))n (test example). For few-shot CoT prompting, we append the rationale after for each in-context example. Table 3 provides list of examples for the considered datasets.
Few-shot Examplars We use the few-shot examples from Wang et al. (2022) for BoolQ and from Wei et al. (2022) for CoinFlip. For ProofWriter and PrOntoQA, we directly utilize the examplars from the original datasets.
Table 3: Data examples.
Dataset
Context
Query
BoolQ
The myotonic goat, otherwise known as the fainting goat, is domestic goat that temporarily seizes when it feels panic. If startled by sudden movements loud noises, they will attempt escape from the disturbance, generally followed by a startle reaction. In more severe cases, this reaction results strong tetanic contractions of the agonist and antagonist muscles, causing an uncontrolled stiffness that may
True or false: is there such thing as faint- ing goat
the goat remain "frozen" in the position that it was in previous the attack, cause fall to the ground on side. During an attack, which may last from 5-20 seconds, the goat can often be picked up without any bending or movement occurring in its body. In the case of goats that are less severely affected with the condition, there may be stiffiness observed running away. This behaviour caused by a hereditary genetic disorder called myotonia congenita. The myotonic goat, similar humans with congenital myotonia, exhibits obvious muscle wasting, rarely incapacitated the condition, and lives normal and healthy life span.
minor localized the legs, however. they are still capable of
CoinFlip
is heads up. Whitney flips the coin. Erika does not flip the coin. Tj does not flip the coin. Benito flips the coin.
True or false: the coin still heads up?
PrOntoQA
Every tumpus sour. Tumpuses are rompuses. Every rompus is small. Every rompus is an impus. Impuses are floral. Impuses are dumpuses. Dumpuses are not kind. Every dumpus is a yumpus. Each yumpus feisty. Yumpuses are numpuses. Numpuses are not opaque. Each numpus is a zumpus. Every wumpus is opaque. Every zumpus is temperate. Each zumpus a jompus. Every jompus is dull. Jompuses are vumpuses. Alex is dumpus.
True false: Alex not opaque.
ProofWriter
Fiona is blue. Harry is cold. Harry white. Harry is and Harry is green then Harry is not round. If something is green then it is young. All white things are young. If something is green and not furry.
True or false: Harry round.
not white then is not blue. Young, round things something is white and young then it is round. If something is young and not cold then it is round. If something is green and not young then it is not round.
B.3 Least-to-Most Prompting
In addition to chain-of- -fahhuttt prompting, we evaluated our approach using Icast-to-most prompt- ing Zhou et al. (2022) across four datasets. Due to the absence of existing least-to-most prompting templates for these datasets, we developed a systematic process to generate and validate appropriate prompts.
For each dataset, we selected 4 representative instances as our prompt engineering samples. We utilized GPT-4o to generate initial teast-to-moss decompositions for these instances, followed by manual inspection and refinement. Table 4 presents examples from each dataset demonstrating our lcast-to-mov most decomposition approach.
15

---

Table 4: Examples of least-to-most prompting decomposition across different datasets.
Dataset
Context
Decomposition Steps
BoolQ
System Down, sometimes shortened System and abbreviated as SOAD, is an Armenian-Americal heavy metal band from Glendale, California. formed in 1994. The band currently consists of Serj Tankian (lead vo- cals, keyboards), Daron Malakian (vocals, guitar), Shavo Odadjian (bass, backing vocals) and John Dolmayan (drums).
Who are the mem- bers of System of Down? 2. What are the roles of each mem- ber? 3. How many members are listed as vocalists?
CoinFlip
A coin is heads up. Ka flips the coin. Sherrie flips the coin.
What the starting position of the coin? 2. What happens to the coin after each flip?
PrOntoQA
Every tumpus is sour. Tumpuses are rompuses. Every rompus small. Every rompus is impus. Impuses are floral. Impuses are dumpuses. Dumpuses are not kind. Every dumpus is a yumpus. Fach yumpus is feisty. Yumpuses are numpuses. Numpuses are not opaque. Each numpus is zumpus. Every wumpus is opaque. Every zumpus temperate. Each zumpus is jompus. Every jompus is dull. Jompuses are vumpuses. Alex is a dumpus.
1. What are the prop- erties of dumpus? Alex a dumpus? Is a dumpus opaque?
ProofWriter
Fiona is blue. Harry is cold. Harry is white. If Harry is blue and Harry is green then Harry is not round. If something is green then is young. All white things are young. If something green and not white then it not blue. Young, round things are not furry. something is white and young then it is round. If something is young and not cold then it is round. If something green and young then is round.
1. What are Harry's characteristies? 2. What does being white imply about heing young? 3. What does being white and young imply about being round?
Our decomposition strategy focuses breaking down complex reasoning tasks into simpler, sequen- tial steps. This systematic approach helps the model handle complex reasoning tasks by addressing simpler sub-problems sequentially.
Generation In our few-shot CoT experiments, we use Nucleus sampling (Holtzman et al., 2019) with a temperature of 0.7 and a top-p of 0.95 to generate reasoning paths.
B.4 The Issue of Miscalibration in Latent Predictions
Ideally, for a balanced dataset with an equal proportion of True and False instances, we would expect the LLM to assign approximately 50% probability to each class. However, our preliminary analysis shows that latent predictions tend to exhibit skewed distribution, allocating disproportionately high probabilities to one class. Therefore, we implement specific steps to balance the predictions at each layer before use, following Halawi et (2023).
Denote the probability assigned by the LLM to the True class as p( (Tuue) and the probability assigned to the False class pFFasse). We apply the following steps for each layer:
1. We collect the model's output logits for all instances in the dataset.
ps Pr:Tuu (True) 2. We normalize the probabilities in the i-th instance such that p. (True) p. (True) +p; (False)
3. We determine the median of the set {p be
4. For any given instance, if the model's output P: (True) 2 1, we classify it as True; otherwise, classify it as False.
16

---

B.5 Self-attention Layer Analysis
We are interested in the attention value on the context, the query and the generated rationale part. Our calculation can be formally represented as:
AttnScoro(e,P)) H h=liEP
MHSA (i, (i,tam)
where AttnScore(t, P) denotes the average attention weight across the attention heads on layer for part P, lans denotes the index of the answer token, MHSA (t.h) (i, (i,
signifies the sum of attention
scores from the i-th token the j-th token within the sequence, as computed by the h-th attention head at the E-th layer.
B.6 FEN Layer Analysis
B.6.1 Value vector
We utilize the concept of the value vector and the weighted value vector of FFN given by Geva et al. (2022). Specifically, each FFN in layer & is made up of two linear transformations with point-wise activation function between:
FFN o(Wka")We
where W Wy E [Rdm xd are the parameter matrices and a is the activation function. We define the value vector v as the i-th column W.. For an input x', the keys produce a vector coefficients mf := (W fxx²) E Rd", then the output of FFN is the weighted average of the value vectors with respect m that is:
dm
FFN FFN'(#*)
Σ m
B.6.2 Implementation Details
We execute the models each dataset and extract their final output (True False) and the last hidden state. Subsequently, we apply logistic regression (without intercept term) in the Scikit-learn package (Pedregosa et al., 201 1), and determine the optimal La regularization strength À using hyperparameter grid search, based on the last hidden state and the final output of the model. The accuracy evaluated by 5-fold cross-evaluation is above 93% consistently (see Table 5 for details). We observe that using ground truth labels as the training target of logistic regression results in similar result with Figure but the training accuracy is poorer as the model may not know the correct answer.
Then we calculate the cosine similarity of the value vectors and the probe vector. As the average cosine similarity across all layers is relatively noisy because many value vectors are not relevant to the task, we plot the count value vectors of top 0. % the cosine similarity values instead.
Table 5: Cross-validation accuracy of the training of the probe vector.
ProofWriter
PrOntoQA
Mistral-7B Llama-2-7B
97.0% 93.6%
97.6%
96.3%
B.6.3 Semantics Analysis
To uncover the concepts that the value vectors of the top cosine similarity promote, project them onto the vocabulary space. Specifically, we are interested in the top tokens of the followings:
The probe vector, which we refer to as W prohe-
The value vectors with top cosine similarity with the probe vector. We refer the i-th value vector at layer & as vf (i and & are indexed from 0).
17

---

We stack the value vectors of top 100 of cosine similarity to a N x d matrix (N = 100) and apply SVD, obtaining the top singular value vectors. We refer the singular value vector of the largest singular value as U.
Filtering out the tokens that are Unicode escape sequences (e.g. \u2705), the result is depicted in Table 9. Note that some token ids encode the same word.
Table 6: Top tokens promoted by the probe vector, value vectors and top singular value vector for Mistral-7B on ProofWriter.
Vector
Top tokens
W proce
truc, true, Truc, Truc, sort, right, kind, fin true, truth, TRUE, True, true, correct, True, reality
19
28 130rr
ver, ver, Ver, Vern, verd, VER, verification, verify right, Right, right, Right, RIGHT, droit, Rights, rights thinking, really, pretty, constantly, calm, misunder, why, how
31
144
V507
v 128644
stronger, infl, elev, loaded, augment, aug, strong, rich most, none, most, none, current, newest, now, more positive, posit, praise, affirm, Pos, triumph, approval, appro right, right, Right, Right, RIGHT, correct, correct, rights both, both, Both, beiden, neither, sia, tanto, Neither
31
v
v 1308
24
6604
U
Table 7: Top tokens promoted by the probe vector, value vectors and top singular value vector for Mistral-7B on PrOntoQA.
Vector
Top tokens
W prob
True, Truc, Trust, TRUE, TRUE, Tru, truc, Truth ver, ver, Ver, Vern, verd, VER, verification, verify true, truth, TRUE, True, true, correct, True, reality positive, posít, praise, affirm, Pos, triumph, approval, appro yes, Yes, Yes, did, does, yes, Roh, rog
130773
v 8222
v 130s
11
V550s
18
positive, affirm, strengths, yes, confirmed, endors, Sull, inclusion true, True, true, True, TRUE, TRUE, truly, Tru, false persistence, esso, mero, triumph, /* tol, lica, akov cligible, legitimate, compatible, permitted, constitutional, ethical, legit, olv
29
VS7os
V6717
V9740
U
FALSE, false, False, absent, FALSE, false, absence, lessness
18

---

Table 8: Top tokens promoted by the probe vector, value vectors and top singular value vector for Llama-2-7B on ProofWriter.
Vector
Top tokens
W proee
True, tr, Tru, Tr, True, TR, true, TRUE t. tub, tap, tile, tut, tum, tin pra, triumph, bless, vict, Pra, positive, solutions, rejo confirm, confirm, yes, -" .. confir, otto, dia, unfortunately grande, greater, gradle. groupe, Grad, gradu, great, grand sam, uniform, identical, uniform, hom, similarity, similar, smooth
678s
v V194
V1777
31
V 6356
20 V 2369
V9429
pra, positive, Pos, Pra, posit, happy, Pos, bless, blo, alive, forced, oku, replacing, eu, active, once ways, ways, MDb, torn, modo, hm, zik, Tourn, etch NOT, nicht, W, notin, False, ikke, false, FALSE, moins
V722
v 3680
Table Top tokens promoted by the probe vector, value vectors and top singular value vector for Llama-2-7B on PrOntoQA.
Vector
Top tokens
W probe
True, TRUE, TRUE, TR, True, true, TR, truth pra. triumph, bless, vict, Pra, positive, solutions, rejo blo, alive, forced, oku, replacing, cu, active, once grande greater, gradle, groupe, Grad, gradu, great, grand active, straight, bold, walt, visible, Active, Active, dx confirm, confirm, yes, confir, otto, dia. unfortunately pra, positive, Pos, Pra, posit, happy, Pos, bless, eu pra, positive, esc, salv, escaped, triumph, rag, gem
v V9949
18
31
16
4462
16
10757
V 9429
V 8600
v V3927
Tr, Tr, tr, tram, trig, tr, trim, trim false, NOT, FALSE, False, notin, negative, fails
U
The result to verify that the value vectors with the highest cosine similarity to the probe vector indeed encode crucial information for determining the final prediction, so as verify the emergence of internal inconsistency from the perspective of pattern of attention and FFN layers demonstrated in Section 4.5. This is evidenced by their representation of important keywords such as "true", "false", "verification", and others.
In addition, we also have some interesting discoveries: 1) Some value vectors appear to be of the top cosine similarity with the probe vector in both datasets, indicating their importance and relevance. 2) The value vectors consistently bias towards "True", and the singular value vector of the largest singular value tends lean towards negative tokens, indicating the necessity of calibration. 3) Some value vectors encode tokens of opposite concepts, which reveals degree of internal inconsistency within a single value vector.
C Additional Results
In this section, we provide additional results to support analysis. Figure 6 illustrates the patterns
of internal consistency across different layers, as discussed in Section 4.2. This figure shows variable convergence patterns across different models and datasets, with significant increases in consistency observed in the final layers. Figures 7-10 present detailed results for each dataset and model corresponding to the findings in Figure 3.
19

---

Llama-2-78
Llama-2-138
Mistral-78
Mixtral-9x76
100
Boolg
100
Cuindlip
90
PrOntsGA
88
Agremment
Proaftiitter
60
50
60
60
40
40
50
20
30
1.0
20
Layer
Layer
Figure 6: The patterns of internal consistency are similar across different tasks and models. The "agreement" value represents the ratio of data instances where the latent predictions match the final predictions. Results are obtained from the zero-shot prompting setting.
Llama-2-PP 78 on BoolQ
Llama-2-138 on BoolQ
Mistral-P8 7B on BoolQ
Mixtral-8x78 on BoolQ
90
80
80
80
70
60
40
50
40
CDl
CoT
CoinFlip
on CoinFlip
100
100
100
80
60
40
20
Zero shot
Few shot
Few shot CoT
Zero shot
Fow shot
Few-shot shat CCT
Zero shot
Few shot
Few-shit CST
shro Few-shot
Few shat CoT
Llama-2-78 PrOntoQA
Llama-2-138 on PrOntoQA
Mistral-78 on PrOntoQA
Mixtral-8x7B PrOntoOA
80
60
60
Interral
40
20
Zerosshot
Feneshot
Feweshet
Zerosshot
Femeshot:
Feoeshot
Zeroshoi
Fwe-shot
Fansshol
Zerovahat Few-shot
Fwneshet
Uama-2-78 on ProofWriter
Uama-2-138 ProofWilter
Mistral-7B ProofWitee
Mixtral-6x7B on ProofWiter
100
100
100
100
80
60
60
40
40
20
20
20
Zeroshot
Fwneshot
Feuesho CoT
Zerowsot
Foreshot
Feweshat GST
Zerosbol
Fwesshtt
Fegsshal CoT
Zerowbat
Fweeshht
Foeessot tolf
Figure 7: Detailed results for each dataset and model of the first panel Figure 3: the effect of different prompting techniques the model's internal consistency.
20

---

Uama-2- 7-88 on BoolQ
Llama-2 2-13B BoalQ
Mistral-78 17 Boolq
Mixtral-xx77 al-8x78 BoolQ)
2.5
Wrong
Whong
Weron
Wromn
2.0
3.5
3.0
1.0
0.5
0.5
3.0
25
100
25
50
75
100
75
1000
75
100
Liama-2 -2---a---88-- on CoinFlip
Llama-2-138 CoinFlip
Mistral-7B CoinFlip
Mixtral-Bx78 ColinFlip
7
Correct Wering
Wronn
Whong
Wronn
Dondhy
2
2
20
40
60
80
100
25
50
75
100
20
40
60
80
100
40
80
Llama-2-78 PrOntoQA
Llama-2-133 PrOnto Q/A
Mistral-78 PrOntoOA
Mixtral-8x78 PrOntoQA
Cornect
Correct
Wrong
Whong
Wroong
Wrongg
2.5
2.5
2.0
2.0
1.5
3.5
1.0
1.0
0.5
0.5
20
40
60
80
100
25
50
75
100
25
50
75
100
20
40
60
80
100
Llama-2 2-78 ProofWrriter
Liama 2-138 ProcfWirter
Mistral-78 ProofWriter
Mixtral 8x/8 Proc/Wirier
Correct Wrong
Correct Whong
Correct Weoinn
Correct Wring
3.0
2.0
2.0
3.5
2.0
3.0
1.5
1.0
0.5
0.5
25
50
75
100
25
50
75
100
25
50
75
100
25
50
75
100
interaal Consistency (36)
Internal Condstency (76)
Interal Condistoncy (76)
Internal Concistency (76)
Figure 8: Detailed results for each dataset and model of the second panel in Figure 3: the distribution discrepancy of internal consistency between correct and incorrect model predictions.
21

---

Liama-2- 2-78
Liama-2-1 2-138 BoolQ
Mistral-78 Roolg
Mixtral-8x78 BoolQ
100
Correct Veoongg
100
Corract
100
Correct Whong
100
Correct
90
90
Wrong
90
90
Wrong
80
8G
70
70
60
60
60
50
50
10
20
30
10
20
30
40
20
30
10
20
30
CoinFlip
LJama-2-138 on CoiniFlip
Mistral-78 CoinFlip
Mixtral-8x76 on CoinFlip
100
100
100
100
Agrosment (85)
80
80
60
40
40
40
20
Cornect
Correct
20
Coneel
20
Coneec
20
30
10
20
30
20
30
10
20
30
Llama-2-44 PrOntoQA
Mistral-78 PrOntoQA
Mixtral-8x78 PrOntoQA
Correct Virong
Correct Wrong
100
Cornect virong
100
Cornect Wrong
90
90
Agrocmort
80
892
80
80
70
70
70
70
60
60
60
50
50
50
50
20
20
100
20
30
2.9
30
10
20
39
ProofWiriter
ProofWriter
Mixtral-Bx7B
100
Correct
Correct Wrona
100
Conrect wroog
100
Correct Wrong
90
Werong
90
90
90
Agtsement (56)
80
80
70
70
70
60
60
50
54
50
40
30
1.0
20
30
20
30
20
30
Layer
Layer
Figure 9: Detailed results for each dataset and model of the third panel in Figure 3: pattern variations in agreement values (representing the ratio of data instances where the latent predictions match the final predictions) across layers.
22

---

Llama-2-78
Uama-2-138 Boolg
Mistral-78 BoolQ
Mixtral-Ex7B BoolQ
9.8
0.8
0.5
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.1
20
40
80
100
20
40
60
80
100
20
60
80
100
20
60
100
Coin
CoinFlip
CoinFiip
1.0
1.0
0.9
0.8
0.8
0.6
piceresth
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
20
40
60
80
100
20
40
60
80
100
20
40
60
100
20
40
80
100
Llama-2778 PrOntoQA
PrOntoQA
1.0
9.6
0.6
0.8
0.4
0.3
0.4
0.7
0.2
0.1
0.1
40
60
80
100
20
80
20
60
100
20
100
ProofWriter
0.6
0.6
0.7
0.8
0.5
0.5
9.6
0.4
0.4
0.5
0.3
0.4
0.2
0.2
8.1
0.1
40
80
29
80
20
63
20
80
100
Consistercy 1961
Constetency
Figure 10: Detailed results for each dataset and model of the fourth panel in Figure 3: a calibration plot with bins according to the model's 's internal consistency on the x-axis and the accuracy within each bin the y-axis.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and Section I match experimental results in Section 2.2 and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

This paper does not include theoretical results. We focus on empirical findings and analysis about internal consistency in CoT reasoning.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We present implementation details that are sufficient to reproduce the main experimental results in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due policy constraints, unable to provide the code with submis- sion. We provide sufficient implementation details for readers to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https:/
cc/
public/ guides Subom s cessoooooiiiy olicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See NeurlPS code and data submission guidelines (https: //nips. cc/ scc/publiee / segidde////.e Sodeeuumam ssi onPol enPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies the details necessary to understand the results in Ap- pendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include standard deviation in our main results to show the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the resources in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: neurips cc/ cc/publllccecccccccccceeecccecec c/Eth csGuide inse?
Answer: [Yes]
Justification: We are guided by the NeurIPS Code of Ethics when conducting this research. For example, we clarify the datasets used, their sources, and their licensing Appendix B.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts of our work in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not introduce new models or data and therefore poses no such risks.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper properly credits the assets used, mentions their licenses, and respects them. For more details, please refer to Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thoode com/ dat ssets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets are well documented Appendix B.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "mw9BpTIhDp", "reviewer": "Reviewer_B2Yn", "summary": "The paper explores the concept of \"internal consistency\" in Large Language Models (LLMs), mainly focusing on chain-of-thought (CoT) reasoning, where models articulate step-by-step rationales. The authors probe internal representations of LLMs to assess the alignment between middle and final layer outputs and propose boosting reasoning paths with higher internal consistency. Through studies across various models and datasets, authors demonstrate that models enhanced with internal consistency metrics reach significant improvement on performances in reasoning tasks.", "questions": "1. Can you briefly explain why low LLM layers (<10) can be used to show internal consistency?\n2. With high internal consistency, the results from high LLM layers (>20) may be the same as the last layer. Can we remove the last few layers to reduce the computational complexity of LLMs and increase their reasoning speed? In other words, can we use the method to reduce the size of LLMs?\n3. How does internal consistency interact with other known issues in LLMs, such as hallucination or bias in data?", "limitations": "This paper appropriately describes its limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Leveraging internal consistency within LLMs to improve reasoning performance is novel and promising. It addresses the issue of LLMs producing reasoning paths that are not always aligned with their final predictions.\n2. The paper provides validations across multiple models and datasets, enhancing the confidence of the results.\n3. By demonstrating how internal consistency can be used to calibrate CoT reasoning, the paper contributes practical techniques for improving LLM reliability, which is crucial for applications requiring precise and trustworthy outputs.", "weaknesses": "1. The techniques described require accessing intermediate layer outputs and could be computationally expensive, limiting the scalability of the proposed methods, especially in resource-constrained environments.\n2. There is a risk that optimizing for internal consistency might lead models to overfit specific reasoning patterns or datasets, potentially reducing the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews\\! We appreciate your recognition of our method as “novel and promising”. We aim to address your concerns in the following response.\n\n**W1: The proposed techniques could be computationally expensive**  \nThe proposed techniques incur minimal computational overhead compared to decoding. To assess internal consistency (IC), we apply the model's unembedding matrix to the hidden states of each intermediate layer associated with the answer token, once per reasoning path. Given that these hidden states are computed during decoding, they can be repurposed for evaluating IC without additional computation.\n\nFor a more detailed analysis, we estimate the proportion of computation dedicated to assessing IC in Llama-2-7b. The FLOPS per token during decoding amount to approximately $2P$, where $P$ represents the total number of parameters. For calculating IC, this is bounded by the normalization step prior to unembedding, with a time complexity of $O(dL)$, with $d$ being the hidden size and $L$ being the number of layers. Since $dL \\\\ll P$, the computation overhead is negligible. It is important to note that because IC is computed only once per reasoning path, the actual proportion of allocated computation is very low (\\~0.01% of the time required to decode a reasoning path). Below are statistics from one of our setups:\n\n|  | IC Calculation | Decoding per Token | Decoding per Path |\n| :---- | :---- | :---- | :---- |\n| Time (ms) | 0.249648 | 34.9162 | 2283.64 |\n\n*Time complexity comparison between IC calculation and decoding; benchmarked with Llama-2-7B on PrOntoQA.*\n\n**W2: The overfitting risk**  \nWe would like to clarify that our method requires no training process to calculate IC and apply IC in reasoning. Without tuning on specific data, our method consistently achieves better results over baselines on various models and datasets, validating the generalizability of our method.\n\nIn response to your interest in potential overfitting risks when optimizing for specific data, we conduct additional experiments on tuning weights assigned to each layer during IC calculation (see a more detailed setup in our global response). Despite the minimal number of parameters ($L$, the number of layers) involved, tuning weights yields substantial enhancements on various datasets and models. Importantly, the optimized weights are also adaptable to different datasets, indicating a universal pattern in internal representations that can be leveraged across multiple tasks.\n\n**Q1: Why use lower layers?**  \nThe use of lower layers is motivated by the intuition that predictions with greater certainty can converge earlier in latent predictions. This intuition is supported by Figure 3c, which shows that lower layers exhibit a higher agreement rate for correct predictions than for wrong ones. Consequently, we emphasize consistency from a *comparative perspective*: A stronger consistency between the latent predictions of lower layers and the final prediction indicates higher certainty and thus can be used to weigh reasoning paths.\n\nIndeed, Figure 3c also suggests that lower layers (\\<10) contain less semantic information and are relatively noisy. We conduct a new experiment to exclude lower layers for IC calculation, which results in an improvement for Llama-2 models. However, we also note that it introduces the need to specify layers to exclude and we plan to discuss it along with the new tuning results in the revised version.\n\n|  | Llama-2-7B | Llama-2-13B | Mistral-7B | Mixtral-8×7B |\n| :---- | :---- | :---- | :---- | :---- |\n| SC+IC | 62.4 | 70.6 | 76.1 | 80.6 |\n| SC+IC (w/o lower) | 62.8 | 71.1 | 76.2 | 80.7 |\n\n*Comparison of calculating IC on all layers versus on layers higher than 10\\. We report calibrated accuracy averaged over different datasets.*\n\n**Q2: Removing the last few layers to reduce computational complexity?**  \nYes, it is possible to reduce computational complexity by implementing an early-exiting strategy based on IC, using a proper stopping criterion like saturation \\[1\\]. Nonetheless, our primary aim is to explore how IC can improve reasoning capabilities rather than accelerate inference, which has been the focus of prior research \\[1, 2\\].\n\n**Q3: Interaction between internal consistency with issues like hallucination or bias in data?**  \nIC serves as a metric for calibration, aiding LLMs in expressing their confidence in the prediction. This is essential for mitigating issues like hallucination or bias, by detecting possible hallucinated or biased predictions with IC and correcting them accordingly \\[3\\]. In this regard, our approach to calibration in reasoning aligns well with hallucination and bias detection. We will provide more detailed discussion on this interaction in the final version.\n\n\\[1\\] Schuster T, Fisch A, Gupta J, et al. Confident adaptive language modeling\\[J\\]. Advances in Neural Information Processing Systems, 2022, 35: 17456-17472.  \n\\[2\\] Raposo D, Ritter S, Richards B, et al. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models\\[J\\]. arXiv preprint arXiv:2404.02258, 2024\\.  \n\\[3\\] Pan L, Saxon M, Xu W, et al. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies\\[J\\]. arXiv preprint arXiv:2308.03188, 2023\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "25XfVhcRTL", "reviewer": "Reviewer_7a9f", "summary": "This paper investigates the use of internal consistency as a measure for calibration, and to improve the reasoning capabilities of LLMs. The internal consistency is a measure of the model’s confidence by examining the agreement of latent predictions decoded from intermediate layers, and this measure does not require additional training. The authors conduct extensive empirical studies across various models and datasets, demonstrating that internal consistency effectively distinguishes between correct and incorrect reasoning paths.", "questions": "1. Will the authors provide more probing results regarding the suitability of lower layers for prediction and the proposed measurement?\n2. Will the authors include results using other approaches for reasoning?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed measurement is practical, offering a new way to self-evaluate and calibrate reasoning without additional training or human annotations.\n2. The authors conduct extensive empirical studies across different models and datasets. \n3. This paper is also well-structured and clearly presents its methodology, experiments, and findings.", "weaknesses": "1. The proposed measurement is rather straightforward, using an indicator function to measure if latent predictions match the final prediction and summing over all intermediate layers.\n2. The authors claim that the proposed measure is more accurate than logit-based methods and does not require additional training. However, Table 1 does not show a significant performance gap.\n3. The evaluation is conducted on specific datasets involving true-or-false questions, which may not fully represent the diversity of reasoning tasks encountered in practical applications.\n4. The paper dedicates considerable space to explaining the Transformer architecture, which seems unnecessary given the target reader's background. The listed equations for Transformer architecture and internal representations are not utilized in later sections. In contrast, the authors should elaborate more on the decoding process using the proposed consistency score, which is only briefly mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive and insightful feedback\\! We appreciate your effort in reviewing our paper and are committed to addressing your concerns in the following response.\n\n**W1: Internal consistency is rather straightforward**  \nIndeed, internal consistency (IC) serves as a straightforward measurement for calibration. We consider simplicity a valuable aspect of our approach, given that it incurs minimal computational overhead (approximately 0.01% additional time as discussed in the response to Reviewer B2Yn) and can be seamlessly integrated into various models without the need for tuning.\n\nTo better utilize internal consistency for calibration measurement, we conduct new experiments that tune the weights assigned to intermediate layers during IC calculation. For a comprehensive setup and analysis of the results, please see our global response and our response to W2.\n\n**W2: Table 1 does not achieve great improvement**  \nWe note that `SC+IC` is an unsupervised method that is not tuned on specific data. While the improvements are not substantial, we observe consistent performance gains over `SC` across models and datasets, especially on tasks that require more complex reasoning.\n\nThe improvements can be lifted by applying tuned weights assigned to intermediate layers. As shown in Table 1 of the attached pdf, using a weighted voting of latent predictions can drastically improve performance. Perhaps more interestingly, the tuned weights are transferable across datasets, indicating a universal pattern of internal representations for calibration.\n\n**W3: True-or-false questions may not fully represent the diversity of reasoning tasks**  \nFollowing a similar analytical protocol in previous work \\[1, 2\\], we focus on the true-or-false questions for evaluation in the original paper. To increase the diversity of reasoning tasks, we add new experiments on SWAG \\[3\\] and MathQA \\[4\\]. As shown in Table 1 of the attached pdf, our method consistently outperforms the baselines, especially on the more challenging MathQA dataset. Notably, `SC+IC (transfer)` demonstrates robust transferability even though its weights are optimized for binary questions in PrOntoQA, confirming the effectiveness of internal representations for reasoning.\n\n\t  \n**W4: Paper organization**  \nThank you for your constructive feedback\\!\n\nThe background section on the Transformer architecture aims to establish notations and introduce readers who are unfamiliar with it. However, we acknowledge that this section is overly lengthy and contains redundant information. In the final version, we will reorganize the structure to enhance clarity and presentation.\n\n**Q1: More probing results regarding the suitability of lower layers**  \nWe apply probing in our preliminary analysis of internal consistency. While the pattern observed is valid, we find that probing cannot reliably indicate subtle differences in lower layers–a limitation inherent to probing \\[5\\]. Therefore, we instead focus on CoT experiments for evaluation.\n\nRegarding the suitability of lower layers, please refer to our detailed response to Q1 from Reviewer B2Yn, where we present new experimental results and discuss the use of lower layers.\n\n**Q2: Results using other approaches for reasoning**  \nWe conduct new experiments using least-to-most (L2M) prompting \\[6\\], a representative method other than chain-of-thought reasoning to elicit reasoning. We sample 4 instances from each dataset (8 for coinflip) as in-context learning examples, following the setup in our original paper, and use GPT-4o to generate subproblems and answer to each subproblem. We manually check the prompts to ensure they are correct. The results below indicate that integrating IC into L2M prompting consistently improves reasoning performance, confirming the efficacy of our approach.\n\n|  | L2M | SC | SC+$\\\\Delta$ | SC+IC | SC+IC (tune) |\n| :---- | :---- | :---- | :---- | :---- | :---- |\n| Llama2-7B | 63.8 | 69.8 | 70.3 | 71.0 | 73.4 |\n| Llama2-13B | 73.5 | 79.1 | 79.6 | 79.4 | 80.8 |\n| Mistral-7B | 74.6 | 78.4 | 79.2 | 79.1 | 81.2 |\n| Mixtral-8x7B | 78.7 | 83.1 | 83.6 | 83.2 | 84.4 |\n\n*Results of least-to-most prompting. We report calibrated accuracy averaged over four datasets.*\n\n\\[1\\] Li K, Patel O, Viégas F, et al. Inference-time intervention: Eliciting truthful answers from a language model\\[J\\]. Advances in Neural Information Processing Systems, 2024, 36\\.  \n\\[2\\] Halawi D, Denain J S, Steinhardt J. Overthinking the truth: Understanding how language models process false demonstrations\\[J\\]. arXiv preprint arXiv:2307.09476, 2023\\.  \n\\[3\\] Zellers R, Bisk Y, Schwartz R, et al. Swag: A large-scale adversarial dataset for grounded commonsense inference\\[J\\]. arXiv preprint arXiv:1808.05326, 2018\\.  \n\\[4\\] Amini A, Gabriel S, Lin P, et al. Mathqa: Towards interpretable math word problem solving with operation-based formalisms\\[J\\]. arXiv preprint arXiv:1905.13319, 2019\\.  \n\\[5\\] Alain G, Bengio Y. Understanding intermediate layers using linear classifier probes\\[J\\]. arXiv preprint arXiv:1610.01644, 2016\\.  \n\\[6\\] Zhou D, Schärli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models\\[J\\]. arXiv preprint arXiv:2205.10625, 2022\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtEg9Q3U69", "reviewer": "Reviewer_reW1", "summary": "This paper first reveals that inconsistencies emerge between the internal representations in middle layers and those in final layers, which might undermine the reliability of reasoning processes, and then proposes internal consistency to measure the model’s confidence. Furthermore, the authors propose a method to calibrate CoT reasoning by up-weighting reasoning paths with high internal consistency, which boosts reasoning performance.", "questions": "- Intuitively, as intermediate layers get closer to the final layer, their predictions should increasingly match the final predictions. Why do you think they need to be internally consistent?", "limitations": "see above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper identifies the inconsistencies between intermediate and final layer representations in LLM reasoning, and proposes a measure of internal consistency to evaluate the model’s confidence and calibrate CoT reasoning.\n- Extensive results across different models and datasets show that the proposed measure can effectively distinguish between correct and incorrect reasoning processes.\n- The authors propose a new method to calibrate CoT reasoning by up-weighting reasoning paths with high internal consistency, thereby improving reasoning performance.", "weaknesses": "- The motivation for this paper doesn't sound convincing enough (see Questions part).\n- The improvements in Table 1 seem to be limited to some models/datasets. Although the authors performed ten runs with different random seeds, they mentioned that each dataset contains at least 500 samples, which means that the results may vary between 0-10 samples. I suggest the authors report detailed data statistics.\n- This paper includes extensive experiments and results, but lacks analysis and description; Readers need to observe for themselves, such as Figure 3.\n- In Figure 3 (a), the text on the x-axis is overlapping.\n- For Figure 2 (b) and (d), it is unclear how the different internal consistency results (0, 25, 50, 75, 100) were obtained. For example, the authors mentioned that results are averaged over all models and datasets, so (a) shows results corresponding to different prompting methods, and (c) shows results corresponding to different layers. \n- The authors describe Figure 6 in the main text, but the figure is placed in the appendix (of course this may be improved in the final version).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for rating our paper positively\\! We sincerely appreciate your observant review and insightful suggestions and aim to address your concerns in the following response.\n\n**W1: Motivation behind internal consistency**  \nWe would like to clarify that we do not expect latent predictions to be internally consistent, but to use consistency as a measurement for calibration. More specifically, our study on internal consistency is motivated by our preliminary analysis that inconsistency emerges at middle layers and is exacerbated in CoT reasoning. This leads to an intuition that predictions with greater certainty can converge earlier in latent predictions. Consequently, we emphasize consistency from a comparative perspective: A stronger consistency between the latent predictions at intermediate layers and the final prediction indicates higher certainty and thus can be used to weigh reasoning paths for CoT.\n\nWe apologize for any confusion caused by insufficient clarity regarding our motivations and will revise these sections for improved clarity in the final version.\n\n**W2: The improvements in Table 1 seem to be limited to some models/datasets & Missing detailed data statistics**\n\nWe would like to highlight that, although the improvement is relatively small for datasets such as BoolQ, consistent performance enhancements are evident across various models and datasets. The gains are particularly pronounced in more challenging reasoning tasks, such as logical reasoning. Given that `IC` is an unsupervised method, we consider these results valid in demonstrating the calibration effectiveness of internal consistency.\n\nOur method can achieve further improvements by applying tuned weights assigned to intermediate layers. As shown in Table 1 of the attached pdf, using a weighted voting of latent predictions can drastically improve performance. Perhaps more interestingly, the tuned weights are transferable across datasets, indicating a universal pattern of internal representations for calibration.\n\nRegarding detailed data statistics, we place them in the Appendix B.2 of the original paper, including specific numbers of samples and other details.\n\n**W3: Results lack analysis and description**  \nWe apologize for any confusion caused by the lack of analysis and are happy to discuss with you on specific contents that you feel unclear.\n\nFor Figure 3, the panels sequentially display: a) The boxplot describing the distribution of internal consistency under each prompting setting, highlighting the emergence of inconsistency in CoT setting. b) The distribution of internal consistency level when the model gives correct and wrong predictions, and the discrepancy suggests internal consistency can be an indicator of prediction accuracy. c) The discrepancy of the ratio of data instances where the latent predictions match the final ones across layers, which further illustrates incorrect reasoning paths lead to lower consistency level. d) A plot describing the prediction accuracy under each internal consistency level, shows that internal consistency is a reliable metric for calibration. We will elaborate the description in the final version.  \n**W4-W6: Issues with the figures**  \nThank you for your constructive feedback\\! We will reorganize the figures in our final version based on your suggestions. \n\nRegarding Figures 3(b) and 3(d), we calculate internal consistency for each sampled reasoning path and associate these values to the accuracy of the final predictions. The results are presented as histograms, where each bar corresponds to a specific bin.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zyj3CYwZMs", "reviewer": "Reviewer_fPrR", "summary": "This paper investigates the correlation between CoT reasoning and internal representations in Transformer models. It shows that there are inconsistencies between model’s internal representations in different layers. Such inconsistencies can be used to distinguish correct and incorrect reasoning paths, and thereby calibrate model predictions. By incorporating internal consistency into self-consistency (SC+IC), the authors show that SC+IC improves over SC under the same number of sampled reasoning paths. Besides, the authors also provided insights into how internal inconsistency is caused by Transformer components.", "questions": "- Why do we measure internal consistency against the final prediction? How about measuring the consistency between every pair of layers?\n- Can internal consistency be applied to non-binary answers? Will it be still effective in that case?\n- Line 210: By the term “decreases”, do you mean internal consistency is lower for few-shot CoT than few-shot direct prompting?\n- Line 241-242: focusing on the answer tokens → focusing on what the answer tokens attend to.\n- Line 248: second → last FFN matrix in the FNN layer. This would be clearer to the readers.\n- Figure 5 caption: Are attention weights and number of top value vectors supposed to align? Why?", "limitations": "Yes. The authors has discussed the limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The correlation between internal representations and CoT reasoning is a novel topic and the conclusion of this paper would be useful for a wide range of audience working on CoT prompting.\n- This paper has a clear logic and the experiment design is solid. The authors first observe inconsistency in internal representations, then propose the internal consistency metric, and show that the metric is correlated with prediction correctness.\n- The authors conduct experiments on 4 different datasets and 2 family of models, showing that SC+IC achieves robust improvement over both SC and SC+$\\Delta$ at almost no additional cost.", "weaknesses": "- The improvement of SC+IC over SC is relatively marginal (+1.2% on avearge). This might be due to the fact that the authors just simply used internal consistency $\\in [0, 1]$ as weights in SC. Maybe the authors can try some other weighting strategies (e.g. exponentiating the internal consistency) and obtain better results.\n- Section 4.4 is not clear to me. Could you explain why attention weights and number of top value vectors reveal the source of internal inconsistency? Could you elaborate what the probe vector is and how you obtained it? It is not very clear to me even after reading the appendix.\n- It is hard to understand some important details (e.g. Figure 6, self-attention layers and FFN layers) without looking into the Appendix. I would recommend the authors to reorganize them in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for rating our paper positively\\! Your efforts in reviewing our paper and providing profound advice are sincerely appreciated. We would like to address your remaining concerns in the following response.\n\n**W1: Marginal improvement of `SC+IC` over `SC`**  \nThank you for your constructive feedback. We do not observe significant improvement by exponentiating the internal consistency. Nonetheless, we observe significant improvement by applying tuned weights assigned to intermediate layers (please see our global response for details). As shown in Table 1 of the attached pdf, using a weighted voting of latent predictions can drastically improve performance. Perhaps more interestingly, the tuned weights are transferable across datasets, indicating a universal pattern of internal representations for calibration.\n\n**W2: Section 4.4 is unclear**  \nAs illustrated in the caption of Figure 5, layers with high attention weights on critical tokens do not align with those promoting certain predictions. Such misalignment can result in discrepancy of behavior at different layers, and leads to internal inconsistency. After applying the logistic regression with respect to the last hidden state and the final output (True/False) of the model (as detailed in Appendix B.5.2), the probe vector is the weight matrix (a vector in this case as the labels are binary) of the logistic regression model.\n\nWe apologize for any confusion caused and will polish Section 4.4 following your suggestions in the final version.\n\n**W3: Necessary details for the main paper**  \nThank you for your constructive feedback. We will properly reorganize the details and put some important details forward in the main paper for better reading experience.  \n**Q1: Why against the final layer?**  \nThank you for your suggestion. We choose the final layer because the final layer gives the natural prediction of the whole model, with minimal modification of the original mechanism of the model. While measuring the consistency between every pair of layers is reasonable, we do not observe improvement in the cost of time complexity.\n\n**Q2: Applicability of IC to non-binary answers?**  \nWhile our analysis in the original paper focuses on datasets with true-or-false answers, the concept of internal consistency can be natually extended to non-binary answers. We add new experiments on SWAG \\[1\\] and MathQA \\[2\\]. Please refer to our global response for details. As shown in Table 1 of the attached pdf, our method consistently outperforms the baselines, especially on the more challenging MathQA dataset. Notably, `SC+IC (transfer)` demonstrates robust transferability even though its weights are optimized for binary questions in PrOntoQA, confirming the effectiveness of internal representations for reasoning.  \n**Q3-Q5: Writing requiring clarification**\n\nLine 210 “decrease”: Actually we mean longer reasoning paths tend to result in lower internal consistency, as illustrated in Section 2.2 and Figure 2\\.\n\nLine 241-242 and line 248: Thank you for your advice. Your rephrasing is much clearer than our original expression and we will correct them following your advice.\n\n**Q6: Confusion about Figure 5's caption**\n\nActually we do not expect or suppose the attention weights and number of top value vectors to align. Section 4.4's analysis seeks to explore how transformer components contribute to internal consistency, with the misalignment between these factors offering insight into why internal inconsistency arises.\n\n\\[1\\] Zellers R, Bisk Y, Schwartz R, et al. Swag: A large-scale adversarial dataset for grounded commonsense inference\\[J\\]. arXiv preprint arXiv:1808.05326, 2018\\.  \n\\[2\\] Amini A, Gabriel S, Lin P, et al. Mathqa: Towards interpretable math word problem solving with operation-based formalisms\\[J\\]. arXiv preprint arXiv:1905.13319, 2019\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We greatly appreciate the detailed feedback from the reviewers. Your insightful suggestions have significantly inspired us to enhance our draft. We are committed to address the reviewers’ concerns on a point-by-point basis. Regarding the common concerns raised by the reviewers, we have conducted additional experiments:

**New results on multi-choice SWAG and MathQA datasets**. We sample the first 500 instances from the train split of each dataset and follow the same setup in the original paper to conduct chain-of-thought experiments. These datasets with non-binary answers naturally extend the evaluations and support the effectiveness of our method.  
**New results on variants of SC+IC.** We consider a weighted voting method for IC in Section 3.2: Instead of assigning equal weights to different layers, we apply a weight vector $\\mathbf{w} \\in \\mathbb{R}^{L}$ for aggregation. We consider two variants `SC+IC (tune)` and `SC+IC (transfer)`. For each dataset, we use a small number of 500 held-out samples to tune weights assigned to intermediate layers for internal consistency calculation. We use the Adam optimizer and a learning rate of 0.01 with 1000 optimization steps. For `SC+IC (transfer)`, we apply the same weights tuned for PrOntoQA to different datasets. We observe significant improvement and great transferability, albeit that the number of introduced parameters is only the number of layers.

The additional results are presented in the attached pdf. In Table 1, we provide an updated version of our main results including the new datasets and variants. In Figure 1, we show the tuned weights for different models, showing a relatively important role of middle layers in contributing the effectiveness of internal consistency.

Meta-review: The authors evaluate internal representations to align middle and final layer outputs, suggesting improvements to reasoning paths using higher internal consistency. Studies across various models and datasets show that enhancing internal consistency metrics significantly boosts performance in reasoning tasks.
Reviewers agree that the novelty of the internal consistency idea.
Decision: Accept (poster)