Title: Calibrating Reasoning in Language Models with Internal Consistency
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large language models

reasoning

faithfulness

internal consistency
Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought prompting that elicits verbalized reasoning. However, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales. In this work, we investigate reasoning in LLMs through the lens of internal representations, focusing on how these representations are influenced by generated rationales. Our preliminary analysis reveals that while generated rationales improve answer accuracy, inconsistencies emerge between the model’s internal representations in middle layers and those in final layers, potentially undermining the reliability of their reasoning processes. To address this, we propose internal consistency as a measure of the model’s confidence by examining the agreement of latent predictions decoded from intermediate layers. Extensive empirical studies across different models and datasets demonstrate that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Motivated by this, we propose a new approach to calibrate reasoning by up-weighting reasoning paths with high internal consistency, resulting in a significant boost in reasoning performance. Further analysis uncovers distinct patterns in attention and feed-forward modules across layers, providing insights into the emergence of internal inconsistency. In summary, our results demonstrate the potential of using internal representations for self-evaluation of LLMs.

Calibrating Reasoning in Language Models with Internal Consistency

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought prompting that elicits verbalized reasoning. However, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales. In this work, we investigate reasoning in LLMs through the lens of internal representations, focusing on how these representations are influenced by generated rationales. Our preliminary analysis reveals that while generated rationales improve answer accuracy, inconsistencies emerge between the model's internal representations in middle layers and those in final layers, potentially undermining the reliability of their reasoning processes. To address this, we propose internal consistency as a measure of the model's confidence by examining the agreement of latent predictions decoded from intermediate layers. Extensive empirical studies across different models and datasets demonstrate that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Motivated by this, we propose a new approach to calibrate reasoning by up-weighting reasoning paths with high internal consistency, resulting in a significant boost in reasoning performance. Further analysis uncovers distinct patterns in attention and feed-forward modules across layers, providing insights into the emergence of internal inconsistency. In summary, our results demonstrate the potential of using internal representations for self-evaluation of LLMs.

1 Introduction

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought (CoT) prompting that elicits verbalized reasoning (Wei et al., 2022; Merrill and Sabharwal, 2024). This approach directs the model to articulate step-by-step rationales before answering a question, simulating the reasoning process used by humans. With these verbalized rationales, it is expected that not only will the model's problem-solving capabilities be enhanced, but also the interpretability of its predictions will improve. Understanding how LLMs reason is essential for aligning them with human values (Bai et al., 2022; Li et al., 2024).

Despite the continued improvement in performance and the emergence of new capabilities, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales (Ye and Durrett, 2022; Turpin et al., 2023). One notable failure mode is unfaithful reasoning, where LLMs provide rationales that contradict their final predictions (Lyu et al., 2023; Lanham et al., 2023). This makes it difficult to determine the trustworthiness of their predictions, highlighting the need for effective calibration methods to assess the reliability of rationales.

Is the average temperature in the Himalayas lower than

that in New York in summer?

The Himalaya is closer to the

equator compared to New

York. Regions with lower

latitude tend to be hotter.

So, the answer is False

The Himalayas' elevation is

much higher than New York.

Higher elevations typically

have cooler temperatures.

So, the answer is True

Consistency=#F/#Layers=55%

Consistency=#T/#Layers=85%

HUMAN

ASSISTANT

TTTFT..TT..FTT

Layer

Latent Predictions

TFTTF..FT..TTF

Layer

Latent Predictions

ASSISTANT

✅

❌

Correct

wrong

Density

InternalConsistency

In this work, we present the first exploration of leveraging internal representations to calibrate reasoning in LLMs. Our study is based on the intuition that the inner workings of LLMs contain latent structures that can assess the reliability of reasoning paths. To investigate this, we begin by probing the model's intermediate activations for answer correctness in CoT reasoning. Our preliminary analysis reveals that CoT reasoning leads to inconsistencies between the model's internal representations in middle layers and those in final layers, potentially indicating a degree of uncertainty.

Based on this observation, we propose internal consistency (illustrated in Figure 1) as a measure of the model's confidence in the reasoning process. Inspired by previous works on eliciting latent predictions from intermediate layers (nostalgebraist, 2020; Belrose et al., 2023), we examine internal consistency by assessing the agreement of latent predictions decoded from intermediate layers. Unlike methods that require additional training and human annotations (Ye and Durrett, 2022; Khalifa et al., 2023), internal consistency provides a reliable and off-the-shelf self-evaluation for reasoning.

We conduct extensive empirical studies on various reasoning tasks, including reading comprehension, symbolic reasoning, and logical reasoning. Our analysis across different models and datasets demonstrates that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Moreover, by up-weighting reasoning paths with high internal consistency, we achieve a significant improvement in reasoning performance. These results highlight the potential of using internal representations for the self-evaluation of LLMs.

This work makes three main contributions. 1) We identify the emergence of inconsistencies between intermediate and final layer representations in LLM reasoning, highlighting a potential issue where CoT reasoning leads to higher uncertainty. 2) We propose internal consistency as a novel measure to evaluate the model's confidence and calibrate reasoning by up-weighting paths with high internal consistency. 3) We provide insights into the cause of internal inconsistency in reasoning by analyzing Transformer components (i.e., attention and feed-forward networks) across layers. We believe our results show promise in leveraging internal representations to enhance reasoning in LLMs.

2 Preliminaries

In this section, we present background information, notations, and preliminary analysis to provide context for our study.

2.1 Transformer Architecture

Our analysis focuses on the prevalent decoder-only Transformer architecture (Vaswani et al., 2017; Radford et al., 2018). To set notation and context, we briefly describe the key components as follows.

Given a sequence x = ( x 1 , . . . , x n ) of input tokens, the inference process of the Transformer begins by projecting these tokens into a sequence of representations h 0 1 , . . . , h 0 n ∈ R d . This representation is then updated by a series of L residual blocks (Elhage et al., 2021), each consisting of a multihead self-attention (MHSA) layer followed by a feed-forward network (FFN) layer 2 . In each block ℓ ∈ [0 , L -1] , the representation of each token i is updated as follows:

Finally, the next token distribution is produced by applying an unembedding on h L i :

As the dimension of h i remains constant across layers, we can view it as a dynamic distribution processed by the model (Geva et al., 2022), providing insight into how LLMs reason internally.

2.2 Preliminary Analysis of Internal Representations in CoT Reasoning

Building upon works that explore the inner workings of LLMs (Burns et al., 2023; Ferrando et al., 2024), our study starts with exploring the encoded information in internal representations during reasoning. In the context of CoT prompting-a prevalent technique for eliciting reasoning capabilities of LLMs-we analyze these representations along two dimensions: horizontally across reasoning steps, and vertically across network layers. Zoom-in Evaluation 0.47 0.67

Weconduct preliminary experiments using the Llama2-7B model (Touvron et al., 2023) on the PrOntoQA dataset (Saparov and He, 2023), which challenges the model to determine whether a proposition is true based on a series of assumptions. We apply 4-shot CoT prompting and greedy decoding to generate reasoning paths and predictions. To discern encoded information crucial for reasoning, we utilize linear probing (Alain and Bengio, 2016; Belinkov, 2022) to train classifiers on the activations with respect to the ground truth labels, aiming to differentiate between true and false propositions. Reasoning paths are segmented into individual steps using the NLTK punkt sentence tokenizer (Bird et al., 2009), with a dedicated probe classifier for each layer at the last

0 1 2 3 4 5 6 7

Reasoning Step

0

10

20

30

Layer

0

10

20

30

Layer

0.50

0.55

0.60

0.65

Probe Accuracy

Step 6

Step 7

token of every reasoning step. Implementation details are provided in Appendix B.1.

Figure 2 presents an evaluation of the probe accuracy on the validation set, including a detailed zoom-in on the probe accuracy at the last two reasoning steps, which reveals two distinct patterns:

Improved accuracy through verbalized reasoning: Probe accuracy increases monotonically as the model processes and verbalizes reasoning paths, indicating that LLMs extract and utilize task-solving information from their generated rationales.

Emergent inconsistency across layers: Acloser examination at the last two reasoning steps reveals a notable inconsistency in probe accuracy between middle and later layers. This inconsistency suggests that while middle layers capture essential reasoning information, it may not be fully utilized or maintained by the later layers, potentially impacting the model's overall reasoning performance.

This analysis provides an initial understanding of how LLMs internally process and handle information during CoT reasoning. While LLMs effectively gather information from verbalized reasoning paths, there is a noticeable tendency for these models to underutilize the processed information at intermediate layers. This motivates us to further investigate into the internal representations of LLMs and explore methods to better harness them to enhance reasoning capabilities.

2.3 Calibration

Calibration refers to the alignment between a model's predicted probability estimates and their actual prediction correctness (Guo et al., 2017; Geng et al., 2023). Calibration is crucial for assessing the reliability and trustworthiness of LLMs, and can also help improve performance (Wang et al., 2022).

More formally, we want to find a measure Score such that for any two prompt-answer-prediction triplets ( x i , y i , ˆ y i ) and ( x j , y j , ˆ y j ) , the following holds:

In this work, we reveal the potential of using internal representations for calibration and introduce a new measure based on internal consistency, as described in Section 3.2. The new measure does not require additional training and is more accurate than logit-based methods.

3 Internal Consistency as a Lens to Calibrate Reasoning

In Section 2.2, we uncover patterns of internal inconsistency in reasoning with linear probes. While probe accuracy indicates the 'extractability' of task-solving information, it does not necessarily show how LLMs process internal representations for generation (Alain and Bengio, 2016; Belinkov, 2022). Therefore, we introduce a new method to measure internal consistency in this section.

3.1 Interpreting Internal Representations

The internal reasoning process of a Transformer towards its final prediction can be comprehended through iterative inference (Jastrz˛ ebski et al., 2017; Geva et al., 2022). A direct method to inspect this internal process during generation is to early exit from intermediate layers. Specifically, instead of applying unembedding on h L i , we can obtain the token distribution over the vocabulary from any intermediate layer using the logit lens (nostalgebraist, 2020):

Building on this interpretation, we can decode latent predictions from any layer ℓ :

While Equation 1 provides a measurement of latent predictions, we find that the decoded distributions are often miscalibrated, biasing towards specific answers (Zhao et al., 2021; Belrose et al., 2023). For instance, in the PrOntoQA task with Llama-2-7B, the penultimate layer consistently assigns over 90% probability to the answer "True," regardless of the context. To address this, we balance the latent predictions across the possible answers for each layer separately. See Appendix B.4 for more details.

3.2 Calibration Measurement with Internal Consistency

Built on our findings in Section 2.2, we investigate consistency of internal representations as an indication of uncertainty. Intuitively speaking, if the model's internal representations are highly inconsistent with those of later layers, it may not faithfully say as it thinks. Based on these intuitions, we propose a simple metric called internal consistency to measure the agreement of latent predictions.

Internal Consistency During inference, we collect latent predictions at the answer tokens and measure how often latent predictions match the final prediction:

This measure offers a straightforward method to gauge the level of internal consistency, serving as a useful indicator of the output's correctness (i.e., calibration), without requiring additional training.

Calibrating Reasoning with Internal Consistency As LLMs often require long reasoning paths to resolve complex tasks (Wei et al., 2022), a process that introduces cumulative errors and uncertainty, we integrate this consistency measure into the generation process to enhance reasoning calibration. Specifically, we propose a new decoding method that assigns weights to each sampled reasoning path based on their calculated consistency score at the final answer token. Paths that exhibit higher internal consistency-indicating robust alignment between intermediate and final predictions-are assigned greater weight in determining the final model output. This weighted strategy aims to prioritize reasoning paths that not only maintain self-consistency but are also more likely to converge on the correct answer, thus potentially enhancing the model's accuracy in complex reasoning tasks.

4 Experiments

To study internal consistency, we conduct extensive experiments to address the following questions: 1) To what degree does internal consistency correlates with reasoning performance? 2) Can we leverage internal consistency to boost reasoning? 3) How do different components of the Transformer architecture contribute to the emergence of internal inconsistency?

4.1 Setup

We evaluate internal consistency across a spectrum of models and tasks. See Appendix B.2 for further details of the experimental setup.

Models Our experiments are conducted using two prominent series of open-source Transformerbased models: the Llama-2 series (Touvron et al., 2023) and the Mistral series (Jiang et al., 2023). Specifically, we evaluate both the 7B and 13B configurations of the Llama-2 series to understand how model scale impacts internal consistency. Additionally, for the Mistral series, we explore the 7B and the 8 × 7B versions, the latter of which incorporates mixture of experts layers (Jiang et al., 2024). These models were chosen due to their extensive use and distinct architectural features, enabling a comprehensive evaluation of internal consistency.

Datasets We evaluate various datasets that span reading comprehension, symbolic reasoning, and logical reasoning tasks. We choose these datasets because they involve explicit reasoning processes with unambiguous single-token answers (i.e., 'True' and 'False'), which facilitates our analysis. To standardize the evaluation process, each dataset is transformed into a true-or-false QA format. For a detailed description of the datasets and examples, please refer to Appendix B.2.

BoolQ (Clark et al., 2019): A reading comprehension dataset where each instance involves a yes/no question grounded in a related passage.

CoinFlip (Wei et al., 2022): A dataset that challenges the model's symbolic reasoning abilities by presenting a task where the model must determine the outcome of a coin (heads or tails) after a series of flips.

PrOntoQA (Saparov and He, 2023): A dataset designed for logical reasoning. Each question requires a 3-hop deduction to determine the truth value of a proposition based on a set of assumptions with fictional concepts.

ProofWriter (Tafjord et al., 2020): A logical reasoning dataset that, in contrast to PrOntoQA, uses real concepts for all assumptions. Each question requires 3 hops of reasoning.

We balance the labels and each dataset has at least 500 samples for evaluation. Our method requires no training procedure. To evaluate reasoning performance, we use calibrated accuracy following Zhao et al. (2021); Burns et al. (2023), balancing predictions to be 50/50 across the two labels.

4.2 Internal Consistency is a Good Calibration Measure

Our analysis begins by examining patterns of internal consistency across different layers of LLMs. As illustrated in Figure 6 (Appendix C), latent predictions exhibit variable convergence during inference; notably, there are significant increases in consistency in the middle and final layers, while the early layers are characterized by considerable noise. Additionally, these patterns vary across models and

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Internal Consistency (%)

Effect of Prompting

0

25

50

75

100

Internal Consistency (%)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Density

Correct

Wrong

0

10

20

30

Layer

50

60

70

80

90

100

Agreement (%)

Pattern Discrepancy

Correct

Wrong

0

25

50

75

100

Internal Consistency (%)

0.0

0.2

0.4

0.6

0.8

p(correct)

Calibration Curve

datasets. Smaller models, such as Llama-2-7B, show high variability in latent predictions across layers, indicating less stability in their reasoning processes. In contrast, larger models like Mixtral 8 × 7B demonstrate more robust internal reasoning, as evidenced by more reliable convergence of latent predictions. For BoolQ, the inconsistency is less pronounced compared to other datasets that require more complex symbolic and logical reasoning.

Furthermore, the correlation between internal consistency and model accuracy is particularly evident when distinguishing between correct and incorrect answers in CoT reasoning. As depicted in Figure 3, the impact of prompting techniques on internal consistency is significant, highlighting the emergence of inconsistency brought by CoT prompting as discussed in Section 2.2. While few-shot demonstrations generally enhance consistency, CoT prompting decreases it. A closer examination of CoT reasoning reveals that reasoning paths leading to incorrect answers typically exhibit lower internal consistency. These results underscore the potential of internal consistency not only as a diagnostic tool but also as a reliable measure for calibrating reasoning in LLMs.

4.3 Enhancing Reasoning with Internal Consistency

Our analysis reveals a strong correlation between internal consistency and prediction accuracy, suggesting its potential as a mechanism for improving reasoning performance. As demonstrated in Section 4.2, CoT prompting, while effective, tends to decrease internal consistency. This decrease is possibly attributable to the generated reasoning paths that incorporate incorrect rationales, which confuse the model to make inconsistent predictions across layers. To investigate the generality of this phenomenon and its potential solutions, we also least-to-most (L2M) prompting (Zhou et al., 2022), which decomposes complex problems into simpler sub-problems for reasoning. For a detailed description of our least-to-most decomposition approach, please refer to Appendix B.3.

To incorporate internal consistency, we build on the self-consistency ( SC ) approach (Wang et al., 2022), which utilizes an ensemble of multiple sampled reasoning paths to increase accuracy. Our proposed method, which we term SC+IC , integrates internal consistency to weight these reasoning paths accordingly. Specifically, if a reasoning path demonstrates high internal consistency in its answer prediction, the probability of accepting its corresponding answer as the final answer increases. We accumulate the sum of internal consistency scores across all reasoning paths for each answer and choose the answer with the highest sum. For baselines, we also compare with greedy decoding Greedy and a logit-based approach SC+ ∆ (Wang and Zhou, 2024) that selects the final answer based on a confidence measure ∆ k = p ( ˆ y 1 | x ) -p ( ˆ y 2 | x ) , where ˆ y 1 and ˆ y 2 represent the top two tokens for answer prediction of the k -th path.

Table 1 summarizes the results, which demonstrate that SC+IC consistently outperforms the others across different models and tasks, with improvements ranging between 1.8% to 4.9% across models. Figure 4 plots the calibrated accuracy with respect to varying numbers of sampled paths. We observe clear improvements of SC+IC over the baselines, highlighting the effectiveness of leveraging internal

consistency to enhance reasoning. This improvement is particularly notable in tasks involving symbolic and logical reasoning, which depend heavily on the correctness of reasoning paths. In addition, we observe that internal consistency effectively distinguish between correct and incorrect paths (e.g., in the PrOntoQA examples in Table 2, internal consistency helps filter out flawed rationales), validating our hypothesis on why internal consistency can enhance CoT reasoning.

4.4 Layer-weighted Aggregation Finds Transferable Patterns

While results of SC+IC have demonstrated the effectiveness of leveraging latent predictions for reasoning, the relative importance of different layers remains unexplored. Recent studies have shown that certain intermediate layers specialize in specific types of reasoning (Yang et al., 2024). These observations suggest that treating all layers equally when computing internal consistency might be suboptimal. Motivated by these insights, we introduce an aggregation strategy for internal consistency with two variants: SC+IC (tune) and SC+IC (transfer) . Specifically, instead of assigning equal weights to each layer in Equation 2, we consider a learned weight vector w ∈ R L (where L is the number of layers) for aggregation. For SC+IC (tune) , we optimize layer weights using 500 held-out samples per dataset. The weights are learned using Adam optimizer with a learning rate of 0.01 for 1,000 iterations. SC+IC (transfer) evaluates cross-task generalization by applying weights tuned on PrOntoQA to other datasets.

The results are presented in Table 1 and Figure 4. Despite introducing only L parameters, this weighted approach yields substantial improvements. Notably, the strong performance of SC+IC

0

10

20

30

40

Number of Votes

75.0

76.0

77.0

78.0

Accuracy (%)

Reading Comprehension

SC

SC+

SC+IC

SC+IC (transfer)

0

10

20

30

40

Number of Votes

85.0

86.0

87.0

88.0

89.0

90.0

0

10

20

30

40

Number of Votes

59.0

60.0

61.0

62.0

Logical Reasoning

(transfer) across diverse tasks indicates that optimal layer importance patterns generalize well, demonstrating the broad applicability of internal consistency for reasoning tasks.

4.5 Investigating the Role of Transformer Components in Internal Inconsistency

Having observed that CoT prompting exaggerates internal inconsistency and that reasoning performance can be improved by re-weighting reasoning paths based on internal consistency, we now aim to understand why internal inconsistency emerges from CoT reasoning. Specifically, we explore how different network components contribute to the final predictions in CoT reasoning and, consequently, the calibration effect of internal consistency. One hypothesis is that this effect arises from better elicitation of information processed in the middle layers, which tends to be underutilized in later layers. To investigate this, we conduct a detailed analysis focusing on the roles of self-attention and feed-forward network (FFN) layers.

Self-attention Layers We analyze the average attention weights across all attention heads at each layer, focusing on what the answer tokens attend to. Consistent with the CoT process, we segment the processed tokens into three parts: the context, the query, and the generated rationale. Further details on the methodology and calculations can be found in Appendix B.5.

FFN Layers For FFN layers, we use a probe vector trained on the model's last hidden state of the answer token with respect to the model's output. We compute cosine similarities between this probe vector and the value vectors (Geva et al., 2022; Lee et al., 2024), each corresponding to a column

Table 2: Examples where internal consistency helps select correct reasoning paths for Mistral-7B on PrOntoQA. We show the paths with the highest and lowest internal consistency. The question has been reformatted for better presentation. Incorrect steps are colored in gray, whereas wrong predictions are highlighted in red.

Question : Zumpuses are not transparent. Each zumpus is a wumpus. Wumpuses are red. Each wumpus is a jompus. Impuses are amenable. Every jompus is not spicy. Jompuses are tumpuses. Every tumpus is floral. Tumpuses are yumpuses. Every yumpus is not amenable. Every yumpus is a vumpus. Each vumpus is metallic. Every vumpus is a numpus. Numpuses are temperate. Numpuses are dumpuses. Dumpuses are not small. Dumpuses are rompuses. Polly is a jompus. True or false: Polly is amenable.

Sampled Paths :

P1 (IC=0.875) : Polly is a jompus. Jompuses are tumpuses. Polly is a tumpus. Every tumpus is floral. Tumpuses are yumpuses. Polly is a yumpus. Every yumpus is not amenable. Polly is not amenable. False

P2 (IC=0.656) : Polly is a jompus. Jompuses are impuses. Impuses are amenable. Polly is amenable. True

Ground-truth Path : Polly is a jompus. Jompuses are tumpuses. Polly is a tumpus. Tumpuses are yumpuses. Polly is a yumpus. Every yumpus is not amenable. Polly is not amenable. False

0

10

20

30

Layer

0.0

0.1

0.2

0.3

Attention Weight

Llama-2-7B on PrOntoQA

Context

Query

Rationale

0

10

20

30

Layer

0.0

0.1

0.2

0.3

Llama-2-7B on ProofWriter

0

10

20

30

Layer

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Mistral-7B on PrOntoQA

0

10

20

30

Layer

0.0

0.1

0.2

0.3

0.4

0.5

Mistral-7B on ProofWriter

10

20

30

40

50

Llama-2-7B on PrOntoQA

0

10

20

30

40

Llama-2-7B on ProofWriter

10

20

30

40

Mistral-7B on PrOntoQA

10

20

30

40

50

Number of Top Value Vectors

Mistral-7B on ProofWriter

in the last FFN matrix of the layer. Our analysis focuses on the vectors with the top 0.1% of cosine similarity values, which play a pivotal role in forming final predictions. For more details, please see Appendix B.6.

Results Figure 5 presents our findings: 1) Self-attention layers in the middle layers show a marked focus on the query and reasoning steps, a pattern consistent across both models and datasets; 2) FFN layers in the later layers dominate the final model outputs, as indicated by the fact that value vectors with the highest cosine similarities to the probe vector tend to cluster in the later layers. The layers with strong attention on query and reasoning steps and those where high cosine similarity value vectors cluster do not align, providing insight into the possible mechanism behind the emergence of internal inconsistency. This finding is consistent with Figure 2 and Figure 6, suggesting that the model may make correct predictions in the middle layers but fails to fully utilize them in the later layers. To further understand the specific concepts these high-similarity value vectors represent and verify their importance in determining the final prediction, we project them onto the vocabulary space (nostalgebraist, 2020). Additional analysis are provided in Appendix B.6.3.

5 Related Work

Understanding the Inner Workings of Language Models The rapid progress in LLM development has necessitated simultaneous efforts to interpret the inner workings of advanced models (Bricken et al., 2023; Ferrando et al., 2024). These works aim to provide an internal view of mechanisms to ensure safety and fairness (Burns et al., 2023; Zou et al., 2023; Li et al., 2024) and to further improve model inference (Schuster et al., 2022; Raposo et al., 2024). Some studies also examine CoT reasoning from an internal perspective and propose methods to teach models to perform implicit CoT (Deng et al., 2023). Unlike these approaches, which require additional training for interpretation, our internal consistency measure offers an off-the-shelf solution to calibrate CoT reasoning, providing a practical and efficient tool for enhancing model reliability.

Calibration in Language Models Traditional calibration methods (Platt et al., 1999; Naeini et al., 2015; Guo et al., 2017) train a parameterized classifier on validation data to adjust the final output of a neural network towards expected outcomes. In the context of LLMs, previous works apply trained parameterized models to the logits at the final layer (Zhao et al., 2021; Shen et al., 2024). In comparison, our study focuses on the phenomenon of internal inconsistency in CoT reasoning and demonstrates that internal consistency is a reliable unsupervised calibration measure.

Faithfulness of CoT Reasoning Although the reasoning capabilities of LLMs have been greatly enhanced with techniques like CoT reasoning (Wei et al., 2022; Yao et al., 2022; Xia et al., 2024), previous investigations into the faithfulness of CoT reasoning have shown that the model's generated rationales often are not consistent with their predictions (Lyu et al., 2023; Lanham et al., 2023). These studies primarily focus on the alignment between a model's verbalized reasoning and its outcomes,

without delving into the model's internal reasoning processes. In contrast, our work provides new insights into how unfaithfulness emerges internally during CoT reasoning and proposes solutions to calibrate reasoning with internal consistency.

Closest to our work is that of Halawi et al. (2023), which studies harmful imitation behaviors of LLMs and the underlying internal mechanisms. Unlike their study that focuses on few-shot learning with false demonstrations, we provide new insights on the phenomenon that CoT reasoning leads to internal inconsistency and that we can calibrate reasoning with internal consistency.

6 Conclusions

This work explores the use of internal representations to enhance the reliability of reasoning in LLMs. By examining CoT reasoning, we identify that although generated rationales can improve answer accuracy, they also lead to inconsistencies between middle and final layer representations, potentially affecting reliability. To mitigate this, we introduce internal consistency as a confidence measure, which evaluates the alignment of latent predictions from intermediate layers. Our extensive empirical studies across multiple models and datasets demonstrate that internal consistency is a robust indicator of correct reasoning paths, which we show can be further used for enhancing CoT reasoning by prioritizing paths with high internal consistency. Finally, our analysis on the patterns in attention and feed-forward networks across layers provides insights on why internal inconsistency emerges.

This work has several limitations. First, we restrict our empirical study on decoder-only models. While techniques like decoder lens (Langedijk et al., 2023) is a promising way to extend the concept of internal consistency to encoder-decoder models, we leave it for future work. Second, our analysis focuses on vanilla CoT prompting as the simplest approach for reasoning, whereas many other prompting techniques have been proposed (Gao et al., 2023; Yao et al., 2024). It would be interesting to further investigate internal consistency under these more complex scenarios.

References

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837, 2022.

William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=NjNGlPh8Wh .

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems , 36, 2024.

Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems , 35:30378-30392, 2022.

Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388 , 2023.

Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379 , 2023.

Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702 , 2023.

nostalgebraist. Interpreting GPT: The logit lens. AI Alignment Forum , 2020. URL https://www.alignmentforum.org/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens .

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112 , 2023.

Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Grace: Discriminator-guided chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 15299-15328, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems , 32, 2019.

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680 , 2022.

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=ETKGuby0hcs .

Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer on the inner workings of transformer-based language models. arXiv preprint arXiv:2405.00208 , 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=qFVVBzXxR2V .

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 , 2016.

Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics , 48(1):207-219, 2022.

Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit . " O'Reilly Media, Inc.", 2009.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of language model confidence estimation and calibration. arXiv preprint arXiv:2311.08298 , 2023.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022.

Stanisław Jastrz˛ ebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. arXiv preprint arXiv:1710.04773 , 2017.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning , pages 12697-12706. PMLR, 2021.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019.

Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048 , 2020.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 , 2022.

Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200 , 2024.

Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837 , 2024.

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967 , 2024.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , 2023. https://transformer-circuits.pub/2023/monosemanticfeatures/index.html.

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405 , 2023.

Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems , 35:17456-17472, 2022.

David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258 , 2024.

Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460 , 2023.

John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence , volume 29, 2015.

Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. Thermometer: Towards universal calibration for large language models. arXiv preprint arXiv:2403.08819 , 2024.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022.

Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. Beyond chain-of-thought: A survey of chain-of-x paradigms for llms. arXiv preprint arXiv:2404.15676 , 2024.

Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. arXiv preprint arXiv:2307.09476 , 2023.

Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem Zuidema, and Jaap Jumelet. Decoderlens: Layerwise interpretation of encoder-decoder transformers. arXiv preprint arXiv:2310.03686 , 2023.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning , pages 10764-10799. PMLR, 2023.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems , 36, 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , 2019.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35: 22199-22213, 2022.

Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D. Goodman. Certified deductive reasoning with language models, 2023.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.

A Broader Impacts

This work contributes to the understanding and enhancement of reasoning capabilities in LLMs through the lens of internal consistency. While our findings and methodologies offer significant advancements in the calibration and reliability of LLMs, there are potential negative societal impacts to consider. Enhanced reasoning abilities in LLMs could be misused to generate more convincing disinformation, exacerbating issues related to misinformation and manipulation. To mitigate these risks, it is crucial to implement robust monitoring and access control mechanisms. This includes gated release of models, ensuring that only authorized and ethical usage is permitted. Furthermore, developing and providing defenses against misuse, such as mechanisms for detecting and countering disinformation, is essential. By proactively addressing these potential negative impacts, we aim to responsibly advance the field while safeguarding societal interests.

B Implementation Details

We performed all experiments on a compute node with 8 Nvidia GPU cards and 512 GB of memory. To replicate our results, the total estimated computing time is under 100 GPU hours, even though the entire research project required more computing power than the experiments detailed in the paper.

B.1 Preliminary Analysis

Weconduct linear probing on activations to investigate how information is processed throughout layers and reasoning steps during CoT reasoning. Specifically, for each reasoning step and intermediate layer, we train a logistic regression classifier on the corresponding hidden states of all data instances. We split the dataset randomly by 80%/20% into training and validation subsets. Following Radford et al. (2021), we use the Scikit-learn package (Pedregosa et al., 2011) and determine the L 2 regularization strength λ using a hyperparameter sweep over the range between 10 -6 and 10 6 for logistic regression.

B.2 Main Experiments

Models Our implementation is based on the Huggingface's Transformer library (Wolf et al., 2019). We use meta-llama/Llama-2-7b-hf and meta-llama/Llama-2-13b-hf for the Llama series. For the Mistral series, we evaluate mistralai/Mistral-7B-v0.1 and mistralai/Mixtral-8x7B-v0.1 . The Llama and Mistral series models are released under the custom license and apache-2.0 license, respectively.

Dataset Construction The construction processes for the test datasets are as follows:

BoolQ: We use the original validation set 3 , which has 2033 positive and 1237 negative samples. We balance the classes by randomly sampling 1237 samples from the positive ones, resulting in a subset of 2474 samples. BoolQ is released under the Creative Commons Share-Alike 3.0 license.

CoinFlip: Since the original dataset used in Wei et al. (2022) is not publicly available, we use the reproduced one 4 created by Kojima et al. (2022) which contains 500 instances.

PrOntoQA: We use the official script 5 provided by Saparov and He (2023) to generate 500 instances. Each instance requires 3 hops of reasoning and is based on fictional concept names. PrOntoQA is released under the Apache-2.0 license.

ProofWriter: We use the subset 6 provided by Poesia et al. (2023) and randomly sample 500 instances. ProofWriter is released under the CC BY license.

Prompt Format We use the prompt format {context} \ nQ: {query} \ nA: for all datasets and models. For few-shot prompting, in-context examples are prepended using the following format (taking 2-shot prompting as an example): {icl example 1} \ n \ n{icl example 2} \ n \ n{test example} . For few-shot CoT prompting, we append the rationale after A: for each in-context example. Table 3 provides a list of examples for the considered datasets.

Few-shot Examplars We use the few-shot examples from Wang et al. (2022) for BoolQ and from Wei et al. (2022) for CoinFlip. For ProofWriter and PrOntoQA, we directly utilize the examplars from the original datasets.

B.3 Least-to-Most Prompting

In addition to chain-of-thought prompting, we evaluated our approach using least-to-most prompting Zhou et al. (2022) across four datasets. Due to the absence of existing least-to-most prompting templates for these datasets, we developed a systematic process to generate and validate appropriate prompts.

For each dataset, we selected 4 representative instances as our prompt engineering samples. We utilized GPT-4o to generate initial least-to-most decompositions for these instances, followed by manual inspection and refinement. Table 4 presents examples from each dataset demonstrating our least-to-most decomposition approach.

Our decomposition strategy focuses on breaking down complex reasoning tasks into simpler, sequential steps. This systematic approach helps the model handle complex reasoning tasks by addressing simpler sub-problems sequentially.

Generation In our few-shot CoT experiments, we use Nucleus sampling (Holtzman et al., 2019) with a temperature of 0.7 and a top-p of 0.95 to generate reasoning paths.

B.4 The Issue of Miscalibration in Latent Predictions

Ideally, for a balanced dataset with an equal proportion of True and False instances, we would expect the LLM to assign approximately 50% probability to each class. However, our preliminary analysis shows that latent predictions tend to exhibit a skewed distribution, allocating disproportionately high probabilities to one class. Therefore, we implement specific steps to balance the predictions at each layer before use, following Halawi et al. (2023).

Denote the probability assigned by the LLM to the True class as p ( True ) and the probability assigned to the False class as p ( False ) . We apply the following steps for each layer:

We collect the model's output logits for all instances in the dataset.

We normalize the probabilities in the i -th instance such that ˆ p i ( True ) = p i ( True ) p i ( True )+ p i ( False ) .

We determine the median of the set { ˆ p i } n i =1 to be t .

For any given instance, if the model's output ˆ p i ( True ) ≥ t , we classify it as True; otherwise, we classify it as False.

B.5 Self-attention Layer Analysis

We are interested in the attention value on the context, the query and the generated rationale part. Our calculation can be formally represented as:

where AttnScore( ℓ, P ) denotes the average attention weight across the attention heads on layer ℓ for part P , t ans denotes the index of the answer token, MHSA ( ℓ,h ) ( i, j ) signifies the sum of attention scores from the i -th token to the j -th token within the sequence, as computed by the h -th attention head at the ℓ -th layer.

B.6 FFN Layer Analysis

B.6.1 Value vector

We utilize the concept of the value vector and the weighted value vector of FFN given by Geva et al. (2022). Specifically, each FFN in layer ℓ is made up of two linear transformations with a point-wise activation function in between:

where W ℓ K , W ℓ V ∈ R d m × d are the parameter matrices and σ is the activation function. We define the value vector v ℓ i as the i -th column of W ℓ V . For an input x ℓ , the keys produce a vector of coefficients m ℓ := σ ( W ℓ K x ℓ ) ∈ R d m , then the output of FFN is the weighted average of the value vectors with respect to m ℓ , that is:

B.6.2 Implementation Details

We execute the models on each dataset and extract their final output (True or False) and the last hidden state. Subsequently, we apply logistic regression (without intercept term) in the Scikit-learn package (Pedregosa et al., 2011), and determine the optimal L 2 regularization strength λ using hyperparameter grid search, based on the last hidden state and the final output of the model. The accuracy evaluated by 5-fold cross-evaluation is above 93% consistently (see Table 5 for details). We observe that using ground truth labels as the training target of logistic regression results in similar result with Figure 5 but the training accuracy is poorer as the model may not know the correct answer.

Then we calculate the cosine similarity of the value vectors and the probe vector. As the average cosine similarity across all layers is relatively noisy because many value vectors are not relevant to the task, we plot the count of value vectors of top 0.1% of the cosine similarity values instead.

B.6.3 Semantics Analysis

To uncover the concepts that the value vectors of the top cosine similarity promote, we project them onto the vocabulary space. Specifically, we are interested in the top tokens of the followings:

The probe vector, which we refer to as W probe .

The value vectors with top cosine similarity with the probe vector. We refer the i -th value vector at layer ℓ as v ℓ i ( i and ℓ are indexed from 0 ).

We stack the value vectors of top 100 of cosine similarity to a N × d matrix ( N = 100 ) and apply SVD, obtaining the top singular value vectors. We refer the singular value vector of the largest singular value as U .

Filtering out the tokens that are Unicode escape sequences (e.g. \u2705), the result is depicted in Table 6- 9. Note that some token ids encode the same word.

The result is to verify that the value vectors with the highest cosine similarity to the probe vector indeed encode crucial information for determining the final prediction, so as to verify the emergence of internal inconsistency from the perspective of pattern of attention and FFN layers demonstrated in Section 4.5. This is evidenced by their representation of important keywords such as "true", "false", "verification", and others.

In addition, we also have some interesting discoveries: 1) Some value vectors appear to be of the top cosine similarity with the probe vector in both datasets, indicating their importance and relevance. 2) The value vectors consistently bias towards "True", and the singular value vector of the largest singular value tends to lean towards negative tokens, indicating the necessity of calibration. 3) Some value vectors encode tokens of opposite concepts, which reveals a degree of internal inconsistency within a single value vector.

C Additional Results

In this section, we provide additional results to support our analysis. Figure 6 illustrates the patterns of internal consistency across different layers, as discussed in Section 4.2. This figure shows variable convergence patterns across different models and datasets, with significant increases in consistency observed in the final layers. Figures 7-10 present detailed results for each dataset and model corresponding to the findings in Figure 3.

0

10

20

30

Layer

40

60

80

100

Agreement (%)

Llama-2-7B

BoolQ

CoinFlip

PrOntoQA

ProofWriter

0

10

20

30

40

Layer

40

50

60

70

80

90

100

Llama-2-13B

0

10

20

30

Layer

20

40

60

80

100

Mistral-7B

0

10

20

30

Layer

40

60

80

100

Mixtral-8x7B

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Internal Consistency (%)

Llama-2-7B on BoolQ

Zero-shot

Few-shot

Few-shot

CoT

40

50

60

70

80

90

100

Llama-2-13B on BoolQ

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Mistral-7B on BoolQ

Zero-shot

Few-shot

Few-shot

CoT

40

60

80

100

Mixtral-8x7B on BoolQ

Zero-shot

Few-shot

Few-shot

CoT

40

60

80

100

Internal Consistency (%)

Llama-2-7B on CoinFlip

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Llama-2-13B on CoinFlip

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Mistral-7B on CoinFlip

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Mixtral-8x7B on CoinFlip

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Internal Consistency (%)

Llama-2-7B on PrOntoQA

Zero-shot

Few-shot

Few-shot

CoT

40

60

80

100

Llama-2-13B on PrOntoQA

Zero-shot

Few-shot

Few-shot

CoT

40

60

80

100

Mistral-7B on PrOntoQA

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Mixtral-8x7B on PrOntoQA

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Internal Consistency (%)

Llama-2-7B on ProofWriter

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Llama-2-13B on ProofWriter

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Mistral-7B on ProofWriter

Zero-shot

Few-shot

Few-shot

CoT

20

40

60

80

100

Mixtral-8x7B on ProofWriter

0

25

50

75

100

0.0

0.5

1.0

1.5

2.0

2.5

Density

Llama-2-7B on BoolQ

Correct

Wrong

0

25

50

75

100

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Llama-2-13B on BoolQ

Correct

Wrong

0

25

50

75

100

0.0

0.5

1.0

1.5

2.0

2.5

Mistral-7B on BoolQ

Correct

Wrong

0

25

50

75

100

0.0

0.5

1.0

1.5

2.0

2.5

Mixtral-8x7B on BoolQ

Correct

Wrong

20

40

60

80

100

0

1

2

3

4

Density

Llama-2-7B on CoinFlip

Correct

Wrong

0

25

50

75

100

0

1

2

3

4

Llama-2-13B on CoinFlip

Correct

Wrong

20

40

60

80

100

0

1

2

3

4

5

6

7

Mistral-7B on CoinFlip

Correct

Wrong

40

60

80

0

1

2

3

4

5

6

7

Mixtral-8x7B on CoinFlip

Correct

Wrong

20

40

60

80

100

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Density

Llama-2-7B on PrOntoQA

Correct

Wrong

25

50

75

100

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Llama-2-13B on PrOntoQA

Correct

Wrong

25

50

75

100

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Mistral-7B on PrOntoQA

Correct

Wrong

20

40

60

80

100

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Mixtral-8x7B on PrOntoQA

Correct

Wrong

25

50

75

100

Internal Consistency (%)

0.0

0.5

1.0

1.5

2.0

2.5

Density

Llama-2-7B on ProofWriter

Correct

Wrong

0

25

50

75

100

Internal Consistency (%)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Llama-2-13B on ProofWriter

Correct

Wrong

0

25

50

75

100

Internal Consistency (%)

0.0

0.5

1.0

1.5

2.0

2.5

Mistral-7B on ProofWriter

Correct

Wrong

0

25

50

75

100

Internal Consistency (%)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Mixtral-8x7B on ProofWriter

Correct

Wrong

0

10

20

30

50

60

70

80

90

100

Agreement (%)

Llama-2-7B on BoolQ

Correct

Wrong

0

10

20

30

40

50

60

70

80

90

100

Llama-2-13B on BoolQ

Correct

Wrong

0

10

20

30

50

60

70

80

90

100

Mistral-7B on BoolQ

Correct

Wrong

0

10

20

30

50

60

70

80

90

100

Mixtral-8x7B on BoolQ

Correct

Wrong

0

10

20

30

0

20

40

60

80

100

Agreement (%)

Llama-2-7B on CoinFlip

Correct

Wrong

0

10

20

30

40

20

40

60

80

100

Llama-2-13B on CoinFlip

Correct

Wrong

0

10

20

30

20

40

60

80

100

Mistral-7B on CoinFlip

Correct

Wrong

0

10

20

30

0

20

40

60

80

100

Mixtral-8x7B on CoinFlip

Correct

Wrong

0

10

20

30

40

50

60

70

80

90

100

Agreement (%)

Llama-2-7B on PrOntoQA

Correct

Wrong

0

10

20

30

40

50

60

70

80

90

100

Llama-2-13B on PrOntoQA

Correct

Wrong

0

10

20

30

40

50

60

70

80

90

100

Mistral-7B on PrOntoQA

Correct

Wrong

0

10

20

30

50

60

70

80

90

100

Mixtral-8x7B on PrOntoQA

Correct

Wrong

0

10

20

30

Layer

50

60

70

80

90

100

Agreement (%)

Llama-2-7B on ProofWriter

Correct

Wrong

0

10

20

30

40

Layer

50

60

70

80

90

100

Llama-2-13B on ProofWriter

Correct

Wrong

0

10

20

30

Layer

40

50

60

70

80

90

100

Mistral-7B on ProofWriter

Correct

Wrong

0

10

20

30

Layer

50

60

70

80

90

100

Mixtral-8x7B on ProofWriter

Correct

Wrong

0

20

40

60

80

100

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

p(correct)

Llama-2-7B on BoolQ

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

Llama-2-13B on BoolQ

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

Mistral-7B on BoolQ

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

Mixtral-8x7B on BoolQ

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

p(correct)

Llama-2-7B on CoinFlip

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

1.0

Llama-2-13B on CoinFlip

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

1.0

Mistral-7B on CoinFlip

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

1.0

Mixtral-8x7B on CoinFlip

0

20

40

60

80

100

0.0

0.1

0.2

0.3

0.4

0.5

0.6

p(correct)

Llama-2-7B on PrOntoQA

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

1.0

Llama-2-13B on PrOntoQA

0

20

40

60

80

100

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Mistral-7B on PrOntoQA

0

20

40

60

80

100

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Mixtral-8x7B on PrOntoQA

0

20

40

60

80

100

Internal Consistency (%)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

p(correct)

Llama-2-7B on ProofWriter

0

20

40

60

80

100

Internal Consistency (%)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Llama-2-13B on ProofWriter

0

20

40

60

80

100

Internal Consistency (%)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Mistral-7B on ProofWriter

0

20

40

60

80

100

Internal Consistency (%)

0.0

0.2

0.4

0.6

0.8

Mixtral-8x7B on ProofWriter

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and Section 1 match our experimental results in Section 2.2 and Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results. We focus on empirical findings and analysis about internal consistency in CoT reasoning.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We present implementation details that are sufficient to reproduce the main experimental results in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Due to policy constraints, we are unable to provide the code with our submission. We provide sufficient implementation details for readers to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies the details necessary to understand the results in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include standard deviation in our main results to show the statistical significance of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information on the resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We are guided by the NeurIPS Code of Ethics when conducting this research. For example, we clarify the datasets used, their sources, and their licensing in Appendix B. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impacts of our work in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not introduce new models or data and therefore poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly credits the assets used, mentions their licenses, and respects them. For more details, please refer to Appendix B.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets are well documented in Appendix B.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the correlation between CoT reasoning and internal representations in Transformer models. It shows that there are inconsistencies between model’s internal representations in different layers. Such inconsistencies can be used to distinguish correct and incorrect reasoning paths, and thereby calibrate model predictions. By incorporating internal consistency into self-consistency (SC+IC), the authors show that SC+IC improves over SC under the same number of sampled reasoning paths. Besides, the authors also provided insights into how internal inconsistency is caused by Transformer components.\n\nSTRENGTHS:\n- The correlation between internal representations and CoT reasoning is a novel topic and the conclusion of this paper would be useful for a wide range of audience working on CoT prompting.\n- This paper has a clear logic and the experiment design is solid. The authors first observe inconsistency in internal representations, then propose the internal consistency metric, and show that the metric is correlated with prediction correctness.\n- The authors conduct experiments on 4 different datasets and 2 family of models, showing that SC+IC achieves robust improvement over both SC and SC+$\\Delta$ at almost no additional cost.\n\nWEAKNESSES:\n- The improvement of SC+IC over SC is relatively marginal (+1.2% on avearge). This might be due to the fact that the authors just simply used internal consistency $\\in [0, 1]$ as weights in SC. Maybe the authors can try some other weighting strategies (e.g. exponentiating the internal consistency) and obtain better results.\n- Section 4.4 is not clear to me. Could you explain why attention weights and number of top value vectors reveal the source of internal inconsistency? Could you elaborate what the probe vector is and how you obtained it? It is not very clear to me even after reading the appendix.\n- It is hard to understand some important details (e.g. Figure 6, self-attention layers and FFN layers) without looking into the Appendix. I would recommend the authors to reorganize them in the main paper.\n\nQUESTIONS:\n- Why do we measure internal consistency against the final prediction? How about measuring the consistency between every pair of layers?\n- Can internal consistency be applied to non-binary answers? Will it be still effective in that case?\n- Line 210: By the term “decreases”, do you mean internal consistency is lower for few-shot CoT than few-shot direct prompting?\n- Line 241-242: focusing on the answer tokens → focusing on what the answer tokens attend to.\n- Line 248: second → last FFN matrix in the FNN layer. This would be clearer to the readers.\n- Figure 5 caption: Are attention weights and number of top value vectors supposed to align? Why?\n\nLIMITATIONS:\nYes. The authors has discussed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first reveals that inconsistencies emerge between the internal representations in middle layers and those in final layers, which might undermine the reliability of reasoning processes, and then proposes internal consistency to measure the model’s confidence. Furthermore, the authors propose a method to calibrate CoT reasoning by up-weighting reasoning paths with high internal consistency, which boosts reasoning performance.\n\nSTRENGTHS:\n- This paper identifies the inconsistencies between intermediate and final layer representations in LLM reasoning, and proposes a measure of internal consistency to evaluate the model’s confidence and calibrate CoT reasoning.\n- Extensive results across different models and datasets show that the proposed measure can effectively distinguish between correct and incorrect reasoning processes.\n- The authors propose a new method to calibrate CoT reasoning by up-weighting reasoning paths with high internal consistency, thereby improving reasoning performance.\n\nWEAKNESSES:\n- The motivation for this paper doesn't sound convincing enough (see Questions part).\n- The improvements in Table 1 seem to be limited to some models/datasets. Although the authors performed ten runs with different random seeds, they mentioned that each dataset contains at least 500 samples, which means that the results may vary between 0-10 samples. I suggest the authors report detailed data statistics.\n- This paper includes extensive experiments and results, but lacks analysis and description; Readers need to observe for themselves, such as Figure 3.\n- In Figure 3 (a), the text on the x-axis is overlapping.\n- For Figure 2 (b) and (d), it is unclear how the different internal consistency results (0, 25, 50, 75, 100) were obtained. For example, the authors mentioned that results are averaged over all models and datasets, so (a) shows results corresponding to different prompting methods, and (c) shows results corresponding to different layers. \n- The authors describe Figure 6 in the main text, but the figure is placed in the appendix (of course this may be improved in the final version).\n\nQUESTIONS:\n- Intuitively, as intermediate layers get closer to the final layer, their predictions should increasingly match the final predictions. Why do you think they need to be internally consistent?\n\nLIMITATIONS:\nsee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the use of internal consistency as a measure for calibration, and to improve the reasoning capabilities of LLMs. The internal consistency is a measure of the model’s confidence by examining the agreement of latent predictions decoded from intermediate layers, and this measure does not require additional training. The authors conduct extensive empirical studies across various models and datasets, demonstrating that internal consistency effectively distinguishes between correct and incorrect reasoning paths.\n\nSTRENGTHS:\n1. The proposed measurement is practical, offering a new way to self-evaluate and calibrate reasoning without additional training or human annotations.\n2. The authors conduct extensive empirical studies across different models and datasets. \n3. This paper is also well-structured and clearly presents its methodology, experiments, and findings.\n\nWEAKNESSES:\n1. The proposed measurement is rather straightforward, using an indicator function to measure if latent predictions match the final prediction and summing over all intermediate layers.\n2. The authors claim that the proposed measure is more accurate than logit-based methods and does not require additional training. However, Table 1 does not show a significant performance gap.\n3. The evaluation is conducted on specific datasets involving true-or-false questions, which may not fully represent the diversity of reasoning tasks encountered in practical applications.\n4. The paper dedicates considerable space to explaining the Transformer architecture, which seems unnecessary given the target reader's background. The listed equations for Transformer architecture and internal representations are not utilized in later sections. In contrast, the authors should elaborate more on the decoding process using the proposed consistency score, which is only briefly mentioned.\n\nQUESTIONS:\n1. Will the authors provide more probing results regarding the suitability of lower layers for prediction and the proposed measurement?\n2. Will the authors include results using other approaches for reasoning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the concept of \"internal consistency\" in Large Language Models (LLMs), mainly focusing on chain-of-thought (CoT) reasoning, where models articulate step-by-step rationales. The authors probe internal representations of LLMs to assess the alignment between middle and final layer outputs and propose boosting reasoning paths with higher internal consistency. Through studies across various models and datasets, authors demonstrate that models enhanced with internal consistency metrics reach significant improvement on performances in reasoning tasks.\n\nSTRENGTHS:\n1. Leveraging internal consistency within LLMs to improve reasoning performance is novel and promising. It addresses the issue of LLMs producing reasoning paths that are not always aligned with their final predictions.\n2. The paper provides validations across multiple models and datasets, enhancing the confidence of the results.\n3. By demonstrating how internal consistency can be used to calibrate CoT reasoning, the paper contributes practical techniques for improving LLM reliability, which is crucial for applications requiring precise and trustworthy outputs.\n\nWEAKNESSES:\n1. The techniques described require accessing intermediate layer outputs and could be computationally expensive, limiting the scalability of the proposed methods, especially in resource-constrained environments.\n2. There is a risk that optimizing for internal consistency might lead models to overfit specific reasoning patterns or datasets, potentially reducing the generalizability of the findings.\n\nQUESTIONS:\n1. Can you briefly explain why low LLM layers (<10) can be used to show internal consistency?\n2. With high internal consistency, the results from high LLM layers (>20) may be the same as the last layer. Can we remove the last few layers to reduce the computational complexity of LLMs and increase their reasoning speed? In other words, can we use the method to reduce the size of LLMs?\n3. How does internal consistency interact with other known issues in LLMs, such as hallucination or bias in data?\n\nLIMITATIONS:\nThis paper appropriately describes its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}