On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

## Abstract

Abstract As Large Language Models (LLMs) are increasingly being employed in real-world applications in critical domains such as healthcare, it is important to ensure that the Chain-of-Thought (CoT) reasoning generated by these models faithfully captures their underlying behavior. While LLMs are known to generate CoT reasoning that is appealing to humans, prior studies have shown that these explanations do not accurately reflect the actual behavior of the underlying LLMs. In this work, we explore the promise of three broad approaches commonly employed to steer the behavior of LLMs to enhance the faithfulness of the CoT reasoning generated by LLMs: in-context learning, fine-tuning, and activation editing. Specifically, we introduce novel strategies for in-context learning, fine-tuning, and activation editing aimed at improving the faithfulness of the CoT reasoning. We then carry out extensive empirical analyses with multiple benchmark datasets to explore the promise of these strategies. Our analyses indicate that these strategies offer limited success in improving the faithfulness of the CoT reasoning, with only slight performance enhancements in controlled scenarios. Activation editing demonstrated minimal success, while fine-tuning and in-context learning achieved marginal improvements that failed to generalize across diverse reasoning and truthful question-answering benchmarks. In summary, our work underscores the inherent difficulty in eliciting faithful CoT reasoning from LLMs, suggesting that the current array of approaches may not be sufficient to address this complex challenge.

## 1 Introduction

Large Language Models (LLMs) are increasingly being employed in diverse real-world applications ranging from content generation and education to commerce and healthcare []. One of the primary reasons behind the widespread adoption of these models is their enhanced reasoning capabilities, which enable them to generate responses that appeal to human end users [, ]. Furthermore, these models are also capable of explaining the rationale behind the responses they generate, in a manner that is appealing to humans.
Despite the aforementioned advantages, LLMs also suffer from some critical drawbacks. For instance, while LLMs are adept at producing explanations that cater to human preferences, recent research [, ] demonstrated that the explanations generated by these models – e.g., Chain-of-Thought (CoT) reasoning – do not *faithfully* capture their underlying behavior. The faithfulness of the generated explanations turns out to be an important desideratum in high-stakes applications such as medical diagnostics and legal counseling. Ensuring the faithfulness of LLM-generated CoT reasoning is crucial for decision-makers, such as doctors, who rely on them to determine if, when, and how much to trust the recommendations made by these LLMs.

Despite the criticality of the faithfulness of LLM-generated reasoning, there is very little research on measuring and enhancing this aspect of LLMs. Recently, Lanham et al. [] introduced a slew of metrics for measuring the faithfulness of the CoT reasoning generated by LLMs. For instance, they propose an *early answering* metric, which considers a generated CoT to be faithful if truncating that CoT causes the model to change its final response.
While measuring the faithfulness of an LLM-generated CoT is one critical aspect, another piece of this puzzle is figuring out ways to improve the faithfulness of the CoT reasoning generated by LLMs. While prior works have developed approaches to make CoT more aligned with human understanding or knowledge [], there are no solutions that focus on improving the faithfulness of LLM-generated CoTs in such a way that they accurately capture the behavior of the underlying model (please refer to Appendix for a more detailed discussion on related work).
Furthermore, it remains unclear how difficult it is to improve the faithfulness of LLM-generated CoT reasoning.

Present work. In this work, we address the aforementioned challenges by exploring the promise of three broad approaches—activation editing, fine-tuning, and in-context learning—to enhance the faithfulness of the CoT reasoning generated by LLMs. Activation editing [] involves probing the internal structures of LLMs and strategically updating them to improve certain properties, while fine-tuning focuses on updating model parameters by leveraging curated datasets. In-context learning, on the other hand, involves providing a handful of samples to the model at inference time to tweak its behavior. These three approaches represent different classes of interventions commonly employed in the literature to steer the behavior of LLMs in a desired direction, such as reducing biases and hallucinations. While these approaches have previously been utilized for various tasks [, ], including the reduction of biases and hallucinations, they have not been explored in the context of improving the faithfulness of LLM-generated CoT reasoning.

Here, we introduce novel activation editing, fine-tuning, and in-context learning strategies with the goal of improving the faithfulness of LLM-generated CoT reasoning. Specifically, we introduce an activation editing strategy that involves probing LLMs to first identify a vector/direction that corresponds to faithfulness, and then editing specific attention heads by translating along the identified faithfulness vector. Our fine-tuning and in-context learning strategies involve leveraging the metrics outlined in Lanham et al. [] to identify specific instances and their corresponding faithful CoT reasoning, and providing these as inputs to the LLM during the fine-tuning or in-context learning phases, respectively.

Despite the promise of these techniques, our findings reveal that none of them significantly enhance the faithfulness of the CoT reasoning generated by LLMs. While activation editing approach demonstrates limited success in amplifying faithful behavior of CoT reasoning, the fine-tuning and ICL approaches slightly improved CoT faithfulness in controlled scenarios but did not generalize well across diverse datasets. Our results underscore the inherent difficulty in eliciting faithful reasoning from LLMs, suggesting that the current array of techniques available to us is insufficient for addressing this complex challenge. Our research emphasizes the need for fundamentally new methodologies that can delve into the inner workings of LLMs to enhance the faithfulness of LLM-generated CoT reasoning, ensuring that LLMs are not only generating correct responses but also doing so in a manner that faithfully reflects their internal reasoning processes.

## 2 Preliminaries

Next, we define the notion of faithfulness we use to quantify the reasoning of LLMs and then discuss some notations used to describe different strategies for eliciting faithful reasoning from LLMs.

Figure: Figure 1: Examples for Unfaithful (left) and Faithful (right) explanations generated by state-of-the-art Gpt-4 (left) and Llama-3-8b-Instruct (right) LLMs. The faithfulness score is calculated using the early answering metric proposed in Lanham et al. []. We observe a faithful CoT reasoning gradually improves the prediction probability of the correct answer with an increase in CoT steps.
Refer to caption: x1.png

Chain-of-Thought. CoT reasoning in LLMs provides a structured response where the model explicitly generates the step-by-step thought process leading to its final response. This technique is particularly useful in complex reasoning tasks, such as solving math problems or logical question-answering scenarios, and high-stakes decision-making, where transparency in decision-making is crucial. By eliciting intermediate steps, CoT significantly improves the accuracy of LLMs on reasoning tasks and simultaneously leads to greater user trust and understanding. A relevant stakeholder can now see how the LLM processes the input information and relies on it to generate the final output response. See Fig. 1 for examples of CoT reasoning.
This CoT reasoning makes the LLM’s process more transparent and easier to trust. Further, this also mimics human problem-solving approaches, allowing for easier debugging and refinement of model reasoning. Formally, let $\mathcal{F}:Q\to A$ denote a large language model that maps a sequence of $n$ input tokens $Q=(q_{1},q_{2},\dots,q_{n})$ to sequence of $m$ answer tokens $A=(a_{1},a_{2},\dots,a_{n})$, where $q_{i}$ and $a_{i}$ are text tokens belonging to the model vocabulary $\mathcal{V}$. For CoT reasoning, we append the input tokens $Q$ with a prompt that follows the template: “Read the question, give your answer by analyzing step by step, …”.

Notations. For the activation editing of LLMs, we train different linear classifiers $f:x\to y$, where $x\in\mathbb{R}^{d^{l}_{\text{head}}}$ are the intermediate layer activations of model $\mathcal{F}$ for a given input sequence $X$, $d^{l}_{\text{head}}$ is the dimension of the model activations at layer $l$ and attention head, and $y$ is the respective label associated with the input. We define sampling functions $S(\tau,~{}p,~{}\texttt{mode})$ and $S(\tau,~{}\texttt{nshot},~{}\texttt{mode})$ that we use to sample different fine-tuning and in-context examples in our strategies in Sec. 3, where $\tau$ determines the temperature parameter of the LLM used to control the randomness in the generated answers by using the probability distribution of each generated token, $p$ denotes the percentage of training examples we use in fine-tuning, nshot denotes the number of training examples we use in the ICL prompting, and mode denotes the sampling technique, i.e., whether we want to randomly sample examples from the train split or select the examples with most faithful explanation.

Figure: (a) Example of CoT reasoning
Refer to caption: x2.png

Measuring Faithfulness. While faithfulness is formally defined as how well an explanation accurately reflects the reasoning process of the underlying LLM, operationalizing this definition in the context of LLMs is non-trivial, partly due to the billion parameter scale, black-box nature of LLMs, and partly due to the internal reasoning (typically a combination of multiple complex nonlinear functions) being in a different representation space from textual CoT reasoning []. We utilize faithfulness metrics proposed in Lanham et al. [] that quantify the faithfulness of CoT reasoning from LLMs. Specifically, we employ the Early Answering strategy, which evaluates the faithfulness of a CoT by sequentially adding each CoT step to the question and querying the LLM for its answer, conditioned on the truncated set of CoT steps. If the answer from the LLM converges towards the final answer as it encounters more CoT steps, it indicates that the CoT explanation is guiding the answer and is more likely to be faithful.

To evaluate the faithfulness of a CoT $E$ shown in Fig. 2(a), the early answering strategy involves providing different truncated versions of $E$ and analyzing how the LLM responds to it. For example, if we provide just the first step of $E$, i.e., Prompt = “5! equals what ? 1: 5! = 1$\times$2$\times$3$\times$4$\times$5.” and the LLM does not return 120, but it returns 120 when provided with all the steps in $E$, i.e., Prompt = “5! equals what ? Step 1: 5! = 1$\times$2$\times$3$\times$4$\times$5. Step 2: 1$\times$2$\times$3$\times$4$\times$5 = 120. Step 3: So the final answer is 120”, then we can conclude that $E$ is likely to be faithful. Finally, faithfulness is quantified by the area over the curve (AOC) of explanation fraction vs. the percentage of answers consistent with a full explanation. Note that Lanham et al. [] measures the faithfulness of CoT reasoning at a dataset level. In contrast, we measure faithfulness of each CoT reasoning using probability scores rather than binary correct or incorrect assessments. Following [], faithfulness is quantified by the area over the curve (AOC) of explanation fraction vs. probability of final answer consistent with a full explanation as shown in Fig. 2(b).

## 3 Eliciting Faithful Reasoning from LLMs

Next, we describe three strategies to improve the faithfulness of CoT reasoning generated by LLMs focusing on different aspects (data, weight, activations) of an LLM, i.e., in-context examples (Sec. 3.1), fine-tuning weights (Sec. 3.2), and activation editing (Sec. 3.3).

### 3.1 Faithful Reasoning via In-Context Learning

In contrast to traditional ML approaches that require explicit training or fine-tuning on task-specific data, In-Context Learning (ICL) allows an LLM to generalize and adapt its knowledge by learning patterns from a limited set of demonstrations added within the prompt during inference. ICL is a computationally efficient technique that shows an LLM’s capability to transfer knowledge to novel tasks without additional parameter updates and can be used for both open- and closed-source LLMs.

In order to improve the faithfulness of CoT reasoning using ICL, we include demonstrations of faithful CoT in-context before the question. The intuition is that each ‘faithful’ explanation constitutes a set of logical reasoning blocks expressed in natural language, and steering LLMs towards using these filtered faithful reasoning to construct CoT to arrive at an answer, in turn makes their reasoning more faithful. In particular, we consider $N$ in-context examples, each represented as a triple $(Q_{i},E_{i},A_{i})$ for $1\leq i\leq N$, where $Q_{i}$ and $A_{i}$ represents the question and answer associated with the $i$-th example, while $E_{i}$ denotes a ‘faithful’ CoT reasoning for the question $Q_{i}$ and answer $A_{i}$. Mathematically, we can express the set of $N$ in-context examples as $\{(Q_{1},E_{1},A_{1}),(Q_{2},E_{2},A_{2}),\ldots,(Q_{N},E_{N},A_{N})\}$.

For a given question $Q$, a language model $\mathcal{F}$ and system prompt $S$ to generate CoT reasoning $A_{e}$ along with an answer $A$, the model $\mathcal{F}$ operates as follows, $\mathcal{F}:(Q+S)\rightarrow(A_{e}+A)$, whereas in-context learning involves passing in the examples as:

$$ $\mathcal{F}((Q_{1},A_{1},E_{1})+(Q_{2},A_{2},E_{2}),\ldots,+(Q_{N},A_{N},E_{N} )+Q+S)=A_{e}+A,$ $$

where $N$ demonstrations chosen for ICL impact both the accuracy and faithfulness of answers and CoT reasoning. In order to systematically assess the influence of the specific ICL examples chosen, we propose the following sampling strategies.

- 1)
Deterministic Uniform (DU). Here, we query the LLM deterministically with temperature $\tau=0$ to yield $(Q,E,A)$ triplets over the full training set. We then uniformly sample $N$ demonstrations for ICL. Mathematically, this can be expressed as $S(\tau{=}0,\ \texttt{nshot}{=}N,\ \texttt{mode}{=}\text{`uniform'})$ (see Sec. 2).
- 2)
Deterministic Faithful (DF). As above, except we select the $N$ most faithful CoT reasoning across the $(Q,E,A)$ triplets, expressed as $S(\tau{=}0,\ \texttt{nshot}{=}N,\ \texttt{mode}{=}\text{`faithful'})$.
- 3)
Stochastic Uniform (SU). With this approach, we introduce diversity in eliciting CoT reasoning by sampling at $\tau>0$, generating 10 samples per question and retaining only the most faithful sample. We then uniformly sample $N$ demonstrations for ICL, expressed as $S(\tau>0,\ \texttt{nshot}{=}N,\ \texttt{mode}{=}\text{`uniform'})$.
- 4)
Stochastic Faithful (SF). Here, we combine stochastic sampling with most faithful selection and select the $N$ most faithful demonstrations for ICL, expressed as $S(\tau>0,\ \texttt{nshot}{=}N,\ \texttt{mode}{=}\text{`faithful'})$.

Note that we use these strategies in our empirical analysis and use a superscript ^c notation to indicate that only $(Q,E,A)$ triplets with correct answers are used, e.g., SF^c indicates that we stochastically generate CoT reasoning, and select the $N$ most faithful triplets that yielded correct answers.

### 3.2 Faithful Reasoning via Fine-Tuning

Recent progress in LLMs has led to a paradigm shift from the traditional development of models from scratch to an adoption of shared pre-trained LLMs, e.g., BERT [], GPT [], Llama [], that can readily be fine-tuned for specific downstream applications.
We utilize a combination of recent techniques like Parameter-Efficient Fine-Tuning (PEFT) [] and Low-Rank Adaptation (LoRA) [] that allows efficient fine-tuning LLMs on smaller datasets and reduces the number of trainable parameters by learning low-rank adaptation matrices, making the fine-tuning process more memory and computationally efficient while retaining information that is important for downstream performance.

Our exploration of faithful CoT reasoning via fine-tuning is motivated by Liu et al. [], Ding et al. [] which argues that few-shot PEFT are more effective and cost-efficient as compared to ICL. Hence, we investigate the possible benefits of fine-tuning techniques to elicit more faithful CoT reasoning from LLMs. Our study explores a series of selection strategies aimed at enhancing the faithfulness of CoT reasoning. To this end, we curate a variety of datasets for fine-tuning state-of-the-art LLMs with the goal of fine-tuning LLMs with different question, answer, and CoT reasoning examples and understanding their effects on the faithfulness of CoT reasoning generated by the LLM for test samples during inference. In particular, the strategies we employ for the selection of $(Q,E,A)$ triplets used in finetuning are directly analogous to their ICL counterparts described in Sec. 3.1:

- 1)
Deterministic Uniform (DU). Selecting all examples (instead of $N$ random examples) for the finetuning dataset: $S(\tau{=}0,\ \texttt{p}{=}100\%,\ \texttt{mode}{=}\text{`uniform'})$.
- 2)
Deterministic Faithful (DF). Selecting a percentage of the most faithful examples (instead of the top $N$) for finetuning: $S(\tau{=}0,\ \texttt{p}<100\%,\ \texttt{mode}{=}\text{`faithful'})$.
- 3)
Stochastic Uniform (SU). Selecting all examples (instead of $N$ random examples) for the finetuning dataset: $S(\tau>0,\ \texttt{p}{=}100\%,\ \texttt{mode}{=}\text{`uniform'})$.
- 4)
Stochastic Faithful (SF). Selecting a percentage of the most faithful examples (instead of the top $N$) for finetuning: $S(\tau>0,\ \texttt{p}<100\%,\ \texttt{mode}{=}\text{`faithful'})$.

As in Sec. 3.1, the superscript ^c notation in the empirical analysis indicates that only $(Q,E,A)$ triplets with correct answers were used for fine-tuning.

### 3.3 Faithful Reasoning via Activation Editing

Seminal works in explainable artificial intelligence have shown that probing analysis [] can find vectors in the activation space of deep neural networks that correlate to specific properties learned by the underlying model.
Formally, editing activations to steer a LLM’s behavior involve two key steps - a probing analysis step to identify which components of the model to intervene on, and an editing step which manipulates the activations at run-time. These two steps are detailed below.

Step 1: Probing for Faithfulness. Analyzing a model’s internal structures, such as individual neurons or specific mechanisms like convolution or attention, can offer insights into the inner workings of LLMs []. A standard tool to understand a model’s inner workings is a “probe” []. Probes are linear classifiers trained on a model’s intermediate activations to predict a property like factual correctness, harmful biases, etc. By assessing how well these probes perform, we can infer the extent to which certain types of (mis)information is encoded at different layers or components of the model.

Specifically, we aim to identify attention heads that encode information for faithful reasoning. Using a probing dataset of questions, we collect intermediate activations at all layers and attention heads in a LLM, and create a dataset $\{(x_{i},y_{i})\}_{i=1}^{n}$ for each head $h$ and each layer $l$, where $x_{i}\in\mathbb{R}^{d_{\text{head}}^{l}}$ represents the intermediate activation at a particular layer and attention head of $i^{th}$ question in the probing dataset and $y_{i}$ represents the faithfulness (measured using approaches described in Sec. 2) of reasoning generated for $i^{th}$ question. The probing dataset is split into 4:1 training and validation sets, and the probe is a logistic regression classifier $\sigma({\theta_{h}^{l}}^{T}\boldsymbol{x})$ to predict faithfulness. As faithfulness is a continuous value, we binarize it using median value as threshold. For a model with $L$ layers and $H$ attention heads, we train a total of $L\times H$ linear probes.

Fig. 3 shows the accuracies of linear probes trained on intermediate activations of Llama-3-8b-Instruct on three reasoning and math word problem datasets (discussed at detail in Sec. 4). We observe a significant variance in probing accuracy, suggesting that certain attention heads capture more information about faithful reasoning than others.

Figure: Figure 3: We probe the attention heads across all layers of Llama-3-8b-Instruct to assess their predictive power regarding faithfulness. We show the attention heads in each layer sorted by accuracy, clearly indicating that certain attention heads are more responsible for generating faithful explanations.
Refer to caption: x3.png

Step 2: Activation Editing. Activation editing is a technique to control the post-training behavior of models by using steering intermediate activation vectors, i.e., simple manipulations like translation, scaling, zeroing out, and clamping, on the internal activations of a model at inference time to achieve a desired outcome. By manipulating specific activations associated with certain behaviors, we can alter the LLM’s responses without requiring further training. In our exploration, we apply activation editing to improve the faithfulness of CoT reasoning in LLMs. As shown in 3, we first identified specific attention heads that encode more information about faithful CoT reasoning. We then use this information to steer the LLM in the direction that amplifies faithful reasoning. Following Li et al. [], we translate the activations of a head by a fixed vector during inference.

To avoid causing OoD inputs for subsequent layers by intervening on every head, we do not translate the activations of all attention heads and focus on the top-K heads ranked by the faithfulness metric (Sec. 2), thereby intervening on the LLM’s behavior in a minimally invasive manner. The parameters of the linear probe classifier indicate the direction in which faithful and unfaithful reasoning are maximally separable. Thus, we translate in the direction represented by the linear probe parameters $\theta$.

Figure: Figure 4: Attention mechanism used for intervention on attention heads. $\mathbf{Q^{\prime}}$, $\mathbf{K^{\prime}}$, and $\mathbf{V^{\prime}}$ represent query, key, and value matrices respectively. $\alpha$ denotes the intervention strength, $\theta_{h}^{l}$ represents the learned parameters from linear probe at layer $l$ and attention head $h$. $\sigma_{h}^{l}$ is a scaling factor.

where $\theta_{h}^{l}$ denotes the linear probe classifier trained on the activations on layer $l$ and attention head $h$ and $\alpha$ is a hyper-parameter to control the strength of intervention. The direction vector $\theta_{h}^{l}$ is scaled by $\sigma_{h}^{l}$, representing the standard deviation of projections of activations in the direction of $\theta_{h}^{l}$, ensuring that translation is in the same scale as activations.

## 4 Experiments

We describe the experimental setup used in our analysis before proceeding to discuss the results.

### 4.1 Experimental Setup

Datasets. We conduct experiments using math word problems, commonsense reasoning, and factuality-based benchmark datasets. i) the AQuA [] dataset contains 100,000 algebraic word problems with natural language rationales, where each problem consists of a question – a definition of the problem to solve, options – five possible answer options, where one is correct, rationale – a description of the solution to the problem and correct – a correct option), ii) the LogiQA [] consists of 8,678 question-answer instances, covering multiple types of deductive reasoning, where each question has four possible answer options, and iii) the TruthfulQA [] dataset contains 817 questions in total, spanning 38 categories (e.g., logical falsehoods,
conspiracies, and common points of confusion). Each question comes with an average of $3.2$ truthful
answers, $4.1$ false answers, and a gold standard answer supported by an online source.

Models. We generate and evaluate the faithfulness of reasoning generated by three large language models – Llama-3-8b-Instruct, Gpt-3.5-Turbo, and Gpt-4.

Baselines. We use three baselines to evaluate the effectiveness of the ICL, fine-tuning, and activation editing strategies. 1) Zero-shot (ZS): Here, we assess the accuracy performance of the LLM by just asking the question with invoking CoT reasoning, 2) Zero-shot CoT (ZS-CoT): We invoke the CoT reasoning capability in LLMs by prompting the LLM to think step-by-step (see Fig. 1) before answering the question, and  3) Ground Truth Answers (GTA): We provide a random set of ground truth question and answer pairs during ICL and fine-tuning, and evaluate whether it aids the LLM in generating more faithful CoT reasoning.

### 4.2 Results

Next, we discuss the impact of in-context learning, fine-tuning, and activation editing on the faithfulness of CoT reasoning. Our findings indicate that current techniques do not conclusively improve the faithfulness of CoT reasoning in LLMs.

Figure: Figure 5: Faithfulness vs Accuracy relationship of CoT reasoning generated by Gpt-3.5-Turbo using different baseline (in red) and ICL strategies (in blue). On average, across all three datasets, we find that deterministic faithful (DF) sampling strategy achieve better accuracy-faithful trade-off.
Refer to caption: x6.png

#### 4.2.1 In-context Learning Analysis

Using ICL, we aim to address the question: Can an LLM learn to elicit faithful CoT reasoning by simply looking at some faithful CoT examples during inference? We investigate this question using the sampling strategies detailed in Sec. 3.1, and different datasets and LLMs described in Sec. 4.1.

More accurate LLMs are less faithful. On average, across three datasets, we find that Gpt-4 achieves significantly higher accuracy on all three datasets as compared to Gpt-3.5-Turbo and Llama-3-8b-Instruct (see Figs. 10,5,6), but it exhibits poor faithfulness performance. For instance, in TruthfulQA, we find that Gpt-4 provides correct answers to questions without using CoT reasoning (i.e., accuracy difference between non-CoT and CoT prompting is zero), resulting in low faithfulness by definition. Also, larger LLMs like Gpt-4 are increasingly optimized for dialogue and generating conversational responses where RLHF rewards coherence to a human evaluator, which may conflict with generating faithful CoT reasoning.

In-context learning (ICL) improves faithfulness, albeit with a trade-off in accuracy. On all datasets and models, we observe that in-context learning improves faithfulness compared to zero shot baseline for almost all sampling strategies as shown in Figs. 10,5,6. Using faithful samples in-context particularly enhances faithfulness, as evidenced by a rise in faithfulness compared to the uniform counterpart, i.e., faithfulness of DF > DU and SF > SU. One exception is Llama-3-8b-Instruct on TruthfulQA dataset. We suspect this is due to TruthfulQA being a dataset of human falsehoods relies less on reasoning to arrive at an answer. While ICL improves faithfulness, this often comes with a drop in accuracy as shown in Figs. 10,5,6.

Certain sampling strategies provide better trade-offs. Using top-K faithful samples (DF), on average, improves the faithfulness of the CoT reasoning but takes a hit on the accuracy, whereas the stochastic uniform sampling (SU) obtains better accuracy without improving faithfulness. Stochastic faithful sampling (SF) provides a middle ground. Moreover, we find better accuracy-faithfulness trade-offs when we perform ICL prompting using only the CoT reasoning from correctly predicted question-answer pairs by the LLM.

In summary, our results show that we cannot elicit faithful CoT reasoning from LLMs by simply using examples from different ICL strategies during inference without sacrificing accuracy.

Figure: Figure 6: Faithfulness vs Accuracy relationship of CoT reasoning generated by Llama-3-8b-Instruct using different baseline (in red) and ICL strategies (in blue). Results show that none of the baseline or sampling strategy consistently achieve high accuracy and faithfulness.
Refer to caption: x9.png

#### 4.2.2 Fine-tuning Analysis

Here, we aim to investigate the possible benefits of fine-tuning techniques to elicit more faithful CoT reasoning from LLMs. We fine-tune Llama-3-8b-Instruct and Gpt-3.5-Turbo models(^1^11Due to OpenAI API errors at the time of experimentation [], we were unable to access or evaluate fine-tuned versions of GPT-4.) using different baselines (Sec. 4.1) and sampling techniques (Sec. 3.2).

Fine-tuned LLMs show contrasting faithfulness performance. Our results in Figs. 7 and 8 for AQuA and LogiQA show that while some sampling strategies lead to improvement in faithfulness of CoT reasoning for fine-tuned Gpt-3.5-Turbo, they obtain lower faithfulness than GTA baseline for fine-tuned Llama-3-8b-Instruct. In addition, we observe that the baseline GTA achieves a good accuracy-faithfulness trade-off for the LogiQA dataset (Fig. 8), it does not follow the same trend for fine-tuned Gpt-3.5-Turbo (Fig. 7). Further, our fine-tuning results on TruthfulQA show that while we can force an LLM to generate faithful CoT reasoning via fine-tuning (verified by an increase in their faithfulness performance), it significantly impacts the accuracy of the model ($\sim$20% drop in accuracy) (see TruthfulQA; Fig. 8).

Fine-tuning using most faithful explanations achieve better accuracy-faithfulness trade-offs. For the fine-tuned Gpt-3.5-Turbo on LogiQA dataset, we find that sampling strategies like DF and SF achieve higher faithfulness as compared to the baselines (in red), highlighting that selecting examples with faithful explanations for fine-tuning can help in generating faithful CoT reasoning from the fine-tuned LLMs. Notably, we observe a better accuracy-faithfulness trade-offs when fine-tuning using only the correctly predicted question-answer pairs with CoT reasoning (see Fig. 7; $\text{DF}^{\text{c}}$ in AQuA and $\text{SF}^{\text{c}}$ in LogiQA).

Figure: Figure 7: Faithfulness vs Accuracy relationship of CoT reasoning generated by fine-tuned Gpt-3.5-Turbo using different baselines (in red) and sampling strategies (in blue). Results show that while the baseline GTA achieves good accuracy-faithfulness trade-off (top-right corner) for AQuA and TruthfulQA dataset, it achieves the worst trade-off (bottom-left corner) for LogiQA dataset.
Refer to caption: x12.png

Figure: Figure 8: Faithfulness vs Accuracy relationship of CoT reasoning generated by fine-tuned Llama-3-8b-Instruct using different baselines (in red) and sampling strategies (in blue). On average, across all datasets, we find that none of the baseline or sampling strategies achieve high faithfulness.
Refer to caption: x15.png

#### 4.2.3 Activation Editing Analysis

Through activation editing, we aim to understand the effect of intervening on a model to amplify faithful behavior. The intervention equation described in 4 has a hyperparameter $\alpha$ indicating the intervention strength $\alpha$. Furthermore, we intervene only on the top-$K$ faithful heads (as identified in Fig. 3) in order to be minimally invasive. The results in Fig. 9 show faithfulness and accuracy on TruthfulQA and AQuA upon intervening on different number of attention heads of Llama-3-8b-Instruct, i.e., $K=\{2,4,8\}$, and intervention strengths, i.e., $\alpha=\{0.25,0.50,1.0\}$.

Intervening on attention heads leads to a drop in accuracy with a marginal gain in faithfulness. The results in Fig. 9 show that intervening on the most faithful attention heads of Llama-3-8b-Instruct doesn’t yield a significant boost in the faithfulness of its CoT reasoning. Interestingly, as compared to the ZS-CoT performance of Llama-3-8b-Instruct (AQuA: {Accuracy: 0.49; Faithfulness: 0.627} and TruthfulQA: {Accuracy: 0.57; Faithfulness: 0.232}), we find no significant improvement in both accuracy (Fig. 9; columns (a),(c)) and faithfulness (Fig. 9; columns (b),(d)). Moreover, the identified faithful attention heads, optimal value of intervention strength ($\alpha$), and optimal number of intervened heads ($K$) are not consistent across different datasets, highlighting the lack of generalization of activation editing strategies to various datasets. In addition, our analysis demonstrates contrasting behaviors in Llama-3-8b-Instruct, where activation editing works for improving truthfulness but shows mixed results for faithfulness, underscoring the challenge of eliciting faithful CoT reasoning from LLMs. Finally, our results also highlight the dichotomy between accuracy and faithfulness, where the values of $\{\alpha,K\}$ for the most faithful attention head (dark green in Fig. 9) are not always equivalent to the most accurate one (dark blue in Fig. 9).

Figure: Figure 9: Accuracy and Faithfulness of LLM reasoning for different intervention configurations ($\alpha,K$). The difference between the accuracy and faithfulness performance of Llama-3-8b-Instruct and highlights that none of the intervention configuration leads to improvement of both accuracy and faithfulness across both datasets compared to ZS-CoT performance ( and markers). Refer to Appendix Fig. 11 for LogiQA dataset.
Refer to caption: x18.png

## 5 Conclusion

In this study, we investigated the challenge of eliciting faithfulness chain-of-thought reasoning in Large Language Models (LLMs). We explored three widely used techniques: activation editing, fine-tuning, and in-context learning (ICL) in our empirical analysis. Our results indicate that while these methods provided marginal improvements, none were sufficient to consistently enhance the CoT faithfulness across diverse datasets and LLMs. Our findings highlight the critical need for novel methodologies and a deeper understanding of LLMs’ internal reasoning processes to generate more faithful CoT explanations.

## Appendix

## Appendix A Broader Impact

Our work focuses on exploring whether we can improve the faithfulness of the CoT reasoning generated by state-of-the-art LLMs. This has significant positive implications for societal benefit. For instance, if CoT reasoning output by LLMs faithfully captures the underlying model behavior, decision-makers and relevant stakeholders can leverage this to determine if, when, and how much to rely on the recommendations provided by LLMs. Therefore, our exploration itself is very valuable and has a substantial positive societal impact. Our analyses and findings indicate that existing techniques commonly used to steer behavior in LLMs are not effective in enhancing the faithfulness of LLM-generated CoT reasoning. While this finding is not particularly positive, we believe it is a step in the right direction, informing us of the complexity of the problem and underscoring the need for fundamentally different frameworks to address it. As far as we understand, our work does not have any potential negative societal impacts, as it is mainly an exploration to improve the faithfulness of LLM-generated CoT reasoning.

## Appendix B Related Work

##### Chain-of-Thought Reasoning

Large Language Models (LLMs) produce Chain-of-Thought (CoT) reasoning [, ] to help provide end users with a peak into the reasoning process leading up to their response. While the CoT reasoning generated by these models is often appealing to human end users [, ], prior research has argued that LLM-generated CoT reasoning does not *faithfully* capture the underlying behavior of these models and that this is a critical challenge particularly in applications involving high-stakes decision making [].
For instance, as discussed in Agarwal et al. [], a doctor would benefit from seeing an explanation that faithfully captures why an LLM is recommending a particular diagnosis for a patient, as opposed to seeing some plausible explanation that could lead to the diagnosis at hand. In the former case, the doctor can actually use this faithful explanation to determine if and how much to rely on the model’s recommendation.

##### Evaluating the Faithfulness of CoT Reasoning

Despite the criticality of the faithfulness of LLM-generated CoT reasoning, there is very little work on analyzing and measuring this aspect of LLMs. Turpin et al. [] were the first to demonstrate that CoT explanations may not faithfully capture the behavior of the underlying models. They showed that these explanations can be heavily influenced by biasing model inputs e.g., by reordering multiple-choice options in a few-shot prompt to always make the answer “(A)”—which these models systematically fail to mention in their explanations. Lanham et al. [] extended the above work and proposed novel metrics to measure the faithfulness of an LLM-generated CoT explanation. For instance, they propose an *early answering* metric, which considers a generated CoT to be faithful if truncating that CoT causes the model to change its final response. Similarly, if *adding mistakes* in a generated CoT causes the model to change its final response, then the original CoT can be considered faithful. Analogously, they proposed other metrics to measure faithfulness based on *paraphrasing* the beginning portions of the original CoT as well as replacing the CoT with *filler* tokens (e.g., ellipses). Using these metrics, they demonstrated that the CoT reasoning produced by state-of-the-art LLMs does not faithfully capture the behavior of the underlying models.

##### Enhancing the Quality of CoT Reasoning

While there are some prior works that tackled the problem of improving the quality of CoT reasoning [], their focus was on improving its quality vis-a-vis human knowledge or understanding. For example, Lyu et al. [] focused on generating a reasoning chain that could then be put through a deterministic math solver, and the resulting answer from this solver was compared to the answer produced by the LLM. The reasoning chain was considered to be faithful if the answers of the solver and the LLM matched. Note that this approach does not account for ensuring that the internal computations or the underlying behavior of the LLM was captured in the reasoning chain, which is the focus of our work.

In summary, our work makes one of the initial attempts at exploring the promise of various popular paradigms, namely, activation editing, fine-tuning, and in-context learning, to improve the faithfulness of the CoT reasoning generated by LLMs.

## Appendix C Experiments

Here, we provide additional results of our experiments in tabular format and perform significance testing of all our empirical analysis.

Figure: Figure 10: Faithfulness vs Accuracy relationship of CoT reasoning generated by Gpt-4 using different baseline (in red) and ICL strategies (in blue). Results show that stochastic faithful sampling strategies, on average across three datasets, achieves higher faithfulness in CoT reasoning.
Refer to caption: x19.png

Figure: Figure 11: Accuracy and faithfulness of LLM reasoning for different intervention configurations ($\alpha,K$) for LogiQA dataset. Activation editing shows different the trade-off between the accuracy and faithfulness performance of Llama-3-8b-Instruct and some configuration leads to an increase in accuracy as compared to the zero-shot CoT performance ( and markers) but doesn’t improve faithfulness significantly.
Refer to caption: x22.png

**Table 1: GPT-3.5-Turbo Faithfulness for Different Fine-tuning Approaches**
| Approach | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Accuracy | Faithfulness | Accuracy | Faithfulness | Accuracy | Faithfulness |
| ZS-CoT | 0.60 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.47 $\pm$ 0.05 | 0.40 $\pm$ 0.03 | 0.64 $\pm$ 0.05 | 0.30 $\pm$ 0.02 |
| GTA | 0.60 $\pm$ 0.05 | 0.65 $\pm$ 0.02 | 0.38 $\pm$ 0.05 | 0.38 $\pm$ 0.03 | 0.83 $\pm$ 0.04 | 0.35 $\pm$ 0.03 |
| DU | 0.57 $\pm$ 0.05 | 0.62 $\pm$ 0.02 | 0.46 $\pm$ 0.05 | 0.49 $\pm$ 0.03 | 0.65 $\pm$ 0.05 | 0.29 $\pm$ 0.03 |
| DU^c | 0.61 $\pm$ 0.05 | 0.60 $\pm$ 0.03 | 0.46 $\pm$ 0.05 | 0.43 $\pm$ 0.03 | 0.65 $\pm$ 0.05 | 0.32 $\pm$ 0.03 |
| DF | 0.58 $\pm$ 0.05 | 0.62 $\pm$ 0.02 | 0.43 $\pm$ 0.05 | 0.51 $\pm$ 0.03 | 0.63 $\pm$ 0.05 | 0.32 $\pm$ 0.03 |
| DF^c | 0.63 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.43 $\pm$ 0.05 | 0.44 $\pm$ 0.03 | 0.64 $\pm$ 0.05 | 0.33 $\pm$ 0.03 |
| SU | 0.61 $\pm$ 0.05 | 0.66 $\pm$ 0.02 | 0.45 $\pm$ 0.05 | 0.48 $\pm$ 0.03 | 0.61 $\pm$ 0.05 | 0.36 $\pm$ 0.03 |
| SU^c | 0.57 $\pm$ 0.05 | 0.65 $\pm$ 0.02 | 0.44 $\pm$ 0.05 | 0.47 $\pm$ 0.03 | 0.67 $\pm$ 0.05 | 0.33 $\pm$ 0.02 |
| SF | 0.58 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.45 $\pm$ 0.05 | 0.51 $\pm$ 0.02 | 0.62 $\pm$ 0.05 | 0.35 $\pm$ 0.03 |
| SF^c | 0.59 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.49 $\pm$ 0.05 | 0.48 $\pm$ 0.03 | 0.63 $\pm$ 0.05 | 0.34 $\pm$ 0.02 |

**Table 2: Llama-3-8B-Instruct Faithfulness for Different Fine-tuning Approaches**
| Approach | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Accuracy | Faithfulness | Accuracy | Faithfulness | Accuracy | Faithfulness |
| ZS-CoT | 0.43 $\pm$ 0.05 | 0.61 $\pm$ 0.02 | 0.38 $\pm$ 0.05 | 0.41 $\pm$ 0.03 | 0.56 $\pm$ 0.05 | 0.23 $\pm$ 0.03 |
| GTA | 0.43 $\pm$ 0.05 | 0.68 $\pm$ 0.01 | 0.44 $\pm$ 0.05 | 0.65 $\pm$ 0.01 | 0.75 $\pm$ 0.04 | 0.23 $\pm$ 0.03 |
| DU | 0.45 $\pm$ 0.05 | 0.62 $\pm$ 0.02 | 0.42 $\pm$ 0.05 | 0.48 $\pm$ 0.02 | 0.51 $\pm$ 0.05 | 0.27 $\pm$ 0.03 |
| DU^c | 0.52 $\pm$ 0.05 | 0.57 $\pm$ 0.02 | 0.41 $\pm$ 0.05 | 0.43 $\pm$ 0.03 | 0.52 $\pm$ 0.05 | 0.24 $\pm$ 0.03 |
| DF | 0.43 $\pm$ 0.05 | 0.63 $\pm$ 0.02 | 0.45 $\pm$ 0.05 | 0.44 $\pm$ 0.02 | 0.55 $\pm$ 0.05 | 0.31 $\pm$ 0.03 |
| DF^c | 0.54 $\pm$ 0.05 | 0.62 $\pm$ 0.02 | 0.40 $\pm$ 0.05 | 0.43 $\pm$ 0.03 | 0.54 $\pm$ 0.05 | 0.29 $\pm$ 0.03 |
| SU | 0.50 $\pm$ 0.05 | 0.67 $\pm$ 0.01 | 0.43 $\pm$ 0.05 | 0.45 $\pm$ 0.03 | 0.53 $\pm$ 0.05 | 0.28 $\pm$ 0.03 |
| SU^c | 0.47 $\pm$ 0.05 | 0.61 $\pm$ 0.02 | 0.42 $\pm$ 0.05 | 0.45 $\pm$ 0.03 | 0.54 $\pm$ 0.05 | 0.24 $\pm$ 0.03 |
| SF | 0.39 $\pm$ 0.05 | 0.61 $\pm$ 0.02 | 0.40 $\pm$ 0.05 | 0.44 $\pm$ 0.03 | 0.50 $\pm$ 0.05 | 0.27 $\pm$ 0.03 |
| SF^c | 0.50 $\pm$ 0.05 | 0.62 $\pm$ 0.02 | 0.39 $\pm$ 0.05 | 0.44 $\pm$ 0.03 | 0.61 $\pm$ 0.05 | 0.27 $\pm$ 0.03 |

**Table 3: GPT-4 Faithfulness for Different In-Context Learning Approaches**
| Approach | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Accuracy | Faithfulness | Accuracy | Faithfulness | Accuracy | Faithfulness |
| ZS-CoT | 0.64 $\pm$ 0.05 | 0.49 $\pm$ 0.03 | 0.56 $\pm$ 0.05 | 0.21 $\pm$ 0.03 | 0.90 $\pm$ 0.03 | 0.04 $\pm$ 0.01 |
| GTA | 0.68 $\pm$ 0.05 | 0.50 $\pm$ 0.03 | 0.67 $\pm$ 0.05 | 0.25 $\pm$ 0.03 | 0.92 $\pm$ 0.03 | 0.06 $\pm$ 0.02 |
| DU | 0.63 $\pm$ 0.05 | 0.51 $\pm$ 0.03 | 0.65 $\pm$ 0.05 | 0.24 $\pm$ 0.03 | 0.93 $\pm$ 0.03 | 0.04 $\pm$ 0.01 |
| DU^c | 0.65 $\pm$ 0.05 | 0.48 $\pm$ 0.03 | 0.66 $\pm$ 0.05 | 0.22 $\pm$ 0.02 | 0.90 $\pm$ 0.03 | 0.05 $\pm$ 0.01 |
| DF | 0.66 $\pm$ 0.05 | 0.50 $\pm$ 0.03 | 0.61 $\pm$ 0.05 | 0.21 $\pm$ 0.02 | 0.81 $\pm$ 0.04 | 0.09 $\pm$ 0.02 |
| DF^c | 0.68 $\pm$ 0.05 | 0.52 $\pm$ 0.03 | 0.66 $\pm$ 0.05 | 0.23 $\pm$ 0.03 | 0.84 $\pm$ 0.04 | 0.08 $\pm$ 0.02 |
| SU | 0.58 $\pm$ 0.05 | 0.50 $\pm$ 0.03 | 0.64 $\pm$ 0.05 | 0.26 $\pm$ 0.03 | 0.89 $\pm$ 0.03 | 0.09 $\pm$ 0.02 |
| SU^c | 0.66 $\pm$ 0.05 | 0.51 $\pm$ 0.03 | 0.66 $\pm$ 0.05 | 0.23 $\pm$ 0.03 | 0.84 $\pm$ 0.04 | 0.08 $\pm$ 0.02 |
| SF | 0.63 $\pm$ 0.05 | 0.51 $\pm$ 0.03 | 0.55 $\pm$ 0.05 | 0.21 $\pm$ 0.02 | 0.82 $\pm$ 0.04 | 0.07 $\pm$ 0.02 |
| SF^c | 0.65 $\pm$ 0.05 | 0.51 $\pm$ 0.03 | 0.60 $\pm$ 0.05 | 0.28 $\pm$ 0.03 | 0.82 $\pm$ 0.04 | 0.11 $\pm$ 0.02 |

**Table 4: GPT-3.5-Turbo Faithfulness for Different In-Context Learning Approaches**
| Approach | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Accuracy | Faithfulness | Accuracy | Faithfulness | Accuracy | Faithfulness |
| ZS-CoT | 0.60 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.47 $\pm$ 0.05 | 0.40 $\pm$ 0.03 | 0.64 $\pm$ 0.05 | 0.30 $\pm$ 0.02 |
| GTA | 0.56 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.48 $\pm$ 0.05 | 0.44 $\pm$ 0.03 | 0.69 $\pm$ 0.05 | 0.33 $\pm$ 0.02 |
| DU | 0.55 $\pm$ 0.05 | 0.67 $\pm$ 0.02 | 0.48 $\pm$ 0.05 | 0.40 $\pm$ 0.03 | 0.65 $\pm$ 0.05 | 0.32 $\pm$ 0.02 |
| DU^c | 0.64 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.40 $\pm$ 0.05 | 0.44 $\pm$ 0.03 | 0.75 $\pm$ 0.04 | 0.30 $\pm$ 0.03 |
| DF | 0.57 $\pm$ 0.05 | 0.67 $\pm$ 0.02 | 0.54 $\pm$ 0.05 | 0.47 $\pm$ 0.03 | 0.74 $\pm$ 0.04 | 0.30 $\pm$ 0.02 |
| DF^c | 0.62 $\pm$ 0.05 | 0.65 $\pm$ 0.02 | 0.44 $\pm$ 0.05 | 0.45 $\pm$ 0.03 | 0.73 $\pm$ 0.04 | 0.34 $\pm$ 0.03 |
| SU | 0.59 $\pm$ 0.05 | 0.66 $\pm$ 0.02 | 0.43 $\pm$ 0.05 | 0.46 $\pm$ 0.03 | 0.78 $\pm$ 0.04 | 0.30 $\pm$ 0.03 |
| SU^c | 0.61 $\pm$ 0.05 | 0.65 $\pm$ 0.02 | 0.44 $\pm$ 0.05 | 0.45 $\pm$ 0.03 | 0.73 $\pm$ 0.04 | 0.32 $\pm$ 0.02 |
| SF | 0.59 $\pm$ 0.05 | 0.67 $\pm$ 0.02 | 0.48 $\pm$ 0.05 | 0.46 $\pm$ 0.03 | 0.70 $\pm$ 0.05 | 0.30 $\pm$ 0.02 |
| SF^c | 0.62 $\pm$ 0.05 | 0.66 $\pm$ 0.02 | 0.39 $\pm$ 0.05 | 0.44 $\pm$ 0.03 | 0.70 $\pm$ 0.05 | 0.31 $\pm$ 0.03 |

**Table 5: Llama-3-8B-Instruct Faithfulness for Different In-Context Learning Approaches**
| Approach | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Accuracy | Faithfulness | Accuracy | Faithfulness | Accuracy | Faithfulness |
| ZS-CoT | 0.43 $\pm$ 0.05 | 0.61 $\pm$ 0.02 | 0.38 $\pm$ 0.05 | 0.41 $\pm$ 0.03 | 0.56 $\pm$ 0.05 | 0.23 $\pm$ 0.03 |
| GTA | 0.48 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.47 $\pm$ 0.05 | 0.40 $\pm$ 0.03 | 0.57 $\pm$ 0.05 | 0.26 $\pm$ 0.03 |
| DU | 0.44 $\pm$ 0.05 | 0.65 $\pm$ 0.02 | 0.43 $\pm$ 0.05 | 0.38 $\pm$ 0.03 | 0.59 $\pm$ 0.05 | 0.24 $\pm$ 0.03 |
| DU^c | 0.55 $\pm$ 0.05 | 0.63 $\pm$ 0.02 | 0.47 $\pm$ 0.05 | 0.43 $\pm$ 0.03 | 0.65 $\pm$ 0.05 | 0.23 $\pm$ 0.03 |
| DF | 0.43 $\pm$ 0.05 | 0.63 $\pm$ 0.02 | 0.35 $\pm$ 0.05 | 0.43 $\pm$ 0.03 | 0.59 $\pm$ 0.05 | 0.24 $\pm$ 0.03 |
| DF^c | 0.37 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.42 $\pm$ 0.05 | 0.39 $\pm$ 0.03 | 0.64 $\pm$ 0.05 | 0.21 $\pm$ 0.03 |
| SU | 0.52 $\pm$ 0.05 | 0.62 $\pm$ 0.02 | 0.45 $\pm$ 0.05 | 0.42 $\pm$ 0.03 | 0.60 $\pm$ 0.05 | 0.25 $\pm$ 0.03 |
| SU^c | 0.45 $\pm$ 0.05 | 0.63 $\pm$ 0.02 | 0.44 $\pm$ 0.05 | 0.42 $\pm$ 0.03 | 0.57 $\pm$ 0.05 | 0.23 $\pm$ 0.03 |
| SF | 0.43 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.41 $\pm$ 0.05 | 0.39 $\pm$ 0.03 | 0.58 $\pm$ 0.05 | 0.23 $\pm$ 0.03 |
| SF^c | 0.42 $\pm$ 0.05 | 0.65 $\pm$ 0.02 | 0.47 $\pm$ 0.05 | 0.41 $\pm$ 0.03 | 0.58 $\pm$ 0.05 | 0.23 $\pm$ 0.03 |

**Table 6: GPT-3.5-Turbo p-values of Faithfulness for Different Fine-tuning Approaches**
| Comparing | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | ZS-CoT | GTA | ZS-CoT | GTA | ZS-CoT | GTA |
| DU | 0.1946 | 0.2247 | 0.0000 | 0.0005 | 0.6353 | 0.1101 |
| DU^c | 0.0718 | 0.0974 | 0.2141 | 0.0597 | 0.3573 | 0.4600 |
| DF | 0.3640 | 0.2610 | 0.0000 | 0.0000 | 0.3607 | 0.4292 |
| DF^c | 0.9523 | 0.7473 | 0.0917 | 0.0201 | 0.2364 | 0.6090 |
| SU | 0.4740 | 0.7740 | 0.0014 | 0.0010 | 0.0473 | 0.9173 |
| SU^c | 0.8063 | 0.9671 | 0.0088 | 0.0028 | 0.2353 | 0.5102 |
| SF | 0.8789 | 0.6707 | 0.0000 | 0.0000 | 0.0579 | 0.9934 |
| SF^c | 0.8324 | 0.6255 | 0.0006 | 0.0001 | 0.1071 | 0.7794 |

**Table 7: Llama-3-8B-Instruct p-values for Different Fine-tuning Approaches**
| Comparing | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | ZS-CoT | GTA | ZS-CoT | GTA | ZS-CoT | GTA |
| DU | 0.4325 | 0.0062 | 0.0027 | 0.0000 | 0.1835 | 0.1687 |
| DU^c | 0.0845 | 0.0000 | 0.5589 | 0.0000 | 0.7541 | 0.7380 |
| DF | 0.3175 | 0.0103 | 0.1958 | 0.0000 | 0.0130 | 0.0194 |
| DF^c | 0.6068 | 0.0011 | 0.3946 | 0.0000 | 0.0580 | 0.0639 |
| SU | 0.0020 | 0.4670 | 0.1636 | 0.0000 | 0.1311 | 0.1476 |
| SU^c | 0.9323 | 0.0020 | 0.1537 | 0.0000 | 0.7327 | 0.6844 |
| SF | 0.9893 | 0.0003 | 0.2321 | 0.0000 | 0.2940 | 0.2954 |
| SF^c | 0.6049 | 0.0012 | 0.3319 | 0.0000 | 0.1527 | 0.2178 |

**Table 8: GPT-4 p-values for Different In-Context Learning Approaches**
| Comparing | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | ZS-CoT | GTA | ZS-CoT | GTA | ZS-CoT | GTA |
| DU | 0.3089 | 0.8058 | 0.1395 | 0.7462 | 0.6632 | 0.2648 |
| DU^c | 0.6307 | 0.1890 | 0.4525 | 0.3024 | 0.2392 | 0.5765 |
| DF | 0.5638 | 0.7929 | 0.9936 | 0.1062 | 0.0048 | 0.0489 |
| DF^c | 0.1322 | 0.4820 | 0.3382 | 0.4337 | 0.0250 | 0.2369 |
| SU | 0.6778 | 0.7624 | 0.0509 | 0.8104 | 0.0063 | 0.0572 |
| SU^c | 0.3145 | 0.7599 | 0.3932 | 0.3125 | 0.0297 | 0.2525 |
| SF | 0.2818 | 0.6367 | 0.9038 | 0.0491 | 0.0478 | 0.5067 |
| SF^c | 0.2677 | 0.5417 | 0.0037 | 0.2679 | 0.0011 | 0.0111 |

**Table 9: GPT-3.5-Turbo p-values for Different In-Context Learning Approaches**
| Comparing | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | ZS-CoT | GTA | ZS-CoT | GTA | ZS-CoT | GTA |
| DU | 0.2748 | 0.1188 | 0.8037 | 0.1245 | 0.4544 | 0.7770 |
| DU^c | 0.8994 | 0.8539 | 0.1285 | 0.8728 | 0.9518 | 0.3670 |
| DF | 0.1845 | 0.0451 | 0.0144 | 0.3093 | 0.9840 | 0.3429 |
| DF^c | 0.8065 | 0.5908 | 0.0505 | 0.7696 | 0.2248 | 0.7428 |
| SU | 0.4238 | 0.2524 | 0.0186 | 0.5463 | 0.9364 | 0.3364 |
| SU^c | 0.8541 | 0.5486 | 0.0323 | 0.6991 | 0.5434 | 0.6946 |
| SF | 0.0992 | 0.0558 | 0.0093 | 0.3931 | 0.8899 | 0.4127 |
| SF^c | 0.2790 | 0.1526 | 0.1431 | 0.8505 | 0.6492 | 0.6452 |

**Table 10: Llama-3-8B-Instruct p-values for Different In-Context Learning Approaches**
| Comparing | AQuA | LogiQA | TruthfulQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | ZS-CoT | GTA | ZS-CoT | GTA | ZS-CoT | GTA |
| DU | 0.0488 | 0.5230 | 0.3320 | 0.4825 | 0.8569 | 0.2464 |
| DU^c | 0.2151 | 0.8029 | 0.3341 | 0.2573 | 0.7859 | 0.1824 |
| DF | 0.2610 | 0.7089 | 0.4776 | 0.3101 | 0.6582 | 0.4255 |
| DF^c | 0.2190 | 0.8713 | 0.6078 | 0.8081 | 0.3891 | 0.0104 |
| SU | 0.6302 | 0.2352 | 0.7058 | 0.5657 | 0.5822 | 0.5463 |
| SU^c | 0.3399 | 0.4633 | 0.6561 | 0.5424 | 0.9518 | 0.2365 |
| SF | 0.2268 | 0.8976 | 0.4570 | 0.7079 | 0.9604 | 0.2342 |
| SF^c | 0.1095 | 0.7537 | 0.8032 | 0.6608 | 0.8679 | 0.1552 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Q1. Do you expect activation steering to be more or less effective for other models/architectures?
- Q2. Will you be releasing code/data for how faithfulness was calculated in this particular case?
- Q3. Do you expect your results to be consistent across how faithfulness metric was defined? So, for example, experimenting with faithfulness metric with paraphrasing vs. early answering strategy?
soundness: 3
strengths: Overall, I thought the paper was strong and has potential for broad impact, because it connects so many concepts that are disparately considered. There has been a significant gap in evaluating *for* interventions, and this work systematically investigates the common and practical techniques for interventions.
- S1. Comprehensive evaluation of intervention methods for a widely used technique, CoT reasoning. Since this is how many researchers as well as practitioners interact with LLMs, this work is widely applicable and can have broad impact in considerations for AI safety.
- S2. I thought the introduction was particularly well motivated and the paper was generally well written.
- S3. Finetuning strategies were tested with multiple sampling strategy of their design. Adding faithfulness metric to the finetuning dataset creation was a particularly convincing experimental strategy.
- S4. Also introduces novel strategy for activation editing based on aligning on faithfulness vectors
- S5. The paper includes salient results, with most of these methods getting partial success. ICL or activation editing seem to get either accuracy or faithfulness performance enhancements, but rarely both. It seems that more finetuning on faithful datasets can improve both more so than ICL and activation editing
summary: In recent years, there have been concerted effort in making language models more faithful and robust with methods such as finetuning, in-context learning and activation editing. This work investigates whether these 3 methods can make CoT reasoning more faithful. Their findings suggest that all of them achieve very limited performance improvements, with activation editing achieving only minimal improvements. Finetuning and in-context learning can be slightly more effective, though they seem to fail to generalize across tasks.
weaknesses: - W1. It seems that activation editing was only experimented with LLaMA-3B. I wonder if this could have been an issue with this particular model, particularly because activation editing could have vastly different results depending on the architecture. For that reason, I think this result could be made more robust by adding other models for comparison such as Gemma or OLMo.

- W2. "Fine-tuning using most faithful explanations achieve better accuracy-faithfulness trade-offs." This seems like an expected result, but I wonder if this holds true across domain. If there could have been a more comprehensive strategy such as sampling by length for comparison, I wonder if there were any observable differences across domain.

- W3. There's slew of methods proposed by lanham et al, but I think this paper only discusses faithfulness with respect to early answering strategy. Faithfulness metric could result in different behavior based on the metric definitions: early answering vs. adding mistakes vs. paraphrase.

- W4. The faithfulness based activation editing strategy was introduced, but the results on it were not included in the paper.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: - What is the formal definition of faithful (CoT) reasoning in LLMs? Unless, I am missing something this was stated to be formally defined in line 93, but I fail to find this definition later in the manuscript.
soundness: 2
strengths: The paper targets an important and timely challenge in LLM. While massive effort is dedicated towards enhancing LLM's capability to reason or even demonstrating that it can reason, it still represents a major bottleneck and prevents using LLM to create AGI.  

Overall, the paper reads well and is well organised. The overall contribution is more technical and focuses on empirical studies of various combined techniques implemented, resulting in comprehensive experimental evaluations.
summary: Recent advances of foundation models, in particular Large Language Models (LLMs) have demonstrated impressive performances in many natural language processing tasks. Nevertheless the capabilities of LLMs at reasoning tasks are still limited and raises significant debate (see [1]). A line of recent works proposed prompt-based techniques to improve LLM capability including, but not limited to, reasoning.  Notably, the most popular techniques are: *chain-of-thought* (CoT) by adding the phrase  'think/solve step by step' at the end of the prompt, and *in-context learning* by including illustrative examples in the prompt to inspire or assist the LLM about the specific context of the query to solve; another line focuses on fine-tuning the LLM on formal reasoning benchmarks data, mathematical problems (Algebra, Geometry, calculus and so on).  

This work combines the three aforementioned techniques to improve LLMs in producing what is referred to as *faithful* CoT reasoning and rational explanations to the delivered output. Moreover, it define a metric to assess the concept of faithful CoT reasoning.  



[1] Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar: GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. EMNLP 2024
weaknesses: - The  paper constitutes incremental research work. The proposed solution is simply combining applied techniques in LLMs, which render the contribution incremental and straightforward. Technically, the contribution lacks in rigor, and many of the applied strategies are not formally justified.   

- The aforementioned techniques have shown several limitations, in past works, and more importantly in many cases techniques like activation patching are deteriorating the LLMs accuracy.   

- Several notions and techniques that this work builds upon, are not formally defined or described earlier in the paper, making it less accessible to a broader audience.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What is the sample size of the benchmark? Correct me if I am wrong but lines 339 - 348 describe original datasets' statistics instead. 
2. When selecting N ICL demonstrations, are we considering questions' similarities or just using faithfulness as the single index? 

Minor:
1. Figures' notation requires browsing around.
2. Please avoid directly using acronyms, a full expression would be more reader-friendly. e.g. out of distribution for OoD in line 303 
3. Please check typos in the manuscript, such as:
a. line 312, Figure 4?
b. line 354 asking the question *without* invoking?
soundness: 3
strengths: 1. **Detailed Experiment** The paper conducted thorough experiments across in-context learning, fine-tuning, and activation editing.

2. **Insights from the Experiment** The empirical experiments provided meaningful insights, which might inform a better LLM alignment methodology for achieving faithful intermediate states in the future.
summary: The paper systematically examines the Chain-of-Thought (CoT) behavior in large language models (LLMs) through experiments involving in-context learning, fine-tuning, and activation editing. The results indicate that activation editing had limited success, while in-context learning and fine-tuning led to only slight, non-generalizable improvements. The authors argue that the training process of these models does not prioritize faithfulness, which contributes to the generation of more self-aware content.
weaknesses: **Novelty in methodology** is the weakness of this position paper. As author(s) stated, changes are made for referenced methods/procedures, some theoretical/numerical supports can better validate the proposal. Please let me know if the following points are biased.

Here are some directions to consider:
1. To measure faithfulness, the Area Over the Curve (AOC) metric from [1] is adopted while the paper proposed to use probability scores for each instance instead of on the dataset level. However, section 2.3.1 of [1] also stated "AOC values are calculated as a weighted sum", thus [1] should also work on the instance level. I suggest editing line 166 to prevent confusion if this is the case.
2. For activation editing, this work selected top-K heads based on faithful probing results instead of top-K truth-relatedness heads in [2], they sound serving similar purposes to me. Can we compare these methods or see if they are transferable?

Reference
[1] Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., ... & Perez, E. (2023). Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702.
[2] Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2024). Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Insights on why faithfulness is difficult to learn, either in the form of mathematical theorems, or carefully designed experiments would be helpful.
soundness: 3
strengths: The paper provided experimental results indicating that in-context learning, fine-tuning, and activation editing did not result in substantial improvement in faithfulness of chain of thought reasoning. This suggests that other techniques may be required if this type of faithfulness is required.
summary: The paper did an empirical study on whether chain of thought reasoning can be made to accurately reflect the underlying reasoning done by the LLM (i.e. whether it can be made faithful) by in-context learning, fine-tuning, or activation editing. The faithfulness measurement tries to measure whether stopping the chain of thought early would results in different outcomes compared to using the full chain to answer the question; if it does not, it is an indication that the LLM already knows the answer before generating the chain and is doing post-hoc explanation of its reasoning in the chain rather than computing the answer within the chain. The study found that in-context learning, fine-tuning, and activation editing are all not successful in substantially improving faithfulness.
weaknesses: The paper provides negative results -- this is fine. However, to make a strong paper, insights that are supported by evidence on why the results are negative would be helpful.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: As the authors claim that "our work underscores the inherent difficulty in eliciting faithful CoT reasoning from LLMs, suggesting that the current array of approaches may not be sufficient to address this challenge", I wonder what could be revealed from the evaluation about the fundamental cause of the limitation for current LLM paradigms? Further, what could be the potential way to address them?
soundness: 2
strengths: (1) The topic of faithful reasoning of LLMs is interesting and sounds reasonable to investigate.

(2) By demonstrating the limited success of conventional strategies, the paper highlights the intrinsic difficulty of faithful reasoning in LLMs, which provides a strong basis for future exploration. ​

(3) The presentation is generally clear and easy to follow.
summary: This paper investigates the challenge of generating faithful Chain-of-Thought reasoning in large language models, specifically focusing on approaches like in-context learning, fine-tuning, and activation editing. While the authors highlight the importance of faithfulness in CoT reasoning for trustworthiness in high-stakes domains like healthcare, their empirical results suggest that none of these methods yield significant improvements in CoT faithfulness.
weaknesses: (1) The paper evaluates standard techniques like in-context learning, fine-tuning, and activation editing to improve CoT faithfulness, but these methods have already been extensively studied in other contexts such as improving accuracy, bias reduction, and factual consistency. The paper does not present any substantial technical adaptations or theoretical contributions to these methods specifically for faithful CoT reasoning. For example, while activation editing is discussed, it largely follows the framework of existing works like Li et al. (2024) without offering any new insights. The novelty of merely applying them to faithful CoT seems limited, and the contribution does not significantly advance the field beyond the status quo.

(2) The "early answering" metric used to evaluate faithfulness is based on whether truncating CoT reasoning affects the model's final output. However, the reason for taking it as the best way to measure faithfulness remains unclear, particularly given the complexity of CoT explanations. The measure seems too simplistic, as it fails to capture nuances in reasoning that may be faithful but not necessarily immediately reflected in the final answer. This could raise a misalignment between the metric and the goal of the research, which is to assess whether CoT explanations reflect the internal logic of the LLM.

(3) Although the paper acknowledges that none of the explored methods significantly improve CoT faithfulness, it does not provide a deep analysis of why these methods fail. For example, the results show only marginal gains in faithfulness, but the paper does not dive into what specifically causes this limitation—whether it is the inherent architecture of LLMs, the quality of training data, or other factors.

(4) While the paper claims to "lay the groundwork" for future research in trustworthy CoT reasoning, it does not propose concrete next steps or actionable insights based on the experimental findings. The conclusion merely restates that current methods are insufficient without suggesting innovative ideas or frameworks that could be explored in the future. This lack of direction limits the potential impact of the paper in advancing the field.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 4
questions: Please refer to the Weaknesses
soundness: 3
strengths: 1. This paper tackles the issue of enhancing the faithfulness of reasoning in large language models, which is vital for applications requiring high reliability.

2. The study is methodologically sound, with rigorous experiments across different models and datasets, demonstrating the limited effectiveness of current strategies in improving reasoning faithfulness.

3. The findings are impactful, highlighting the need for new methodologies to make LLMs more transparent and trustworthy, which is crucial for their adoption in high-stakes domains.
summary: The paper examines the difficulty of making large language models produce reasoning that accurately reflects their internal processes. It tests methods like in-context learning, fine-tuning, and activation editing and finds they only marginally improve a model's ability to produce faithful reasoning. The study concludes that current techniques are insufficient to ensure reasoning transparency in language models.
weaknesses: 1. The study focuses on a limited number of benchmarks. It would benefit from expanding the range of datasets to better understand how these findings generalize across different types of reasoning tasks and domains.

2. The paper could benefit from a more robust theoretical framework that explains why certain strategies might improve faithfulness while others do not.

Meta-review: The paper examined the Chain-of-Thought (CoT) behavior in large language models (LLMs) through experiments involving in-context learning, fine-tuning, and activation editing. This work combines the three aforementioned techniques to improve LLMs in producing what is referred to as faithful CoT reasoning and rational explanations to the delivered output. The authors argue that the training process of these models does not prioritize faithfulness, which contributes to the generation of more self-aware content. The paper targets an important and timely challenge in LLM with detailed experiment design. While there are several major concerns remain regarding the novelty of the proposed method and the investigation of the empirical results. The proposed solution is combining applied techniques in LLMs, which render the contribution incremental and straightforward. Technically, the contribution lacks in rigor, and many of the applied strategies are not formally justified. Furthermore, there is lack of insights on the empirical results. For instance, why faithfulness is difficult to learn, either in the form of mathematical theorems, or carefully designed experiments would be helpful. And it remains unclear why “early answering” is a reasonable metric for measuring faithfulness, particularly for black-box LLMs. Given the above reasons, after discussion with the reviewers, we would encourage the authors to expand their analysis and propose more innovative approaches in future iterations of this work for resubmission.

Decision: Reject