Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness

## Abstract

Abstract As the modern tools of choice for text understanding and generation, large language models (LLMs) are expected to accurately output answers by leveraging the input context.
This requires LLMs to possess both context-faithfulness and factual accuracy.
Extensive efforts have been made to enable better outputs from LLMs by mitigating hallucinations through factuality enhancement methods.
However, they also pose risks of hindering context-faithfulness, as factuality enhancement can lead LLMs to become overly confident in their parametric knowledge, causing them to overlook the relevant input context.
In this work, we argue that current factuality enhancement methods can significantly undermine the context-faithfulness of LLMs.
We first revisit the current factuality enhancement methods and evaluate their effectiveness in enhancing factual accuracy.
Next, we evaluate their performance on knowledge editing tasks to assess the potential impact on context-faithfulness.
The experimental results reveal that while these methods may yield inconsistent improvements in factual accuracy, they also cause a more severe decline in context-faithfulness, with the largest decrease reaching a striking 69.7%.
To explain these declines, we analyze the hidden states and logit distributions for the tokens representing new knowledge and parametric knowledge respectively, highlighting the limitations of current approaches.
Our finding highlights the complex trade-offs inherent in enhancing LLMs.
Therefore, we recommend that more research on LLMs’ factuality enhancement make efforts to reduce the sacrifice of context-faithfulness.

## 1 Introduction

Leveraging their powerful capabilities in text understanding and generation, large language models (LLMs) can effectively integrate contextual information to infer and generate truthful responses  (OpenAI, ; Touvron et al., ; ; Song et al., ).
However, hallucinations (Huang et al., ; Tonmoy et al., ) significantly undermine the LLMs’ reliability, primarily due to inadequate adherence to contextual instructions and failure to produce truthful responses (Zhang et al., ).

With the widespread adoption of Retrieval-Augmented Generation (RAG) (Fan et al., ; Santhanam et al., ; Schick et al., ; Qin et al., ), , the study of context-faithfulness in LLMs is becoming increasingly important  (Chen et al., ; Li et al., ).
This is because they depend on the effective integration of retrieved external information into their generated responses to ensure contextual relevance.
Therefore, exceptional LLMs must not only produce factual outputs based on their parametric knowledge but also comply with contextual requirements.
Context-faithfulness enhances LLMs’ compliance with user instructions and improves performance, particularly when the parametric knowledge is insufficient or outdated.

Prior work (Li et al., ; Zhang et al., ) suggests that the cause of factual hallucinations is LLMs’ inability to accurately “convey” knowledge during inference, even when it has been effectively “learned” during pretraining.
Some emerging works focuses on narrowing the gap between ”knowing” and ”telling” in LLMs to improve LLMs’ factuality.
These approaches can be divided into two categories: one involves editing internal representations, such as attention mechanisms (Li et al., ; Burns et al., ; Chen et al., ) or hidden states (Zhang et al., ), and the other modifies the logits of tokens during decoding (Li et al., ; Chuang et al., ; Zhang et al., ; Kai et al., ).
These methods enable LLMs to output their knowledge more accurately, reducing hallucinations, without altering parameters, as they do not require introducing new factual knowledge through supervised fine-tuning (SFT) (Yang et al., ; Ovadia et al., ) or RLHF (Ouyang et al., ; Bai et al., ).

Figure: Figure 1: Comparison of responses from native LLMs and factuality-enhanced LLMs before and after context injection. The enhanced LLMs, due to overconfidence in their parametric knowledge, struggle to integrate new information from the context, resulting in incorrect answers.
Refer to caption: x1.png

However, is this unsupervised factuality enhancement a free lunch for LLMs?
We argue that while factuality enhancement can reduce factual hallucinations in LLM outputs, it may lead to excessive confidence in their parametric knowledge, making it difficult for them to comply with contextual information, especially when external knowledge contradicts their parametric knowledge (Petroni et al., ; Si et al., ; Xie et al., ).
Figure 1 illustrates a simple example of the scenario we envisioned.
In this work, we first revisit several popular methods for factuality enhancement and evaluate their performance on TruthfulQA (Lin et al., ).
The results show that while most methods provide some improvement in factuality, some approaches exhibit minimal or inconsistent gains.
Therefore, we suspect that current factuality enhancement methods, despite improving factuality, might negatively impact context-faithfulness to a greater extent.

To test our hypothesis, we evaluate these factuality enhancement methods on knowledge editing (KE) (Sinitsin et al., ; De Cao et al., ; Mitchell et al., ; Yao et al., ) benchmarks, comparing them against the baseline performance of LLaMA2-chat.
Specifically, we conduct experiments with in-context editing (ICE) (Zhong et al., ; Zheng et al., ; Bi et al., ) methods on the MQuAKE datasets.
The accuracy of ICE effectively reflects the context-faithfulness of LLMs, as it provides outdated or counterfactual contextual knowledge that may contradict the models’ parametric knowledge.
Experimental results indicate that, compared to the baseline, current popular factuality enhancement methods lead to a significant decline in LLM performance in context-faithfulness.
For example, the editing accuracy of TruthX (Zhang et al., ) on LLaMA2-7B-chat decreased by as much as 60.3%.
This suggests that while factuality enhancement aids LLMs in accurately conveying factual information, it also introduces the risk of overconfidence, resulting in a substantial setback for the context-faithfulness of LLMs.

We explain the decline in context-faithfulness by analyzing the hidden states and token-wise distributions of LLMs’ outputs, based on knowledge tokens captured by our specially designed algorithm.
Compared to the baseline, the hidden states become more concentrated and closely align with the original parametric outputs after inserting contextual information.
Additionally, the logits for tokens representing new knowledge decrease, while the logits for parametric knowledge increase.

Altogether, our study highlights the significant risks posed by current factuality enhancement methods, as we validate and explain their notable decline in context-faithfulness.
This demonstrates the importance of ensuring that LLMs not only enhance factual accuracy but also maintain a strong alignment with the provided context to minimize the occurrence of hallucinations.
We strongly advocate for the development of LLMs that prioritize both faithful adherence to context and the accurate conveyance of factual information.
Therefore, we recommend future research explore strategies to balance these two critical aspects.

## 2 Preliminary

This section revisits the core concepts of language modeling (Section 2.1) and knowledge editing (Section 2.2) of LLMs, with the aim of conducting a more in-depth exploration of existing factuality enhancement strategies (Section 3) and their interpretability (Section 5), as well as the use of knowledge editing to assess context-faithfulness (Section 4).

### 2.1 Language Modeling

The current language modeling of LLMs aims to generate text based on the probability distribution of token sequences.
Formally, given a sequence of tokens ${x}_{t}=\{x_{1},x_{2},\ldots,x_{t-1}\}$, the goal is to predict the next token $x_{t}$ by modeling the conditional probability $\text{I\kern-1.49994ptP}(x_{t}|x_{<t})$.

To achieve this, LLMs process the input token sequence through layers of attention (Attn) and feed-forward networks (FFN).
The hidden state is computed as follows:

$$ $\text{I\kern-1.49994ptH}_{t}=\text{FFN}(\text{Attn}(\mathbf{X}_{<t}))$ (1) $$

Here, $\mathbf{X}_{<t}$ represents the embedded representation of the input tokens ${x}_{<t}$, which is passed through Attn to capture contextual relationships, followed by a FFN to produce the hidden state.

Then, the model can predict the probability distribution of the next token $x_{t}$ over the vocabulary set $\mathcal{V}$ by applying a linear transformation $\phi(\cdot)$ followed by a softmax function:

$$ $\text{I\kern-1.49994ptP}(x_{t}|x_{<t})=\mathrm{softmax}(\phi(\text{I\kern-1.49 994ptH}_{t}),\quad x_{t}\in\mathcal{V}$ (2) $$

The softmax function converts the output logits from $\phi(\text{I\kern-1.49994ptH}_{t})$ into a probability distribution over the vocabulary $\mathcal{V}$, allowing the model to assign a likelihood to each possible next token.
At decoding time, various decoding strategies can be applied at each step $i$ to select the next token $x_{i}$, with the given predicted distribution of the next token $\text{I\kern-1.49994ptP}(x_{t}|x_{<t})$.
This iterative process continues until the sequence generation reaches a designated end token or meets a predefined stopping condition.

### 2.2 Knowledge Editing

Context-faithfulness requires LLMs to effectively adhere to external context.
This means they must fully accept new knowledge provided by that context, even if it conflicts with their parametric knowledge acquired during pretraining, which may be insufficient or outdated.
Therefore, this paper uses the success rate of KE to assess the context-faithfulness of LLMs, as KE provides counterfactual new knowledge that contradicts the LLMs’ own parametric knowledge.

KE aims to efficiently adjust the behavior of the original model $f_{base}$ into post-edit model $f_{e}$ with a specific editing descriptors $z_{e}$.
The editing descriptors $z_{e}$ describe a desired change in model behavior and can be represented as $z_{e}=(x_{e},r_{e},y_{e})$, where $(x_{e},r_{e},y_{e})$ represents a triplet such as (US, President, Joe Biden) meaning Joe Biden is the president of US.
The ultimate objective of KE is to edit the LLM output that $f_{e}(x_{e},r_{e})=y_{e}$ while $f_{base}(x_{e},r_{e})\neq y_{e}$.

A thorough edit should not only modify the relevant knowledge but also update all knowledge affected by this edit in multi-hop relationships.
For instance, consider the two-hop fact triple (WWE Velocity, created by, Vince McMahon) and (Vince McMahon, spouse, Linda McMahon). With a fact edit $z_{e}=(\textit{WWE Velocity},\textit{created by},\textit{Stan Lee})$ and an additional fact $(\textit{Stan Lee},\textit{spouse},\textit{Joan Lee})$, the correctly updated answer should be Joan Lee. We adopt a multi-hop editing task to measure context-faithfulness of LLMs in Section 4 by evaluating the accuracy of multi-hop question answering with fact edits.
This poses challenges for LLMs in adhering to the editing context due to potential conflicts between new knowledge and their parametric knowledge during the reasoning process.

## 3 Factuality Enhancement for LLMs

### 3.1 Current Approaches to Factuality Enhancement

SFT or RLHF methods(Yang et al., ; Ovadia et al., ; Ouyang et al., ; Bai et al., ) enhance the factuality of LLMs by injecting external training data.
However, this approach significantly increases costs and introduces unreliability, potentially jeopardizing the integrity of the model’s inherent knowledge.
Current methods for enhancing factuality do not modify the underlying structure or parameters of LLMs; instead, they focus on improving how LLMs convey the information they have already learned to reduce hallucinations.
These approaches primarily involve modifying the outputs of LLMs, including the hidden states obtained through Attn and FFNs (eq. 1), as well as the logits distribution used to predict the next token during the decoding phase (eq. 2).
The current factuality enhancement methods can be classified into the following two categories based on their different approaches to modifying LLM outputs:

#### Representation Editing

Representation editing methods (Burns et al., ; Li et al., ; Zhang et al., ; Chen et al., ) enhance factuality by editing the internal representations of LLM outputs.
They learn a direction within attention heads or a potential truthful space to modify the attention patterns of the LLM, thereby adjusting the hidden states.

#### Contrastive Decoding

Contrastive decoding methods (Li et al., ; Chuang et al., ; Zhang et al., ; Kai et al., ; Bi et al., ) enhance the factuality of LLMs by comparing the output probabilities of expert and amateur models, different layers within the transformer, various tokens, and the outputs of normal and hallucination-injected models.

**Table 1: Experimental results of factuality evaluation on TruthfulQA and FActScore. We conduct experiments with various factuality enhancement methods, including both contrastive decoding and representation editing, on LLaMA2-chat. Methods marked with ^♠ belong to contrastive decoding, while those marked with ^♢ belong to representation editing. In FActScore, % Resp. indicates the response ratio of LLMs, while # Facts represents the number of extracted atomic facts per response. Here, red highlights a decrease in factuality compared to the Baseline.**
| Model | Method | TruthfulQA | FActScore |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MC1 (%) | MC2 (%) | MC3 (%) | Resp. (%) | Facts (#) | Score (%) |  |  |
|  | Baseline | 37.6 | 54.6 | 28.1 | 37.5 | 45.7 | 63.8 |
| LLaMA2- | DoLa^♠ | 32.9 | 60.8 | 29.5 | 40.7 | 48.7 | 61.3 |
| 7b-chat | ICD^♠ | 46.3 | 69.1 | 41.2 | 36.1 | 46.6 | 66.3 |
| ITI^♢ | 37.0 | 54.7 | 27.8 | 41.9 | 40.8 | 62.4 |  |
|  | TruthX^♢ | 54.2 | 73.9 | 44.3 | 40.6 | 43.7 | 65.3 |
|  | Baseline | 37.7 | 55.6 | 28.2 | 77.0 | 37.6 | 52.5 |
| LLaMA2- | DoLa^♠ | 37.3 | 61.8 | 32.7 | 55.8 | 46.7 | 59.8 |
| 13b-chat | CD^♠ | 28.2 | 54.9 | 29.8 | 74.2 | 39.8 | 53.5 |
| ICD^♠ | 45.6 | 67.5 | 42.3 | 46.9 | 43.4 | 59.5 |  |
|  | ITI^♢ | 38.9 | 53.4 | 29.6 | 78.9 | 34.5 | 55.3 |

### 3.2 Evaluation of Factuality

We evaluate the performance of popular factuality enhancement methods on the LLaMA2-7B-chat and LLaMA2-13B-chat models in terms of factuality of LLMs using the TruthfulQA (Lin et al., ) and FActScore (Min et al., ) benchmarks.
Evaluation on both benchmarks adheres to the settings of previous studies (Chuang et al., ; Li et al., ; Zhang et al., ).
Using TruthfulQA, we employ a multiple-choice task where the LLM selects an answer from various correct and incorrect options, evaluated through multiple-choice accuracy (MC1, MC2, and MC3).
For FActScore, the assessment is conducted through retrieve+chatGPT methodology.
The implementation details of the factuality enhancement methods are in Appendix A.

**Table 2: Experimental results of context-faithfulness evaluation on MQuAKE dataset. $k$-shot represents the number of context demonstrations provided, which is detailed in Figure 2 along with the use of COT. In the subscript ($\downarrow\Delta$), $\Delta$ (%) represents the decrease compared to the baseline.**
| Model | Method | 3-shot | 5-shot | 10-shot | COT (10-shot) |
| --- | --- | --- | --- | --- | --- |
|  | Baseline (-) | 68.5 (-) | 78.7 (-) | 81.3 (-) | 82.5 (-) |
| LLaMA2- | DoLa^♠ | 52.3 ($\downarrow$ 22.4) | 63.7 ($\downarrow$ 15.0) | 66.5 ($\downarrow$ 14.8) | 62.5 ($\downarrow$ 20.0) |
| 7B-chat | ICD^♠ (Zhang et al., ) | 50.1 ($\downarrow$ 26.9) | 61.7 ($\downarrow$ 17.0) | 64.9 ($\downarrow$ 16.4) | 60.5 ($\downarrow$ 22.0) |
| ITI^♢ (Li et al., ) | 51.9 ($\downarrow$ 21.8) | 62.2 ($\downarrow$ 16.5) | 67.5 ($\downarrow$ 13.8) | 65.7 ($\downarrow$ 16.8) |  |
|  | TruthX^♢ (Zhang et al., ) | 26.7 ($\downarrow$ 61.1) | 32.5 ($\downarrow$ 58.8) | 32.7 ($\downarrow$ 59.9) | 26.5 ($\downarrow$ 67.9) |
|  | Baseline | 78.3 (-) | 84.5 (-) | 86.1 (-) | 86.3(-) |
| LLaMA2- | DoLa^♠ (Chuang et al., ) | 58.3 ($\downarrow$ 20.0) | 67.7 ($\downarrow$ 16.8) | 70.2 ($\downarrow$ 15.9) | 70.3 ($\downarrow$ 16.0) |
| 13B-chat | ICD^♠ (Zhang et al., ) | 59.5 ($\downarrow$ 23.2) | 66.2 ($\downarrow$ 18.3) | 72.4 ($\downarrow$ 13.7) | 67.6 ($\downarrow$ 18.7) |
| ITI^♢ (Li et al., ) | 64.3 ($\downarrow$ 14.0) | 69.8 ($\downarrow$ 14.7) | 74.2 ($\downarrow$ 11.9) | 65.7 ($\downarrow$ 20.6) |  |
|  | CD^♠ (Li et al., ) | 55.2 ($\downarrow$ 29.5) | 62.6 ($\downarrow$ 21.9) | 67.1 ($\downarrow$ 19.0) | 64.9 ($\downarrow$ 21.4) |

The experimental results of LLM factulity on LLaMA2-chat are presented in Table 1.
A notable observation is that, compared to the baseline, factuality enhancement methods show improvements across most metrics.
Without altering the structure or parameters of LLMs, these methods reduce the gap between ’know’ and ’tell’ knowledge to mitigate hallucinations, making such improvements highly valuable.
However, these methods still have significant limitations.
The improvement in factuality is modest and quite unstable.
The red-marked sections in Table 1 indicate that the evaluated methods do not show consistent enhancements across all metrics;
in fact, some metrics exhibit declines compared to the baseline.
This suggest that these methods for modifying LLM outputs may negatively impact the model’s natural generation capabilities.

Figure: Figure 2: An illustration of the ICE task to evaluate LLMs’ context-faithfulness. It involves multi-hop question answering with corresponding edits, utilizing $k$ contextual demonstrations to guide editing and align output format. The highlighted text represents the expected output from the LLMs, while Thoughts indicates the additional step taken when using Chain of Thought (COT) reasoning.
Refer to caption: x2.png

## 4 Evaluation of Context-Faithfulness

The factuality enhancement methods increase LLMs’ confidence in their parametric knowledge to produce more accurate outputs.
However, we argue that this can lead to overconfidence, which reduces their faithfulness to context and consequently introduces new risks of hallucination.
To validate this suspicion, we evaluates the impact of these factuality enhancement methods on context-faithfulness.

### 4.1 Experimental Setup

#### Task

We meticulously design the In-Context Editing (ICE) (Zheng et al., ; Zhong et al., ; Wang et al., ; Bi et al., ; ) task to evaluate the context-faithfulness of LLMs.
As the most effective KE method currently, ICE prompts LLMs to reason and generate answers that faithfully incorporate new knowledge retrieved from edit memory in the form of context.
We configure the edit memory to include only question-relevant editing instances, ensuring that all provided editing context influences the LLMs’ outputs.
We assess context-faithfulness based on the accuracy of question-answering with edits, as this reflects whether LLMs remain faithful to external context when it conflicts with their internal knowledge. Figure 2 provides a simple illustration of our task, with additional contextual demonstrations listed in the Appendix B.2.

#### Models and Datasets

We use LLaMA2-7B-chat and LLaMA2-13B-chat as baseline models to conduct experiments on MQuAKE (Zhong et al., ) (^1^11We specifically use the released MQuAKE-CF-3k-v2 data version for our experiments, which addresses the internal knowledge conflicts within the dataset.), comparing various factuality enhancement methods against the natural LLaMA2-chat models.
MQuAKE provides multi-hop knowledge questions to evaluate knowledge editing on counterfactual edits, as introduced in Section 2.2.
This verifies whether knowledge has been properly edited and effectively demonstrates LLMs’ context-faithfulness.
CD compares the decoding logits of the 13B version of LLaMA2-chat with those of the 7B version, using LLaMA2-13B-chat as the baseline for comparison.
Implementation details of the factuality enhancement methods can be found in the Appendix A.

**Table 3: Experimental results of context-faithfulnes evaluation using MeLLo method.**
| Model | Baseline | DoLa^♠ | CD^♠ | ICD^♠ | ITI^♢ | TruthX^♢ |
| --- | --- | --- | --- | --- | --- | --- |
| LLaMA2-7B-chat | 54.5 (-) | 39.4 ($\downarrow$ 27.7) | - | 29.8 ($\downarrow$ 45.3) | 31.5 ($\downarrow$ 42.2) | 22.6 ($\downarrow$ 58.5) |
| LLaMA2-13B-chat | 63.8 (-) | 35.6 ($\downarrow$ 44.2) | 38.6 ($\downarrow$ 39.5) | 39.5 ($\downarrow$ 52.2) | 34.5 ($\downarrow$ 45.9) | - |

### 4.2 Main Results

Figure: Figure 3: Changes (%) in Context-Faithfulness and Factuality of Factuality-Enhanced LLMs.
Refer to caption: x3.png

Based on the above experimental setup, we evaluate the context-faithfulness of current factuality enhancement methods.
The experimental results of ICE on LLaMA2-7B-chat and LLaMA2-13B-chat are shown in Table 2.
As observed, all factuality enhancement methods exhibit a marked decline compared to the baseline in performance on the ICE task under any setting.
Notably, this decline is not minor, with an average decrease exceeding 20%, and occurs without exception across all methods.
Particularly in the state-of-the-art TruthX for factuality, the decline reaches as high as 67.9%.
This suggests that the potential risks to context-faithfulness could be significant.

An interesting observation is that the accuracy of all methods, including the baseline, improves as the number of context demonstrations increases.
This aligns with the findings of  Zheng et al. ()., suggesting that more faithful examples can enhance the context-faithfulness of LLMs.
Additionally, we find that introducing Chain of Thought (COT) (Wei et al., ) reasoning into the baseline improves its accuracy.
However, the performance of COT in factuality enhancement methods is unstable, with most methods showing a decline compared to those without COT.
This may be due to the fact that during the additional COT reasoning process, LLMs with factuality enhancements tend to recall parametric knowledge more, thereby negating the new knowledge from the context. This can be observed in the cases discussed in Section 5.4.

Figure 3 visualizes the changes in both factuality and context-faithfulness of these methods compared to the baseline, based on the results of MC2 in Table 1 and 3-shot in Table 2.
It illustrates that while these factuality enhancement methods provide unstable improvements in factuality, they also lead to a more significant decline in the context-faithfulness of LLMs.

In addition to the aforementioned basic ICE setup, we also employ the advanced KE method MeLLo (Zhong et al., ) to assess LLMs’ context-faithfulness.
MeLLo can decompose complex multi-hop questions into sequential sub-questions, checking for conflicts with relevant new knowledge retrieved from edit memory to determine if edits are necessary.
Therefore, we further validate context-faithfulness using MeLLo, as it performs one-hop edits, thereby avoiding hallucinations introduced by multi-hop reasoning.
The prompt template and implementation details of MeLLo are provided in the Appendix B.1.
The results in Table 3 show that, when using MeLLo, which is more suited for realistic editing scenarios, the accuracy of factuality-enhanced LLMs exhibits a more significant decline compared to the previous simple ICE evaluation.
Most declines exceed 40% relative to the baseline, with larger models experiencing even more pronounced decreases.
All of the editing results that these factuality enhancement methods significantly reduce the accuracy of knowledge editing tasks, highlighting that the current approaches severely impact the context-faithfulness of LLMs.

## 5 In-depth Exploration of Context-Faithfulness

The evaluation of context-faithfulness in Section 4 confirms the earlier suspicion that improved factuality leads to a significant decline in context-faithfulness.
This section aims to further explain this decline from a model interpretability perspective.
We investigate the logit distributions (Section 5.2) and hidden state representations (Section 5.3) of relevant knowledge in LLM outputs captured by our algrithm (Section 5.1), and conduct case study (Section 5.4) based on the generated text, to reveal how these factuality enhancement methods impact context-faithfulness.

Figure: Algorithm 1 Knowledge Token Capturing

### 5.1 Distinguishing New and Parametric Knowledge in LLM Outputs

Our objective is to identify the parts of LLM outputs that distinguish between the new knowledge acquired from context and the parametric knowledge embedded within the LLMs, rather than analyzing repetitive or meaningless outputs.
For example, consider an expected LLM output with injected context, such as “Answer: United States”, compared to the original parametric output without the injected context, which would be “Answer: United Kingdom”.
In this case, we should not capture “Answer:” as it holds no factual meaning, nor should we focus on “United” since it is repetitive and does not reflect the difference.
Instead, our focus should be on capturing tokens with clear factual significance—those that can differentiate between the new knowledge introduced by context and the parametric knowledge inherent to the model. In this example, tokens like “Kingdom” serve as the key markers of distinction, effectively highlighting the critical divergence between the contextual information and the model’s existing knowledge.

To achieve this, we design a novel knowledge token capturing algorithm, with details presented in Algorithm 1.
The algorithm captures the highest-probability tokens that differentiate new and parametric knowledge by matching the decoded tokens with the respective knowledge strings.
We conduct experiments on the MQuAKE-stubborn dataset provided by Bi et al. (), which effectively highlights the distinction between new knowledge and parametric knowledge.
Specifically, we collect 2,000 question-answer instances, each consisting of the output from LLMs prior to injecting context containing new knowledge, as well as the output after the injection.

Figure: Figure 4: Average logits of tokens representing new and parametric knowledge after context injection.
Refer to caption: x4.png

### 5.2 Analysis in Logits Distribution

To further investigate LLM context-faithfulness from the perspective of logits distribution (eq. 2), we capture the tokens corresponding to both the in-context new knowledge and the parametric knowledge after context injection.
The average probabilities of these tokens for the factuality enhancement methods and baselines are shown in Figure 4.
We observe that, compared to the baseline, all factuality enhancement methods show a decrease in the probability of new knowledge tokens, while the probability of parametric knowledge tokens increases.
This provides a clear explanation for the reduction in context-faithfulness discussed in Section 4.
These methods effectively boost the retention of parametric knowledge but at the cost of diminished faithfulness to external context.
Furthermore, the difference in probabilities between new and parametric knowledge correlates well with the editing accuracy in Table 2, where larger differences lead to better editing performance, while TruthX’s poor editing is explained by the dominance of parametric knowledge over new knowledge.

Figure: Figure 5: Statistics of new knowledge distribution for both logits and rank on LLaMA2-chat (left) and LLaMA2-13B-chat (right). Rank is recorded by capturing the position of the new knowledge token within the vocabulary $\mathcal{V}$, based on its logits ranking.
Refer to caption: x5.png

Results in figure 5 provides a deeper statistical analysis of the logits associated with new knowledge.
The baselines of the LLaMA2-chat models consistently exhibit a higher concentration in the probability range of $0.8$ to $1.0$, with top token ranks (1 to 5) clearly dominating.
In contrast, factuality enhancement methods shift the distribution of new knowledge tokens towards lower probability ranges, indicating a diminished confidence in in-context knowledge compared to the baselines.

### 5.3 Analysis in Hidden States

Section 3 shows that all factuality enhancement methods impact the logits distribution of the output, resulting in a decline in context-faithfulness.
Some methods also aim to improve factuality by editing the internal representations (eq. 1) of LLMs, which requires an analysis based on hidden states.
We similarly use theknowledge token capturing algorithm to collect these crucial hidden states.
Since the contrastive decoding method does not alter internal representations, our evaluation focuses on the ITI (Li et al., ) and TruthX (Zhang et al., ) methods.
We used t-SNE to reduce the dimensionality of the hidden states representing new knowledge in LLaMA2-7B-chat from the original 4096 dimensions to 2 dimensions, with the perplexity set to 15.

Figure: Figure 6: Distribution of reduced hidden states for new knowledge before and after context injection. The dashed circles represent the 95% confidence intervals, showing that factuality enhancement methods lead to a post-injection convergence towards the pre-injection distribution.
Refer to caption: x6.png

Figure 6 presents the distribution of hidden states before and after context injection for these representation editing methods.
We find that the distribution of the reduced hidden states after context injection is broader and more dispersed compared to before the injection.
This indicates that context injection alters the distribution of knowledge representations, leading them to deviate from their original factuality, where the hidden states prior to injection represent the parametric factuality inherent in the LLMs.
Notably, compared to the significant difference observed between the pre- and post-context injection hidden states in the baseline, the differences for ITI and TruthX are much smaller.
The hidden states of new knowledge after injection are more confined and closer to those before the injection.
Especially, TruthX exhibits a distribution that closely resembles the original range, which explains its significant decline in context-faithfulness.
This is because it modifies both the FFN and Attn representations, making them more resistant to incorporating new knowledge and more firmly tied to the model’s parametric knowledge.
Overall, these representation editing methods tend to limit the natural dispersion of hidden states after context injection.
As a result, the output decoded from the constrained hidden states remains more aligned with the original parametric knowledge, rather than incorporating the new contextual knowledge effectively.

### 5.4 Case Study

We conduct a case study on the outputs of factuality enhancement methods in the ICE task (Section 4).
We classify the context-unfaithful outputs into three categories: ignoring context, denying context, and misformatting context, with illustrative examples for each type provided in Table 4.
Beyond knowledge conflicts causing the context to be overlooked or contradicted, failing to properly follow the context demonstration format also indicates a lack of context-faithfulness.
Additional examples of real context-unfaithful outputs are provided in Appendix C for further examination.

## 6 Related Work

#### Hallucinations in LLMs

Hallucinations (Kaddour et al., ; Tonmoy et al., ; Wang et al., ; mei2024not; mei2024hiddenguardfinegrainedsafegeneration) have garnered significant attention due to their pronounced side effects, leading to the generation of unreliable content (Gunjal et al., ; Huang et al., ; Liu et al., ; Zhang et al., ).
These hallucinations can arise from various sources at different stages of the LLM lifecycle.
The primary reasons include conflicts between the context provided and the LLMs’ inherent memory, as well as the models’ inability to accurately generate knowledge that they have already acquired during pretraining (Zhang et al., ; Chen et al., ).
This paper primarily explores the interplay between these two aspects.

#### Knowledge Conficts

Various tools (Nakano et al., ; Yao et al., ; Qin et al., ) and retrieval-augmented methods (Guu et al., ; Izacard & Grave, ; Zhong et al., ), such as ChatGPT Plugins and New Bing, have been introduced as effective strategies to supply external knowledge evidence. However, the integration of external knowledge is not without challenges, as it can sometimes clash with the LLMs’ parametric knowledge (Petroni et al., ; Si et al., ; Xie et al., ; Mei et al., ; Ni et al., ; ), resulting in inconsistencies or unreliable outputs, especially when LLMs exhibit overconfidence in their inherent parametric knowledge. This conflict between external sources and the internal knowledge stored in LLMs continues to pose significant challenges for ensuring reliable model performance.
Therefore, we argue that current factuality enhancement methods may actually reinforce the model’s reliance on its own parametric knowledge, exacerbating knowledge conflicts and making it more difficult for LLMs to adapt to or follow the injected context.

#### Knowledge Editing

Knowledge editing (KE) (Yao et al., ) has been proposed to update outdated model knowledge, ensuring accurate responses to current queries.
Generally, Current KE approaches can be divided into following two categories.
Model editing (Meng et al., ; ; Mitchell et al., ; Yao et al., ; Xu et al., ) focuses on identifying and modifying internal components, such as neurons or parameter matrices.
In-context editing (Madaan et al., ; Zhong et al., ; Zheng et al., ; Wang et al., ; Bi et al., ) retrieves relevant knowledge from external memory during inference, allowing LLMs to utilize updated information dynamically.

**Table 4: Case study of context-unfaithful outputs from factuality-enhanced LLMs. The main categories include ignoring, denying new knowledge from the context, and not following the context format in responses. Red text highlights the unfaithful parts or reasons.**
| Question: What is the official language of the country where the creator of WWE Velocity’s spouse is a citizen? |  |  |  |
| --- | --- | --- | --- |
| Injected Context: [k context demonstrations]<br>Edit: The creator of WWE Velocity is Stan Lee. The official language of US is German. |  |  |  |
| Context-faithful output | Ignoring context | Denying context | Misformatting context |
| Thoughts: WWE Velocity was created by Stan Lee, Joan Lee is the spouse of StanLee, Joan Lee is a citizen of US, the official language of US is German.<br>Answer: German. | Thoughts: WWE Velocity was created by Vince McMahon, Vince McMahon’s spouse is Linda McMahon, Linda McMahon is a citizen of US, the official language of US is English.<br>Answer: English. | Thoughts: WWE Velocity was created by Stan Lee, Joan Lee is the spouse of StanLee, Joan Lee is a citizen of US, the official language of the US is not German; it is primarily English.<br>Answer: English. | The official language is German, based on the provided context. However, this is incorrect. WWE was created by Vince McMahon and the official language of the U.S. is English. (Do not follow the context demonstration format) |

## 7 Conclusion and Discussion

In this paper, we conduct an in-depth exploration of popular factuality enhancement methods and their implications for LLMs.
We argue that existing approaches to enhancing factuality can significantly undermine the context-faithfulness of LLMs.
Through evaluations on factual question-answering tasks and knowledge editing scenarios, we demonstrated that while these methods may yield unstable improvements in factual accuracy, they also lead to a substantial decrease in editing accuracy.

By examining the logits distribution and hidden states, we gained insights into the underlying causes of this decline.
Our experimental results further validate our concerns: improved factuality achieved through these methods can indeed result in diminished context-faithfulness of LLMs.
This finding highlights the complex trade-offs inherent in enhancing LLMs.

Ultimately, a high-performing LLM should possess both factual accuracy and context-faithfulness.
Therefore, we recommend that future research should focus on developing strategies to controllably and effectively balance these two critical aspects.
By addressing this duality, we can work towards creating LLMs that not only provide accurate information but also maintain a robust adherence to contextual cues, enhancing their overall reliability and usefulness.

## Ethics Statement

Ethical considerations are of utmost importance in our research endeavors. In this paper, we conscientiously adhere to ethical principles by exclusively utilizing open-source datasets and employing models that are either open-source or widely recognized in the community. Moreover, our proposed method is designed to ensure that the model does not produce any harmful or misleading information. We are committed to upholding ethical standards throughout the research process, prioritizing transparency, and promoting the responsible use of technology for the betterment of society.

## Appendix A Details of the factuality enhancement

This paper focuses on evaluating and analyzing popular factuality enhancement methods.
We provide a detailed overview of these approaches, along with the implementation details used in our experiments.

#### CD.

Contrastive decoding (CD) (Li et al., ) leverages the distinctions between expert and amateur LMs of varying sizes by selecting tokens that maximize the difference in their log-likelihoods.
Consequently, factual knowledge that remains unlearned by the weaker amateur model is highlighted by contrastive decoding in the stronger expert model to enhance factuality.

#### DoLa.

DoLa (Chuang et al., ) leverages a modular encoding of knowledge to magnify factual knowledge within an LM through a contrastive decoding approach. In this method, the next-word probability output is derived from the disparity in logits between a higher layer and a lower layer. By accentuating the knowledge from higher layers and diminishing that from lower layers, LoRa aims to reduce factual hallucinations.

#### ICD.

Zhang et al. () first constructs a factually weak LLM by inducing hallucinations from the original LLMs, and then penalizes these induced hallucinations during decoding to enhance the factuality of the generated content.
Specifically, ICD determines the final next-token predictions by amplifying the predictions from the original model and downplaying the induced untruthful predictions via contrastive decoding.

#### ITI.

Inference-Time Intervention (ITI) (Li et al., ) first identifies a sparse set of attention heads with high linear probing accuracy for truthfulness, as defined by the TruthfulQA benchmark.
Then, during inference, it shifts activations along these truth-correlated directions.
This process is repeated autoregressively until the entire answer is generated.

#### TruthX

TruthX (Zhang et al., ) uses an autoencoder to map the representations of LLMs into both semantic and factual latent spaces, and applies contrastive learning to identify the correct editing direction within the factual space. During inference, TruthX enhances the factuality of LLMs by editing their internal representations in the factual space.

We employ two different sizes of LLMs, LLaMA2-7B-chat and LLaMA2-13B-chat, using the unchanged decoding strategy as the baseline. Consistent with prior studies, we apply robust factuality enhancement strategies.
For DoLa, we configure early-exit layers for both sizes of the LLaMA2-chat model according to the guidelines provided in (^2^22[https://github.com/voidism/DoLa](https://github.com/voidism/DoLa)).
For CD, we utilize the 13B and 7B LLaMA2-chat models as the expert and amateur models, respectively. For ICD, we utilize the hallucination-injected finetuned LLaMA2-7B-chat model. We load this model, which was finetuned on selected samples from HaluEval (Li et al., ) and is provided by (^3^33[https://huggingface.co/HillZhang/untruthful_llama2_7b](https://huggingface.co/HillZhang/untruthful_llama2_7b)), as the amateur model, with LLaMA2-7B-chat and LLaMA2-13B-chat serving as expert counterparts.
For ITI, we utilize the honest-llama2-chat model with activation differences, implemented via pyvene (Wu et al., ), available at (^4^44[https://github.com/likenneth/honest_llama](https://github.com/likenneth/honest_llama)), along with the open-source models available at (^5^55[https://huggingface.co/collections/jujipotle](https://huggingface.co/collections/jujipotle)).
For TruthX, we download its LLaMA2-7B-chat version from (^6^66[https://github.com/ictnlp/TruthX](https://github.com/ictnlp/TruthX)) and deploy it following the methodology outlined by  Zhang et al. ().
During inference, we align all factuality enhancement methods with the baseline settings, including temperature=0.9, top-$p$=0.95, and others.

## Appendix B Context Prompts in KE Task

### B.1 Template for MeLLo

In Section 4, we use MeLLo (Zhong et al., ) to conduct our experiments for context-faithfulness, with the results presented in Table 3.
MeLLo first decomposes a multi-hop question into subquestions during LLMs inference, and then prompts the LLMs to provide tentative answers to these subquestions. Next, it self-checks their compatibility with edited facts by retrieving edit demonstrations from the knowledge base, thereby maintaining or adjusting them accordingly.
Table 5 provides the prompt templates used for MeLLo.

Table: Table 5: A step-by-step illustration of MeLLo solving one simplified example. Blue parts are generated by the language model, and orange parts are facts retrieved by the retriever.

### B.2 Context Demonstrations Used in ICE

We provide the context demonstrations used in our ICE experiments (Table 2), which were omitted in Figure 2 for brevity.
Here, the green text represents the expected context-faithful output.

## Appendix C More Output Cases of Factuality-Enhanced LLMs

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - In algorithm 1, how did you obtain S_new and S_para? Are they both contained in the MQUAKE-STUBBORN dataset?
soundness: 3
strengths: - The findings presented in the paper are interesting and insightful, bringing the trade-off of faithfulness and factuality to the community’s awareness.
- The analysis conducted in this paper is rich and sufficient.
- The paper is overall well written.
summary: The paper discusses the trade-offs between enhancing factual accuracy and maintaining context-faithfulness in large language models (LLMs). The authors argue that current methods aimed at reducing hallucinations and improving factual accuracy in LLMs tend to undermine the models’ ability to remain faithful to the given context. Through a series of evaluations, they find that while factual accuracy might show inconsistent improvement, context-faithfulness suffers significantly, with notable declines observed—up to 69.7%. The analysis of hidden states and logit distributions highlights the limitations of the current approaches, emphasizing the complex balance required between factual knowledge and context adherence. The authors suggest that future research should focus more on minimizing the adverse effects on context-faithfulness when developing methods for enhancing factuality in LLMs.
weaknesses: - If I understand it correctly, Algorithm 1 assumes there is one token difference between S_new and S_para. Is there any rationale behind this assumption? Additionally, if this is true, I wonder if the findings in section 5 generalize to settings where S_new and S_para different by more than 1 token?

- I do not find other critical flaws in the paper other than some typos. For example, a closing parenthesis is missing from equation 2. In Figure 3, the legend of the green bar should be “Factuality”. There are some other few types like repeating words across the paper but these do not hinder the understanding of the paper.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness above
soundness: 3
strengths: 1. The papers provides valuable insight on the dual trade-off between factuality enhancement and context-faithfulness, recommendation future research to reduce the sacrifice of context-faithfulness.

 2. The authors employ a comprehensive experimental setup, testing various methods such as contrastive decoding and representation editing.

3. The presentation of the paper is well-organized and clear, facilitating easy comprehension of experimental findings and details.
summary: This paper aims to analyze the trade-off between context-faithfulness and factuality enhancement. They first revisist factuality enhancement method and assess their effectiveness. Then, they perform experiments to show that these methods might have a severe decline in context-faithfulness. They finally provide analysis on hidden states and logic distributions to explain the limitation of current methods.
weaknesses: 1. In Algorithm 1, the None checks for P_new and P_para mean that once these variables are assigned a value, the algorithm stops capturing additional tokens. This setup results in identifying only the first differing token between S_new and S_para, while ignoring any subsequent distinctions. Is this interpretation correct? If so, this approach seems limited because of ignoring the subsequent tokens. Please correct me if I make it wrong.

2. While the paper effectively highlights the issue of reduced context-faithfulness with factuality enhancement, it could benefit from a forward-looking research agenda or recommendations for addressing this challenge. Practical suggestions or potential strategies for mitigating these trade-offs would strengthen the contribution and provide guidance for future work

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: please see above
soundness: 2
strengths: + exploring the interplay between parametric knowledge and context faithfulness is promising
+ the experiments are comprehensive
summary: This work proposes to examine the effect of "factuality enhancement" approaches that better exploit LLM parametric knowledge, specifically representation editing and contrastive decoding approaches. Experiments find that they could lead to better factuality and worse context faithfulness, while analysis of model internals provide some indicators.
weaknesses: - I'm not sure if I could agree with the framing of "factuality enhancement" methods in this work. In the abstract, the authors claim that "factuality enhancement can lead LLMs to become overly confident in their parametric knowledge" (lines 17-18). However, retrieval-augmented generation is also a way of "factuality enhancement", and that certainly steers models away from parametric knowledge to varying extents. Although the broad term of "factuality enhancement" is employed, the investigated approaches are just a narrow subset.

- Following up on the previous point, the included approaches "representation editing" and "contrastive decoding" are not representing broad walks of "factuality enhancement approaches", but rather approaches that specifically steer the models to better tap into parametric knowledge. As models better rely on internals, they are naturally less faithful to external contexts, hence the conclusion that "models become less context-faithful after you apply these approaches" become trivial. Essentially: if you use approaches that emphasize model internals, they will certainly be less reliant on external context, which is not a surprising/consequential finding.

- Aside from representation editing and contrastive decoding, some earlier prompting-based approaches also claim to better elicit and employ parametric knowledge to enhance factuality, such as [1-2]. I'm not sure if they should be included as baselines, but I wonder if they might change the conclusion as to context faithfulness.

- Section 2.1 is somewhat trivial and don't need to appear in the main paper.

- Again, the authors claim that "Current methods for enhancing factuality do not modify the underlying structure or parameters of LLMs." (lines 188-189) Why would factuality SFT [3] and factuality DPO [4] approaches not count as "current methods for enhancing factuality"? Readers might say that this is twisting the definition of "methods for enhancing factuality" to fit a pre-defined conclusion.

- typo on line 499: "Knowledge Conficts"

- Let's say if indeed "factuality enhancement" and "context-faithfulness" are two conflicting objectives. I wonder if this might be fixed by inference-time approaches that enhance context-faithfulness such as [5]. Essentially, take an LLM, do some "factuality enhancement" which might harm context faithfulness, and then apply things like [5] to patch it and get the best of both worlds.

- The model internal analysis part is great and potentially useful. I wonder if the authors could offer some crisp conclusions/takeaways in addition to presenting some numbers and figures. In the abstract it says "To explain these declines, we analyze the hidden states and logit distributions for the tokens representing new knowledge and parametric knowledge respectively, highlighting the limitations of current approaches." In the conclusion it says "By examining the logits distribution and hidden states, we gained insights into the underlying causes of this decline", which are both very vague. What are "the limitations of current approaches"? What are "the underlying causes of this decline"? Most importantly, I wonder if the authors have some actionable next steps or potential fixes as validated by this internal analysis part.

- Perhaps there is a way to tone down the definition of "factuality enhancement" and make the claims/findings more rigorous or better scoped. This would be essential in making a decision about this submission.

[1] Liu, Jiacheng, et al. "Generated Knowledge Prompting for Commonsense Reasoning." ACL 2022.

[2] Sun, Zhiqing, et al. "Recitation-Augmented Language Models." ICLR 2023.

[3] Hron, Jiri, et al. "Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability." COLM 2024.

[4] Tian, Katherine, et al. "Fine-Tuning Language Models for Factuality." The Twelfth International Conference on Learning Representations. ICLR 2024.

[5] Shi, Weijia, et al. "Trusting Your Evidence: Hallucinate Less with Context-aware Decoding." NAACL 2024.

Meta-review: This paper examines the trade-off between enhancing factual accuracy and maintaining context-faithfulness in LLMs. Through experiments and analysis of model internals, the authors find that methods like representation editing and contrastive decoding improve factuality but significantly harm context-faithfulness. The study highlights the limitations of current approaches and calls for research to mitigate these adverse effects while improving factual accuracy.

In general, all the reviewers appreciated the paper and also engaged in the discussion. I also appreciate the authors for their detailed responses, which led to increased scores. I would like the authors to address the following points in the revised manuscript:

1. One of the reviewers raised serious concerns about the "factuality enhancement" and the broader score in the paper. The authors attempted to address this with elaborative comments, some of which might not be directly relevant to the given context. I suggest adding the responses appropriately in the paper without deviating from the main claims 

2. The other reviewer suggested that practical suggestions or potential strategies for mitigating these trade-offs would strengthen the contribution and provide guidance for future work. I think this is really crucial given that the paper presents a critical aspect of the balance between factual accuracy and faithfulness. 

3. Please address the typos and writing issues pointed out by the reviewers.

Decision: Accept (Poster)