FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"

## Abstract

Abstract Ensuring faithfulness to context in large language models (LLMs) and retrieval-augmented generation (RAG) systems is crucial for reliable deployment in real-world applications, as incorrect or unsupported information can erode user trust. Despite advancements on standard benchmarks, faithfulness hallucination—where models generate responses misaligned with the provided context—remains a significant challenge. In this work, we introduce FaithEval, a novel and comprehensive benchmark tailored to evaluate the faithfulness of LLMs in contextual scenarios across three diverse tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may surface incomplete, contradictory, or fabricated information. FaithEval comprises 4.9K high-quality problems in total, validated through a rigorous four-stage context construction and validation framework, employing both LLM-based auto-evaluation and human validation. Our extensive study across a wide range of open-source and proprietary models reveals that even state-of-the-art models often struggle to remain faithful to the given context, and that larger models do not necessarily exhibit improved faithfulness.
Project is available at: https://github.com/SalesforceAIResearch/FaithEval .

## 1 Introduction

The rapid development of large language models (LLMs) has significantly advanced natural language understanding and generation tasks, enabling systems to produce fluent and coherent responses across a variety of applications (Bubeck et al., ; Zhao et al., ; Wu et al., ). The capabilities of these models have been further enhanced by integrating external information from the Internet or knowledge sources using a popular approach of Retrieval-Augmented Generation (RAG) (Lewis et al., ; Zhao et al., ). In this paradigm, generated outputs are enhanced by retrieving and encoding relevant information as context to the model. While RAG facilitates the integration of additional knowledge, hallucination—where models generate unsupported or ungrounded content—remains a critical challenge (Nguyen et al., ).

Figure: Figure 1: Performance summary on FaithEval Benchmark. Each bar shows the combined accuracy (normalized) for the best model from each organization across three tasks: Counterfactual, Inconsistent, and Unanswerable (Sec 3.1). Different colors in each bar represent the accuracy for each task.
Refer to caption: x1.png

Hallucination in LLMs can be generally categorized into two types: *factual hallucination*, where generated content deviates from established world knowledge, and *faithfulness hallucination*, where the generated response is inconsistent with the provided context (Huang et al., ). While factuality has received extensive attention, with numerous benchmarks designed to evaluate correctness against common sense or world knowledge (Lee et al., ; Min et al., ; Chern et al., ; Wei et al., ), a fine-grained
and holistic evaluation of faithfulness on noisy contexts remains underexplored, particularly when the context contradicts commonly accepted facts. Maintaining faithfulness to the context is especially important for various personalized applications and can be critical in high-stakes domains such as healthcare, finance and law, where inaccurate or ungrounded responses can erode user trust and lead to severe consequences (Bommarito & Katz, ; Pal et al., ).

One of the key challenges in addressing faithfulness hallucination in RAG stems from the retrieval process, where the wealth of documents on the Internet varies in credibility.
This complexity is further compounded when long retrieved content includes multiple relevant paragraphs that omit key details, present conflicting evidence, or propagate counterfactual claims. Existing hallucination evaluation benchmarks fall short in providing fine-grained assessments of how well models align their responses with the context. They often do not disentangle factuality from faithfulness (Li et al., ; Chen et al., ) or capture the full range of contextual nuances (Lin et al., ; Yin et al., ). Moreover, current hallucination detection solutions focus on identifying hallucinations in model outputs (Liu et al., ; Li et al., ; Hu et al., ), which is orthogonal to the task of understanding the impact of contexts on faithfulness hallucination.

In this work, we introduce FaithEval, a comprehensive benchmark specifically designed to evaluate the contextual faithfulness of LLMs across *three* diverse tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may surface incomplete, contradictory, or fabricated information (Figure 2). FaithEval includes a total of 4.9K high-quality samples, constructed using a rigorous four-stage framework with multi-turn LLM-based context verification and human validation. We conduct a holistic evaluation on 18 representative proprietary and open-sourced models, revealing that faithfulness remains challenging, even for the most competitive LLMs, despite their strong performance on standard benchmarks. Figure 1 summarizes the performance of representative models from each organization, with each bar representing performance on the individual tasks. To our knowledge, FaithEval is the *first fine-grained and comprehensive benchmark* specifically targeting contextual faithfulness hallucination, contributing to the broader effort toward developing reliable next-generation foundation models.

Our key contributions are summarized as follows:

- •
We introduce FaithEval, a novel and comprehensive benchmark dedicated to evaluating contextual faithfulness in LLMs across three diverse tasks: unanswerable, inconsistent, and counterfactual contexts.
- •
We develop a scalable four-stage framework for context construction and validation, which incorporates multi-turn LLM-based validation and human annotation to ensure high-quality contextual QA pairs.
- •
We perform an extensive and in-depth study on a wide range of competitive open-source and proprietary models. We highlight that faithfulness remains a significant challenge for recent LLMs and that allegedly larger models, such as GPT-4o and Llama-3-70B-Instruct, do not necessarily lead to improved faithfulness.

## 2 Related Works

Contextual LLM and retrieval-augmented generation.
As the demand for contextual LLMs continues to grow, retrieval-augmented generation (RAG) systems offer a promising solution by integrating external knowledge retrieval with LLMs (Lewis et al., ; Sarto et al., ; Ramos et al., ; Huang et al., ; Zhao et al., ).
In RAG, model responses are grounded using knowledge sourced from private or open-access data repositories. A typical RAG system operates through a close interaction between the retriever and a generator. The retriever (Li et al., ; Meng et al., ; Chen et al., ) identifies relevant documents from the source, and this retrieved information is supplied to the generator (*e.g.,* a language model) to produce grounded outputs (Lewis et al., ; Izacard & Grave, ; Borgeaud et al., ; Ke et al., ). Recent works develop more sophisticate RAG frameworks to improve answer reliability (Asai et al., ; Li et al., ; Xu et al., ). The increased context sizes in LLMs have improved their ability to handle longer text sequences, allowing them to handle complex tasks requiring extensive background knowledge (Gao et al., ; Song et al., ; Shi et al., ). These improvements are particularly beneficial for long-form question answering (Joshi et al., ; Kwiatkowski et al., ; Li et al., ). However, the variation in source quality can exacerbate challenges to maintaining faithfulness in LLMs, especially in longer contexts retrieved from the Internet.

Hallucination and faithfulness evaluation.
Hallucination in LLMs refers to the generation of ungrounded content, either from the provided context or established world knowledge. The former is typically described as factuality hallucination, while the latter is known as faithfulness hallucination, which highlights the discrepancy between the model’s output and the context (Huang et al., ). While there is rich literature on factuality evaluation and benchmarks with and without contexts (Lee et al., ; Min et al., ; Chern et al., ; Wei et al., ), faithfulness has mostly been explored for summarization (Laban et al., ; Jia et al., ) and natural language explanations (Atanasova et al., ; Siegel et al., ). Another line of research focuses on hallucination detection (Liu et al., ; Li et al., ; Hu et al., ), which aims to detect hallucinated outputs. The task focuses on the model output instead of the context (input). Additional efforts have been made to create QA benchmarks based on common misconceptions (Lin et al., ) or questions that are unanswerable by nature (Yin et al., ). However, none of these datasets are contextual. In contrast, each question in FaithEval is accompanied by a multi-paragraph context, mimicking RAG scenarios with long and noisy contexts.

Adversarial context generation.
Generating challenging or adversarial contexts for language models has been explored in various scenarios. One line of research focuses on context modification. Shi et al. () propose a template-based framework that adds irrelevant facts to the context and studies the effectiveness of different prompting techniques. Yu et al. () leverage LLMs to perturb original evidence, potentially altering the answers, while Manakul et al. () utilize LLMs to generate purely synthetic contexts that support given statements. To study the impact of knowledge conflicts, Xie et al. () use LLMs to create adversarial contexts that conflict with the models’ internal knowledge, improving coherence compared to previous word-level editing methods (Longpre et al., ; Chen et al., ; Zhou et al., ). Another research direction involves modifying the question. Ramakrishna et al. () generate invalid (unanswerable) questions, while Huang et al. () build a small-scale dataset with 209 questions containing adversarial facts or incorrect information based on diverse templates. In contrast, we introduce the first fine-grained, larger-scale (4.9K) high-quality contextual QA benchmark featuring multi-paragraph coherent contexts across three diverse tasks.

## 3 FaithEval Benchmark

Figure: Figure 2: Demonstration of each task in FaithEval. Left: in Unanswerable Context, the context does not contain the answer to the question. Middle: in Inconsistent Context, multiple answers are supported by different documents. Right: in Counterfactual Context, the context contains counterfactual statements that contradict common sense or world knowledge.
Refer to caption: x2.png

### 3.1 Task Overview

To systematically evaluate the contextual faithfulness of LLMs, FaithEval contains three diverse tasks including unanswerable context, inconsistent context, and counterfactual context. Each sample $(\mathbf{c},q,a)$ consists of a question $q$, and a long context passage made up of one or more documents $\mathbf{c}=(d_{1},...,d_{n})$, and a groundtruth answer $a$. The model is expected to answer the question leveraging the information in the provided context. An overview of each task is presented in Figure 2. Next, we illustrate the construction of each task in detail.

#### Unanswerable Context.

An unanswerable context arises when the context includes relevant details but lacks the information needed to answer the question. In FaithEval, answerability is determined *solely* by the context, regardless of whether the question itself is unanswerable.
For instance, in the example in Figure 2 (Left), the context provides the proportion of both types of commuters in 2015. However, the question “Which group of commuters in Dallas in 2009 is larger: carpooling or transit?” is unanswerable, as the context lacks specific data from 2009.
To create such task, we modify the context from a collection of 10 contextual QA datasets, covering a wide range of domains (see Source datasets below). For each sample, we prompt an LLM to modify the original context so that it no longer contains the supporting evidence for the ground truth answer. Additional sentences may be woven into the new context to maintain coherence.
The full prompt is shown in Figure 17. This process resulted in a total of 2.4k contextual QA pairs. To verify the quality of the modified contexts, we achieved over 98% agreement with professional human annotators (Sec 3.2).

#### Inconsistent Context.

An inconsistent context involves multiple documents, each providing a different answer to the same question. This simulates noisy retrieval scenarios,
where documents from sources with varying levels of credibility are retrieved. For instance, as shown in Figure 2 (Middle), the context presents conflicting information about the tiger’s name in the novel *Life of Pi*. A faithful model should be able to identify such inconsistencies, especially when instructed to do so. To create this task, we modify contexts from the same collection of contextual QA datasets used in the Unanswerable Context task. For each sample, the LLM is provided with a context passage, a question, and an original answer, which is supported by the context. The goal is to modify the context so that it introduces fabricated supporting evidence for a new, conflicting answer.
The detailed prompt is shown in Figure 18. Since this task is more challenging, we curated a collection of 1.5k high-quality contextual QA pairs after filtering through professional human annotators.

#### Counterfactual Context.

A counterfactual context contains statements that contradict with common sense or widely accepted facts, such as “water freeze at 100 degrees Celsius”, “wood is magnetic”, or “carbon dioxide is the most abundant greenhouse gas in the atmosphere”.
Unlike the other two tasks, the questions in this task are required to be relevant to such well-known facts. We curate this task based on ARC-Challenge (Clark et al., ), a QA dataset covering grade-school level, multiple-choice science questions. Since the original dataset does not include context, we prompt an LLM to generate a long, multi-paragraph context that seamlessly provides fabricated supporting evidence for a counterfactual answer.
The detailed prompt is shown in Figure 19. This process resulted in a total of 1k contextual QA pairs, each with three to five options. Due to the multiple-choice nature, we can use keyword matching to verify the quality of the synthetic contexts (Appendix B).

#### Source datasets.

We curate new contexts based on a diverse collection of contextual QA datasets, including SQuAD (Rajpurkar et al., ), NewsQA (Trischler et al., ), TriviaQA (Joshi et al., ), NaturalQuestions (Kwiatkowski et al., ), SearchQA (Dunn et al., ), HotpotQA (Yang et al., ), BioASQ (Tsatsaronis et al., ), DROP (Dua et al., ), RACE (Lai et al., ), TextbookQA (Kembhavi et al., ). We adopt the test splits from  Fisch et al. (). Due to variations in human annotations, the final collection consists of 150 samples per dataset for the Inconsistent Context task and around 240 samples per dataset for the Unanswerable Context task.

### 3.2 Task Construction and Validation Framework

Task construction. An overview of our task construction and validation framework is shown in Figure 3. Given a source QA sample and an original context (optional), we prompt an LLM to generate both a new context and a new answer for the Counterfactual and Inconsistent tasks, or only a new context (that supports no answer) for the Unanswerable task. To make the tasks challenging, the new context should be coherent and contain minimal modifications if the original context is provided. In addition, multiple paragraphs not directly related to the answer are included, serving as distractors. The new context is generated with detailed justifications explaining how it satisfies the task criterion. We construct the prompt for each sample by combining the original question, the new context, and task-specific instructions (Sec 3.3). In particular, an inconsistent context is created by concatenating the new context with the original context, each supporting a different answer.

Figure: Figure 3: Illustration of task construction and validation framework. (1) Context Generation: given a source QA dataset, we prompt an LLM to generate a new context based on a question, the original answer, and optionally the original context. (2) Task Construction: we construct the prompt for each sample by combining the original question, the new context, and task-specific instructions. (3) Auto Eval by LLM Judge: we validate the quality of the new context by checking if and only-if the new answer is supported by the new context. (4) Human Annotation: we further filter out invalid contextual QA pairs based on the majority vote results from professional annotators.
Refer to caption: x3.png

Auto validation and human annotation. We validate the quality of the new context by using a separate LLM judge to verify whether the new answer is valid given the context ("if" condition) and whether the context does not support alternative answers ("only-if" condition). For example, the new context in Figure 2 (Right) should not mention wood is buoyant. Samples that fail to meet both conditions are filtered out. Next, we perform meticulous human annotation. Depending on the task’s validation difficulty, we employ different strategies. As the Inconsistent Context task is challenging to validate, we rely on full human annotation. Three Mechanical Turk (Crowston, ) workers judge whether each contextual QA pair meets the "if" and "only-if" conditions, with final inclusion determined by majority agreement. This yields 1.5K samples. For the Unanswerable Context task, which is easier to validate, we use a similar majority-vote approach, achieving over 98% agreement among human annotators. This yields 2.4K samples. For the Counterfactual Context task, since the answer options are provided within the context, we validate using a string-based matching method, where the context passes if all words from the answer appear in the context. This results in 1K samples. After filtering, the FaithEval benchmark contains a total of 4.9K high-quality contextual QA pairs. More details are included in Appendix A.

### 3.3 Evaluation

Models. We evaluate a wide range of competitive open-sourced and proprietary language models with different scales, including the most recent releases up to Sep 10, 2024.
Our initial experiments suggest that instruction-tuned (chat) models significantly outperform base models. Therefore, we consider 18 competitive chat models, including Phi-3-mini-128k-instruct (3.8B), Phi-3-medium-128k-instruct (14B), Phi-3.5-mini-instruct (3.8B) (Abdin et al., ), LLaMA-3-8B-Instruct, LLaMA-3.1-8B-Instruct, LLaMA-3-70B-Instruct, LLaMA-3.1-70B-Instruct (Llama, ), Mistral-7B-Instruct-v0.3, Mistral-Nemo-Instruct-2407 (12B) (Jiang et al., ), Gemma-2-9B-it, and Gemma-2-27B-it (Team, ). For proprietary models, we consider Open AI’s GPT-3.5 Turbo, GPT-4o-mini, GPT-4o, GPT-4 Turbo, Cohere’s Command R (35B), Command R+ (104B), and Anthropic’s Claude 3.5 Sonnet.

Default Evaluation Scheme. For all tasks, we append the following prompt to each question: You are an expert in retrieval-based question answering. Please respond with the exact answer, using only the information provided in the context. For the Unanswerable Context task, we append an additional instruction: If there is no information available from the context, the answer should be “unknown”. Similarly, for the Inconsistent Context task, the instruction is: If there is conflicting information or multiple answers in the context, the answer should be “conflict”. Our primary evaluation metric across all tasks is accuracy (ACC), where a model’s response is considered correct if it mentions the ground truth answer. All models are evaluated using their default configurations with deterministic decoding (temperature = 0). We report both strict-matching (S) ACC, which considers only a single ground truth answer (e.g., “unknown”), and non-strict matching (N) ACC, which allows a broader range of semantically similar phrases. A detailed list of valid phrases is provided in Appendix A.

#### Alternative Evaluation Schemes.

We study alternative evaluation strategies in Section 5. Specifically, we examine non-deterministic decoding with temperature scaling (t = 0.3, top-$p$ = 0.9). Additionally, we investigate the impact of chain-of-thought (CoT) prompting (Wei et al., ; Kojima et al., ) using the following instruction: Given the context, first provide a brief answer to the question. Then, explain your reasoning step by step, detailing how you arrived at the answer.

## 4 Main Results

### 4.1 Unanswerable Context

Abstaining is challenging, even when explicitly instructed.
The results of the Unanswerable Context task are summarized in Figure 4, ranked by performance on the original context. Proprietary model names are highlighted in orange.
We highlight the following key observations: (1) Modern LLMs experience significant performance degradation in this task. Across all chat models, the performance gap ranges from 13.6% to 68.4%. (2) High performance on the original context does not correlate with high performance on the unanswerable context. For example, while Phi-3-medium-128k-instruct achieves 76.8% accuracy on the original context, closely approaching the SoTA (80.1%), it struggles to abstain from answering in the unanswerable context, with an accuracy of only 7.4%. (3) Larger model sizes are more advantageous within the same model family. For instance, compared to the 7B model, Llama-3.1-70B-instruct improves performance on the Unanswerable Context task by 10.3%. Similar trends hold for the Gemma-2 and Llama-3 model families.

Figure: Figure 4: Model performance comparison on the Unanswerable Context task, where no evidence supports the answer. Columns are sorted by performance on the Original task (original context). Proprietary model names are highlighted in orange.
Refer to caption: extracted/5911559/img/original_vs_remove_performance.png

### 4.2 Inconsistent Context

Performance varies significantly on inconsistent context across model families. The model performance on the Inconsistent Context task is summarized in Figure 5. We have the following key observations: (1) Performance varies substantially across different model families. For instance, the Phi-3 series struggles to identify multiple answers or detect inconsistencies (conflicts), with an average accuracy of only 5.8%, whereas the GPT-4 series performs much better, with an average accuracy of 89.35%. (2) Open-source models lag behind proprietary models. Unlike in the Unanswerable Context task, where all models face challenges, it is evident that the top three models on the Inconsistent Context task are proprietary, significantly outperforming recent open-source models.

Figure: Figure 5: Model performance comparison on the Inconsistent Context task. Columns are sorted by the performance on the original task. Proprietary models are colored in orange.
Refer to caption: extracted/5911559/img/model_performance_ic.png

### 4.3 Counterfactual Context

Faithfulness remains a limitation for contextual LLMs. The results on the Counterfactual Context task are shown in Figure 6. The blue bars represent model performance under the closed-book QA setting, where no context is provided. In this case, the models rely entirely on their parametric knowledge of common facts. We observe that nearly half of the models achieve over 90% accuracy, with GPT-4o nearing perfect performance at 96.3%. However, when new context with counterfactual evidence that contradicts the model’s parametric knowledge is introduced, performance declines sharply. For example, GPT-4o achieves only 47.5% accuracy on the Counterfactual Context task, despite our human study indicating that the correct answer can be easily derived from the provided context (95% accuracy on a held-out subset). This highlights a significant gap in faithfulness—the ability to generate outputs that align with the provided context—between current state-of-the-art models and human-level performance.

Figure: Figure 6: Model performance comparison on Counterfactual Context, which contains evidence supporting a counterfactual answer. Proprietary models are colored in orange.
Refer to caption: extracted/5911559/img/model_performance_cc.png

## 5 Discussions and Further Analysis

Figure: Figure 7: Performance decomposition on individual datasets for Unanswerable Context (top row) and Inconsistent Context (bottom row). Full results for all models can be seen in Appendix C.
Refer to caption: extracted/5911559/img/individual/Llama-3.1-8B-Instruct_results_no_text.png

#### Performance breakdown for individual datasets.

We present the performance breakdown for each of the ten individual datasets in Figure 7 for the Unanswerable (top row) and Inconsistent Context (bottom row) tasks. We include three representative smaller-scale models: LLama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9b-it. Full results for other models are provided in Appendix C. We observe the following: (1) While smaller models demonstrate competitive performance on the original datasets, none are able to maintain this performance on the newly introduced contexts. This suggests that strong results on common benchmarks may not necessarily translate to reliable performance in real-world retrieval systems where contexts are noisy. (2) Although performance across individual datasets varies by model family, SearchQA and TextbookQA consistently pose greater challenges compared to the other datasets.

Figure: Figure 8: Model performance comparison on the Original Context v.s. New Context for Inconsistent Context task. Proprietary models are colored in orange.
Refer to caption: extracted/5911559/img/original_vs_new.png

#### A closer look at Inconsistent Context.

Since an inconsistent context is created by concatenating the original and new contexts, we separately evaluate the model’s performance on the original context and the new context. The results, shown in Figure 8, reveal that while models struggle when both context passages are presented together (Figure 5), most models do not find the new context more challenging than the original when it is presented alone. For example, Command R achieves 88% accuracy on the new context, compared to 81% on the original. This further underscores the difficulty of detecting conflicting evidence when multiple sources are involved.

**Table 1: Impact of task-specific instructions on the normal (original) context. Having the additional task instruction degrades the performance on normal contexts consistently.**
| Task-spec. Inst. | BioASQ | DROP | HotpotQA | NQ | NewsQA | RACE | SQuAD | SearchQA | TextbookQA | TriviaQA | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Claude 3.5 Sonnet |  |  |  |  |  |  |  |  |  |  |  |
| ✗ (original prompt) | 0.90 | 0.94 | 0.89 | 0.86 | 0.81 | 0.84 | 0.93 | 0.97 | 0.97 | 0.86 | 0.90 |
| ✓ (+conflict prompt) | 0.85 $\downarrow$ | 0.84$\downarrow$ | 0.86$\downarrow$ | 0.78$\downarrow$ | 0.75 $\downarrow$ | 0.77$\downarrow$ | 0.95 | 0.91$\downarrow$ | 0.90$\downarrow$ | 0.86 | 0.85$\downarrow$ |
| GPT-4o |  |  |  |  |  |  |  |  |  |  |  |
| ✗ (original prompt) | 0.84 | 0.81 | 0.89 | 0.80 | 0.78 | 0.79 | 0.95 | 0.90 | 0.93 | 0.87 | 0.85 |
| ✓ (+conflict prompt) | 0.77$\downarrow$ | 0.83 | 0.85$\downarrow$ | 0.79$\downarrow$ | 0.73$\downarrow$ | 0.81 | 0.93$\downarrow$ | 0.87$\downarrow$ | 0.91$\downarrow$ | 0.86$\downarrow$ | 0.83$\downarrow$ |

Sycophancy with task-specific instructions. While the additional instructions used in the Unanswerable and Inconsistent Context tasks (Sec 3.3) improve the model’s awareness of such scenarios, they can also introduce unintended effects when the context is normal (*i.e.*, answerable and consistent). This can lead to what is known as sycophantic behavior (Perez et al., ; Wei et al., ), where models adjust their responses to align with the user’s expectations, even when those expectations are objectively incorrect. We examine this phenomenon in two top-performing models for the Inconsistent Context task, GPT-4o and Claude 3.5 Sonnet. Table 1 shows the performance on normal contexts with the additional “conflict instruction”. We observe a consistent performance drop for both models, with Claude 3.5 experiencing a 5% decrease in average accuracy. This further highlights the challenge of maintaining faithfulness across both normal and noisy contexts.

Figure: (a) Direct Ans vs. CoT on Unanswerable Context
Refer to caption: extracted/5911559/img/cot_remove_performance.png

#### Does chain-of-thought prompting improve faithfulness?

Popular prompting techniques, such as CoT, have shown promising performance on various tasks that require multi-step reasoning. We adopt the prompt format in Section 3.3 and summarize the results in Figure 9. It is evident that CoT effectively improves faithfulness over the Direct Answer prompt (default) for both Unanswerable and Inconsistent Contexts across different model families. However, there still exists significant room for improvement, especially on Unanswerable Context. For instance, the leading model achieves only 71.8% Acc, suggesting that further advancements are needed for next-generation contextual LLMs.

Figure: Figure 10: Impact of strict vs. non-strict matching on Unanswerable Context. Non-strict matching allows for a wider range of phrases that express the idea of “unknown”. The comparison on Inconsistent Context is shown in Appendix C.
Refer to caption: extracted/5911559/img/strict_vs_non_strict.png

#### Strict vs. non-strict matching.

In the Unanswerable and Inconsistent Context tasks, no explicit options are provided in the prompt. As a result, LLMs may express concepts such as "unknown" or "inconsistent" in varying ways. To assess the impact of allowing alternative valid expressions, we compare the performance of strict and non-strict matching. Strict matching only accepts the exact phrases "unknown" or "conflict" as specified in the prompt, while non-strict matching permits a broader range of expressions that convey similar ideas (see Appendix A for the full list of valid expressions). We observe that performance remains stable across most models. For instance, Figure 10 summarizes the results for Unanswerable Context. For competitive models such as gpt-4o and Claude 3.5, the gaps are less than 1%. Full results can be found in Table 3 and Table 4 in Appendix C.

Impact of decoding strategies. By default, we adopt greedy decoding. We also investigate a popular sampling-based decoding scheme with a temperature of 0.3 and top-p of 0.9. The results are shown in Figure 11 based on Counterfactual Context. Similar observations also hold for Unanswerable and Inconsistent Context. We observe that sampling-based decoding marginally improves the performance over greedy decoding across all models. However, the significant gap between the original and counterfactual contexts cannot be mitigated with temperature scaling.

Figure: Figure 11: The impact of decoding strategy. The plot displays greedy (non-sampling) vs. sampling-based decoding (t=0.3, top-p=0.9) on Counterfactual Context.
Refer to caption: extracted/5911559/img/model_performance_temp.png

## 6 Conclusion

In this work, we propose FaithEval, a novel and challenging benchmark designed to assess the faithfulness of contextual LLMs. FaithEval comprises 4.9K high-quality contextual problems spanning multiple domains and includes three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. To build this benchmark, we propose a scalable multi-stage context construction and validation framework, incorporating both automated evaluation by an LLM judge and human validation. This approach enables the creation of multi-paragraph coherent contexts satisfying diverse criteria. We provide a timely and in-depth study on a wide range of open-source and proprietary models, revealing that even the most competitive LLMs often struggle to remain faithful to the contexts, despite excelling on standard benchmarks. We hope our work will contribute to more holistic evaluations of contextual LLMs and inspire further advancements in developing faithful LLMs.

## Appendix A Additional Experiment Details

#### Context Generation Model.

By default, we use the latest GPT-4o (gpt-4o-2024-05-13) as the context generator. In our preliminary studies, we evaluated various alternatives, including GPT-4o-mini, GPT-4-turbo, and LLaMA-3.1-70B. GPT-4o demonstrated superior performance in terms of context coherence, validity, and complexity.

#### Context Evaluation Model.

We use gpt-4o-mini as the judge model for context verification. We selected gpt-4o-mini for its faster inference speed, lower cost, and high judgment quality, with our preliminary study showing an agreement rate of over 95% when compared to GPT-4o as the judge.

#### Valid phrases for non-strict matching.

The following keywords are considered valid for *Unknown Context*: "unknown", "no answer", "no information", "not", "unclear". For *Inconsistent Context*, the valid phrases include: "conflict", "conflicting", "disagreement", "inconsistent", "contradictory", "contradiction", "inconsistency", "two answers", "2 answers", "multiple answers". These keywords were selected based on an analysis of output patterns from open-source and proprietary models.

#### Model sizes.

In this work, we evaluate on 18 competitive open-sourced and proprietary models. A summary of model sizes from different model families are shown in Table 2.

**Table 2: Model sizes across different model families.**
| Model Name | Model Size |
| --- | --- |
| Phi-3 Family (Abdin et al., ) |  |
| Phi-3-mini-128k-instruct | 3.8B |
| Phi-3-medium-128k-instruct | 14B |
| Phi-3.5-mini-instruct | 3.8B |
| LLaMA-3 Family (Llama, ) |  |
| LLaMA-3-8B-instruct | 8B |
| LLaMA-3.1-8B-instruct | 8B |
| LLaMA-3-70B-instruct | 70B |
| LLaMA-3.1-70B-instruct | 70B |
| Mistral Family (Jiang et al., ) |  |
| Mistral-7B-instruct-v0.3 | 7B |
| Mistral-Nemo-instruct-2407 | 12B |
| Gemma-2 Family (Team, ) |  |
| Gemma-2-9B-it | 9B |
| Gemma-2-27B-it | 27B |
| OpenAI |  |
| GPT-3.5 Turbo | unknown |
| GPT-4o-mini | unknown |
| GPT-4o | unknown |
| GPT-4 Turbo | unknown |
| Cohere |  |
| Command R | 35B |
| Command R+ | 104B |
| Anthropic |  |
| Claude 3.5 Sonnet | unknown |

## Appendix B Verification for Counterfactual Context

For the Counterfactual Context task, since the answer options are provided within the context, we can validate the new context using a simple keyword-based matching method, where a context passes if the answer phrase exists in the context. This results in a pass rate of 68.9%, where the new context clearly contains the new answer. The results on this filtered subset are shown in Figure 12. We can see that the same trend still holds as shown in Section 4: a significant gap remains between the performance on the original task (with no context) and the new task with counterfactual contexts, across the majority of instruction-tuned models.

Figure: Figure 12: Model performance comparison on the clean subset of Counterfactual Context. The same trend still holds as in Section 4.
Refer to caption: extracted/5911559/img/model_performance_cc.png

## Appendix C Full Experiment Results

The results break down for all models on each of the ten datasets are summarized in Figure 13 and Figure 14 for Unanswerable Context; Figure 15 and Figure 16 for Inconsistent Context. The detailed results for strict vs. non-strict matching can be found in Table 3 and Table 4.

Figure: Figure 13: Performance decomposition on individual datasets for Unanswerable Context (part I).
Refer to caption: extracted/5911559/img/app_individual/Llama-3.1-8B-Instruct_results.png

Figure: Figure 14: Performance decomposition on individual datasets for Unanswerable Context (part II).
Refer to caption: extracted/5911559/img/app_individual/gpt-4-turbo_results.png

Figure: Figure 15: Performance decomposition on individual datasets for Inconsistent Context (part I).
Refer to caption: extracted/5911559/img/app_individual/Llama-3.1-8B-Instruct_results_c.png

Figure: Figure 16: Performance decomposition on individual datasets for Inconsistent Context (part II).
Refer to caption: extracted/5911559/img/app_individual/gpt-4-turbo_results_c.png

**Table 3: Performance comparison on Unanswerable Context with strict (S) and non-strict (N) matching.**
| Model | BioASQ | DROP | HotpotQA | NQ | NewsQA | RACE | SQuAD | SearchQA | TextbookQA | TriviaQA | AVG |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N |
| Llama-3-70B-Instruct | 0.508 | 0.512 | 0.644 | 0.656 | 0.548 | 0.552 | 0.508 | 0.516 | 0.512 | 0.520 | 0.396 | 0.408 | 0.588 | 0.592 | 0.112 | 0.116 | 0.488 | 0.488 | 0.455 | 0.463 | 0.476 | 0.482 |
| Llama-3-8B-Instruct | 0.268 | 0.272 | 0.528 | 0.532 | 0.252 | 0.252 | 0.224 | 0.232 | 0.344 | 0.364 | 0.336 | 0.352 | 0.436 | 0.444 | 0.140 | 0.144 | 0.232 | 0.236 | 0.248 | 0.252 | 0.301 | 0.308 |
| Llama-3.1-70B-Instruct | 0.452 | 0.460 | 0.676 | 0.684 | 0.628 | 0.632 | 0.468 | 0.500 | 0.500 | 0.524 | 0.408 | 0.424 | 0.612 | 0.616 | 0.096 | 0.108 | 0.436 | 0.440 | 0.517 | 0.525 | 0.479 | 0.491 |
| Llama-3.1-8B-Instruct | 0.292 | 0.300 | 0.632 | 0.636 | 0.448 | 0.452 | 0.388 | 0.400 | 0.420 | 0.428 | 0.396 | 0.420 | 0.532 | 0.544 | 0.116 | 0.128 | 0.212 | 0.228 | 0.326 | 0.339 | 0.376 | 0.388 |
| Mistral-7B-Instruct-v0.3 | 0.304 | 0.360 | 0.476 | 0.528 | 0.400 | 0.460 | 0.336 | 0.468 | 0.252 | 0.484 | 0.256 | 0.412 | 0.392 | 0.516 | 0.132 | 0.184 | 0.116 | 0.232 | 0.236 | 0.306 | 0.290 | 0.395 |
| Mistral-Nemo-Instruct-2407 | 0.192 | 0.196 | 0.436 | 0.444 | 0.380 | 0.380 | 0.284 | 0.288 | 0.308 | 0.332 | 0.316 | 0.340 | 0.416 | 0.428 | 0.224 | 0.224 | 0.108 | 0.112 | 0.140 | 0.140 | 0.280 | 0.288 |
| Phi-3-medium-128k-instruct | 0.024 | 0.204 | 0.124 | 0.428 | 0.176 | 0.312 | 0.116 | 0.288 | 0.060 | 0.360 | 0.044 | 0.312 | 0.128 | 0.448 | 0.024 | 0.052 | 0.036 | 0.068 | 0.008 | 0.062 | 0.074 | 0.253 |
| Phi-3-mini-128k-instruct | 0.020 | 0.096 | 0.108 | 0.248 | 0.096 | 0.148 | 0.100 | 0.168 | 0.076 | 0.188 | 0.040 | 0.104 | 0.076 | 0.232 | 0.072 | 0.084 | 0.012 | 0.024 | 0.029 | 0.045 | 0.063 | 0.134 |
| Phi-3.5-mini-instruct | 0.076 | 0.140 | 0.292 | 0.348 | 0.176 | 0.244 | 0.188 | 0.260 | 0.116 | 0.264 | 0.100 | 0.152 | 0.220 | 0.316 | 0.064 | 0.080 | 0.032 | 0.044 | 0.050 | 0.116 | 0.131 | 0.196 |
| gemma-2-27b-it | 0.712 | 0.716 | 0.680 | 0.684 | 0.628 | 0.628 | 0.636 | 0.640 | 0.512 | 0.512 | 0.488 | 0.488 | 0.676 | 0.680 | 0.156 | 0.160 | 0.412 | 0.412 | 0.558 | 0.562 | 0.546 | 0.548 |
| gemma-2-9b-it | 0.628 | 0.632 | 0.660 | 0.664 | 0.588 | 0.588 | 0.604 | 0.612 | 0.492 | 0.500 | 0.444 | 0.448 | 0.664 | 0.668 | 0.152 | 0.152 | 0.356 | 0.356 | 0.442 | 0.450 | 0.503 | 0.507 |
| Command R | 0.448 | 0.452 | 0.436 | 0.440 | 0.460 | 0.464 | 0.380 | 0.396 | 0.396 | 0.412 | 0.344 | 0.352 | 0.496 | 0.508 | 0.236 | 0.236 | 0.436 | 0.436 | 0.380 | 0.393 | 0.401 | 0.409 |
| Command R+ | 0.368 | 0.376 | 0.548 | 0.552 | 0.424 | 0.424 | 0.372 | 0.372 | 0.440 | 0.456 | 0.352 | 0.356 | 0.588 | 0.596 | 0.192 | 0.196 | 0.312 | 0.316 | 0.364 | 0.364 | 0.396 | 0.401 |
| gpt-3.5-turbo | 0.056 | 0.072 | 0.132 | 0.156 | 0.288 | 0.292 | 0.216 | 0.228 | 0.152 | 0.204 | 0.108 | 0.160 | 0.212 | 0.236 | 0.084 | 0.084 | 0.072 | 0.076 | 0.017 | 0.033 | 0.134 | 0.154 |
| gpt-4-turbo | 0.432 | 0.468 | 0.684 | 0.692 | 0.660 | 0.660 | 0.508 | 0.524 | 0.480 | 0.520 | 0.488 | 0.532 | 0.636 | 0.640 | 0.164 | 0.164 | 0.320 | 0.336 | 0.310 | 0.326 | 0.468 | 0.486 |
| gpt-4o | 0.588 | 0.592 | 0.728 | 0.732 | 0.812 | 0.812 | 0.728 | 0.732 | 0.604 | 0.604 | 0.540 | 0.544 | 0.748 | 0.748 | 0.180 | 0.184 | 0.576 | 0.576 | 0.463 | 0.463 | 0.597 | 0.599 |
| gpt-4o-mini | 0.576 | 0.588 | 0.688 | 0.692 | 0.680 | 0.680 | 0.688 | 0.692 | 0.460 | 0.472 | 0.512 | 0.532 | 0.676 | 0.680 | 0.244 | 0.244 | 0.388 | 0.396 | 0.322 | 0.326 | 0.523 | 0.530 |
| Claude 3.5 Sonnet | 0.832 | 0.840 | 0.784 | 0.792 | 0.716 | 0.716 | 0.640 | 0.648 | 0.512 | 0.544 | 0.524 | 0.536 | 0.704 | 0.712 | 0.228 | 0.236 | 0.648 | 0.648 | 0.624 | 0.624 | 0.621 | 0.630 |

**Table 4: Performance comparison on Inconsistent Context with strict (S) and non-strict (N) matching.**
| Model | BioASQ | DROP | HotpotQA | NQ | NewsQA | RACE | SQuAD | SearchQA | TextbookQA | TriviaQA | AVG |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N | S | N |
| Llama-3-70B-Instruct | 0.433 | 0.433 | 0.347 | 0.347 | 0.460 | 0.460 | 0.593 | 0.593 | 0.480 | 0.480 | 0.453 | 0.453 | 0.780 | 0.780 | 0.100 | 0.100 | 0.293 | 0.293 | 0.347 | 0.347 | 0.429 | 0.429 |
| Llama-3-8B-Instruct | 0.313 | 0.313 | 0.307 | 0.307 | 0.387 | 0.387 | 0.260 | 0.260 | 0.280 | 0.280 | 0.187 | 0.187 | 0.273 | 0.273 | 0.073 | 0.073 | 0.113 | 0.113 | 0.173 | 0.173 | 0.237 | 0.237 |
| Llama-3.1-70B-Instruct | 0.627 | 0.627 | 0.587 | 0.587 | 0.680 | 0.680 | 0.773 | 0.773 | 0.593 | 0.593 | 0.407 | 0.407 | 0.847 | 0.847 | 0.293 | 0.293 | 0.367 | 0.367 | 0.460 | 0.460 | 0.563 | 0.563 |
| Llama-3.1-8B-Instruct | 0.407 | 0.407 | 0.500 | 0.500 | 0.533 | 0.533 | 0.587 | 0.587 | 0.333 | 0.333 | 0.333 | 0.333 | 0.473 | 0.473 | 0.200 | 0.200 | 0.087 | 0.087 | 0.267 | 0.267 | 0.372 | 0.372 |
| Mistral-Nemo-Instruct-2407 | 0.440 | 0.440 | 0.607 | 0.607 | 0.600 | 0.600 | 0.507 | 0.507 | 0.587 | 0.587 | 0.560 | 0.560 | 0.580 | 0.580 | 0.460 | 0.460 | 0.300 | 0.300 | 0.380 | 0.380 | 0.502 | 0.502 |
| Mixtral-8x7B-Instruct-v0.1 | 0.547 | 0.547 | 0.567 | 0.567 | 0.400 | 0.400 | 0.507 | 0.507 | 0.213 | 0.213 | 0.460 | 0.460 | 0.727 | 0.727 | 0.227 | 0.227 | 0.307 | 0.313 | 0.207 | 0.207 | 0.416 | 0.417 |
| Phi-3-medium-128k-instruct | 0.020 | 0.020 | 0.007 | 0.007 | 0.047 | 0.047 | 0.120 | 0.120 | 0.020 | 0.020 | 0.067 | 0.067 | 0.093 | 0.093 | 0.007 | 0.007 | 0.007 | 0.007 | 0.013 | 0.013 | 0.040 | 0.040 |
| Phi-3-mini-128k-instruct | 0.027 | 0.027 | 0.013 | 0.013 | 0.020 | 0.020 | 0.087 | 0.087 | 0.007 | 0.007 | 0.027 | 0.027 | 0.060 | 0.060 | 0.013 | 0.013 | 0.000 | 0.000 | 0.007 | 0.007 | 0.026 | 0.026 |
| Phi-3.5-mini-instruct | 0.107 | 0.107 | 0.060 | 0.060 | 0.133 | 0.133 | 0.180 | 0.180 | 0.053 | 0.053 | 0.120 | 0.120 | 0.313 | 0.313 | 0.060 | 0.060 | 0.033 | 0.033 | 0.033 | 0.033 | 0.109 | 0.109 |
| gemma-2-27b-it | 0.280 | 0.280 | 0.153 | 0.153 | 0.340 | 0.340 | 0.620 | 0.620 | 0.213 | 0.213 | 0.133 | 0.133 | 0.400 | 0.400 | 0.073 | 0.073 | 0.093 | 0.093 | 0.187 | 0.187 | 0.249 | 0.249 |
| gemma-2-9b-it | 0.347 | 0.347 | 0.293 | 0.293 | 0.307 | 0.307 | 0.580 | 0.580 | 0.193 | 0.193 | 0.233 | 0.233 | 0.413 | 0.413 | 0.100 | 0.100 | 0.027 | 0.027 | 0.213 | 0.213 | 0.271 | 0.271 |
| Command R | 0.493 | 0.493 | 0.433 | 0.433 | 0.447 | 0.447 | 0.600 | 0.600 | 0.467 | 0.467 | 0.433 | 0.433 | 0.527 | 0.527 | 0.360 | 0.360 | 0.207 | 0.207 | 0.460 | 0.460 | 0.443 | 0.443 |
| Command R+ | 0.373 | 0.373 | 0.620 | 0.620 | 0.533 | 0.533 | 0.733 | 0.733 | 0.427 | 0.427 | 0.487 | 0.487 | 0.760 | 0.760 | 0.393 | 0.393 | 0.247 | 0.247 | 0.533 | 0.533 | 0.511 | 0.511 |
| gpt-3.5-turbo | 0.087 | 0.087 | 0.100 | 0.100 | 0.227 | 0.227 | 0.320 | 0.320 | 0.147 | 0.147 | 0.147 | 0.147 | 0.220 | 0.220 | 0.040 | 0.040 | 0.100 | 0.100 | 0.053 | 0.053 | 0.144 | 0.144 |
| gpt-4-turbo | 0.800 | 0.800 | 0.933 | 0.933 | 0.913 | 0.913 | 0.920 | 0.920 | 0.893 | 0.893 | 0.807 | 0.807 | 0.967 | 0.967 | 0.773 | 0.773 | 0.767 | 0.767 | 0.773 | 0.773 | 0.855 | 0.855 |
| gpt-4o | 0.960 | 0.960 | 0.900 | 0.900 | 0.960 | 0.960 | 0.960 | 0.960 | 0.960 | 0.960 | 0.880 | 0.880 | 0.980 | 0.980 | 0.813 | 0.813 | 0.973 | 0.973 | 0.933 | 0.933 | 0.932 | 0.932 |
| gpt-4o-mini | 0.660 | 0.660 | 0.527 | 0.527 | 0.787 | 0.787 | 0.660 | 0.660 | 0.407 | 0.407 | 0.373 | 0.373 | 0.620 | 0.620 | 0.740 | 0.740 | 0.347 | 0.347 | 0.580 | 0.580 | 0.570 | 0.570 |
| Claude 3.5 Sonnet | 0.993 | 0.993 | 0.960 | 0.960 | 0.960 | 0.960 | 0.953 | 0.953 | 0.987 | 0.987 | 0.913 | 0.913 | 0.987 | 0.987 | 0.800 | 0.800 | 0.947 | 0.947 | 0.940 | 0.940 | 0.944 | 0.944 |

## Appendix D Prompts for Contextual QA Generation

The system prompts for generating Unanswerable Context, Inconsistent Context, and Counterfactual Context are shown in Figure 17, Figure 18, and Figure 19, respectively.
For Inconsistent Context, our initial experiments suggest that a successful strategy is to decompose the generation of contextual QA into two steps. Step 1: generate a new answer that is fabricated and challenges common sense or well-known facts. Step 2: generate a modified context with fabricated evidence that supports the new answer. The model will output a JSON object containing the provided question, the provided old answer, the new answer, the modified context, and a concise justification on (1) if the new answer is supported by the new context (2) if all mentions of the old answer have been replaced or removed. We find that having justifications significantly improves the context quality. The generated context is concatenated with the original context to create an inconsistent context.

Figure: Figure 17: System prompt for Unanswerable Context.
Refer to caption: x4.png

Figure: Figure 18: System prompt for Inconsistent Context.
Refer to caption: x5.png

Figure: Figure 19: System prompt for Counterfactual Context.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - The construction processes of the three kinds of datasets are described only briefly in the paper; a more detailed description should be provided.
- In Line 296, the performance of Phi-3-medium-128k-instruct is stated as 76.8%, but in Figure 4, it is listed as 75.8%. Is this a typo?
- It may be useful to include some relevant papers in citations:
  - Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
  - Cognitive Mirage: A Review of Hallucinations in Large Language Models
  - A Survey on the Honesty of Large Language Models
soundness: 3
strengths: - FaithEval fills a gap by specifically addressing contextual faithfulness, unlike other benchmarks focused primarily on factuality or general performance.

- The paper details a comprehensive four-stage process involving both automated and human validation.

- The experiments compare diverse LLMs, providing insights into model limitations and the correlation between model size and faithfulness. The analysis across multiple decoding and prompting strategies is useful.
summary: The paper introduces FaithEval, a benchmark designed to evaluate the faithfulness of LLMs in responding accurately to context, especially in retrieval-augmented generation systems. FaithEval encompasses three tasks: unanswerable, inconsistent, and counterfactual contexts, emulating situations where retrieved information might be incomplete, contradictory, or counterfactual. The benchmark comprises 4.9K rigorously validated samples, and an evaluation across open-source and proprietary models highlights persistent challenges in achieving contextual faithfulness. This study suggests that even advanced models like GPT-4 struggle to align responses with given contexts.
weaknesses: - When constructing Unanswerable Context, this paper opts to instruct an LLM to check sentence by sentence whether a single sentence contains the answer to the question (as shown in Figure 17). However, sometimes a question cannot be answered based on any individual sentence in the context; it requires reasoning across multiple sentences. For example, if the question is about the relationship between two characters, multiple sentences in the context might need to be combined to infer the answer. In such cases, when checking each sentence individually, none of them alone will reveal the answer, allowing all sentences to be retained. Yet, the LLM can still infer the answer by reasoning across multiple sentences in the context, making this sample technically not unanswerable. Does the original dataset used in this paper contain such cases? (As far as I know, the HotpotQA dataset used in this paper seems to include such cases.) If so, how frequently do they occur? Are there improved methods for constructing Unanswerable Context to address this issue?
- When constructing Inconsistent Context, only the sentence in the original context that contains the old answer is changed to a new answer, while the rest remains unchanged. This approach makes the new context highly similar to the original context (potentially differing by only one sentence). Additionally, the new context is mixed with the old context to form the constructed context, resulting in an Inconsistent Context that contains a large amount of repetitive information, which does not seem to be a desirable characteristic for a good dataset.
- When constructing Counterfactual Context, the paper does not provide detailed information on the selection of the counterfactual answer. Additionally, are the answer options taken directly from the original dataset, or are they entirely re-generated? If it’s the former, does each question in the ARC-Challenge dataset include a counterfactual answer among its options? If some questions in the ARC-Challenge dataset do not contain a counterfactual answer among the candidate answers, how is this counterfactual answer generated? Conversely, if some questions in the ARC-Challenge dataset include multiple counterfactual answers, is one randomly selected as the counterfactual answer? Furthermore, is the constructed Counterfactual Context dataset still presented as multiple-choice questions? Please clarify these issues.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: please see weakness above
soundness: 2
strengths: FaithEval provides detailed contextual faithfulness assessment, which is a critical metric missing from many traditional benchmarks. The methodology is rigorous, combining both automated and human verification to ensure the benchmark's reliability.
summary: This paper introduces FaithEval, a benchmark specifically designed to evaluate large language models' (LLMs) ability to maintain contextual accuracy in complex scenarios. FaithEval focuses on the issue of faithfulness hallucination - instances where model-generated content either deviates from the given context or includes unverified information. This benchmark tests models through three main tasks (unanswerable queries, inconsistencies, and counterfactual contexts) to assess whether models can avoid hallucinations and remain faithful to provided content when faced with incomplete, contradictory, or counterfactual information. FaithEval helps address the limitations of current evaluation tools in examining models' contextual faithfulness.
weaknesses: 1. About unanswerable questions:
The paper uses prompts to control that answers must be based on context, even though LLMs might know these answers. This approach seems to test instruction following rather than true reasoning. Two questions:
a. why not create synthetic questions that LLMs couldn't know beforehand
b. why not analyze cases where LLMs answer correctly despite a lack of context support, including frequency and reasons
2. About inconsistent contexts:
The paper's approach to constructing inconsistent contexts shares similarities with counterfactual construction. Key points: The distinction between inconsistent and counterfactual contexts needs clarification.
The paper would benefit from separate evaluations of the following:
a. Original context performance
b. Counterfactual reasoning
c. Inconsistency eval
3. About evaluation methodology:
The separate evaluation of these three aspects (original, counterfactual, inconsistent) may not reflect real-world scenarios where these challenges occur simultaneously. A comprehensive evaluation combining all three aspects would be more practical and insightful.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Q1: In Section 3.2, for the counterfactual context task, it’s mentioned that a context passes if all words from the answer are present in the context. Given this criterion, why not use an automated method instead? Additionally, if the context includes a phrase like “[answer] is wrong,” would that still be considered as passing?

Q2: Is the prompt used for setting up the counterfactual task specified in paper?

Suggestion: It seems that the template used might be outdated.
soundness: 2
strengths: 1. Context faithfulness is essential for enhancing LLM reliability in RAG. This paper offers a well-structured benchmark that captures three common issues in RAG contexts: incomplete, inconsistent, and counterfactual. The resulting dataset is a valuable resource for advancing research on RAG-related challenges.
2. The paper is well-written and organized, with a logical flow that clearly explains the motivation, benchmark construction, and validation process. Figures are highly informative, effectively supporting the paper’s core findings and methodology.
3. The experimental evaluation is extensive and thorough, covering 18 state-of-the-art LLMs.
summary: This paper introduces a comprehensive benchmark, FaithfulEval, designed to assess the faithfulness of LLMs when they are provided with contextual information—a crucial aspect for applications like RAG The benchmark categorizes contextual scenarios into three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. The authors implement a four-stage framework for constructing and validating contexts, involving both automated LLM assessments and human validation, resulting in a dataset of 4.9k problems. The study evaluates several open-source and proprietary models, demonstrating that current LLMs face significant challenges in maintaining faithfulness to the given context.
weaknesses: 1. **Overlap with Prior Work:** The three tasks—unanswerable, inconsistent, and counterfactual—each has been partially covered in prior benchmarks. The paper could clarify its unique contributions in methodology and purpose, with more detailed comparisons to similar works [1-4].

2. **Feasibility concerns of unanswerable task setup:** The definition implies that models should refuse to answer if the context lacks sufficient information, even if models possess the internal knowledge to answer. However, ensuring original contexts are “truly answerable” is challenging. For example, with the context “California is a place where tech is booming and Stanford University is located here,” and the question “Which US state has Stanford University?”, the model must use its internal knowledge that "California is a US state" to answer. Should such cases be classified as answerable or unanswerable? If answerable, the authors might need to clarify which common-sense knowledge doesn’t require explicit presentation in the context. If unanswerable, many contextual QA datasets may include contexts that rely on common knowledge. Additionally, in RAG setups, contexts typically complement internal knowledge to yield accurate answers, so defining answerability based on model knowledge might be more practical.
3. **Limitations in Simulating Inconsistent Contexts:** The study claims that the “inconsistent” task replicates real RAG cases with contradictory contexts. However, it only tests cases with one correct and one false context concatenated. A more realistic simulation would include multiple contradictory contexts with different proportions in varied orders. Alternatively, the authors might consider tempering their claim to reflect a focus on cases with limited context contradiction.  In that case, additional analysis on the model’s behavior in these limited scenarios would further strengthen the paper.
4. **Evaluation Limitation**. The paper’s findings that models struggle with unanswerable or inconsistent contexts are unsurprising, as instruction tuning often biases models toward affirmative responses rather than outputs like “unknown” or “conflict.” This does not necessarily mean models lack the ability to recognize unanswerable or conflicting contexts. For example, log-probs for “unknown” may still increase in unanswerable contexts, even if not generated. Bias toward affirmative answers could likely be reduced through methods such as few-shot demonstrations, logit calibration for “unknown”/“conflict” tokens, or fine-tuning on smaller datasets, potentially altering the study’s conclusions.

[1] Jiawei Chen, et al. "Benchmarking Large Language Models in Retrieval-Augmented Generation". arxiv.org/abs/2309.01431 \
[2] Yikang Pan, et al. "On the Risk of Misinformation Pollution with Large Language Models". arxiv.org/abs/2305.13661 \
[3] Chong Xiang, Tong Wu, et al. "Certifiably Robust RAG against Retrieval Corruption".  arxiv.org/abs/2405.15556 \
[4] Kevin Wu, et al. "ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence". arxiv.org/abs/2404.10198

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weakness section, and I hope the authors can provide some insights on improving the faithfulness of LLMs based on the experimental results.
soundness: 2
strengths: 1. Faithfulness is a critical issue in the field of language models and RAG domain. The author's exploration of Unanswerable Context, Inconsistent Context, and Counterfactual Context is very reasonable.

2. The author's four-step context construction and validation framework is clear, concise, and reliable, aligning with my expectation of automating benchmark construction with minimal human correction.

3. Overall, the writing is clear and easy to understand.
summary: This paper introduces FaithEval, a novel and comprehensive benchmark designed to evaluate the faithfulness of large language models (LLMs) in contextual scenarios, covering three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may yield incomplete, contradictory, or fabricated information. FaithEval comprises a total of 4,900 high-quality problems, validated through a rigorous four-stage context construction and validation framework, employing both LLM-based auto-evaluation and human validation. Extensive studies reveal that even state-of-the-art models often struggle to maintain faithfulness to the given context, and that larger models do not necessarily exhibit improved faithfulness.
weaknesses: 1. This paper primarily focuses on the impact of unfaithful knowledge in documents/contexts on language models. However, there is already a key paper [1] that established a benchmark evaluation for the counterfactuality of documents, which this paper does not cite or clarify its differences from.

2. While the experimental results contain extensive performance comparisons, only experiments on “Does chain-of-thought prompting improve faithfulness?” provide some empirical conclusions that reveal the existence of faithfulness issues, lacking a deeper exploration and valuable conclusions.


3. The authors did not provide supplementary materials such as test data, evaluation code, and dataset licenses, which leaves me with a limited understanding of the reproducibility of their evaluations.

4. As a sensitive aspect, human annotator should provide more comprehensive information, including the number of annotators, their level of expertise, and whether they meet the local minimum wage standards.

[1] Benchmarking Large Language Models in Retrieval-Augmented Generation

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: ### Primary Question
How well do you think the tests you present really capture what you want models to do? I think that the unanswerable context is well-founded, but I am less sure of the other two sections of the benchmark. For inconsistent contexts, the question shown is not only inconsistent, but one of the answers is right and the other is wrong. This is a compounded difficulty. The real correct answer is probably not "inconsistent" but: "These documents do not agree although the correct answer is Richard Parker. Some of these source documents are lower quality or generated by an LLM and not to be trusted. Please use the fact that I recall or else double check this information by directly providing me with the source document." Finding real world examples that better illustrate the phenomena you want to test would/might help illustrate why these things are worth testing. In particular, the counterfactual context doesn't seem so much counterfactual as "incorrect context that is orthogonal to the question." I'm putting the above more strongly than I really think. Some discussion on why these parts of the dataset are useful/realistic is the main question I have for the authors moving forward and also the primary reason I haven't (yet) recommended the paper for acceptance. (Neither the introduction nor Section 3 motivate/justify these sections of the benchmark, best as I can see.)

### Minor Question
* for unanswerable context you note a 98% agreement rate. for inconsistent context I only saw the final outcome (1.5k) + that you required 2/3 agreement. from this information alone it is hard to tell if the results are high-quality. for instance, if the 2/3 agreement rate is at the rate you would expect from random labelers then this would indicate there is little agreement. (what would the agreement rate on the filtered/accepted data be if you labelled it again?) how many samples were there originally? what was the agreement rate?
* does using an llm to judge/filter questions somehow bias the results one way or another? how can the model make these judgements when it fails on the benchmark itself?

### Suggestion
* can you? highlight (somehow?) the ranking of the models before (original) and after the counterfactual changes (edited). right now models are ordered according to the original performance. somehow showing/computing the rank order correlation between the original and edited would be interesting. also find the rank order correlation between the edited performances across the various datasets would be interesting. you discuss this generally, but I think it could be shown more directly.
* figure 1 is missing the openai logo. https://openai.com/brand/ (just in case you want to add it.)

### Related Work
* https://openreview.net/pdf?id=OUiW2DzpzT looks to explain this problem
soundness: 3
strengths: * this paper provides test for faithfulness hallucination
* the authors clearly put a lot of effort/care into filtering the dataset
* the information presented in the paper itself is easy to understand
* the figures are clear; I really like Figure 2.
summary: ### Updated Review
From the thorough responses, it seems that the authors had already considered some of the raised questions. I will stand by my feedback, suggesting that presentation be improved to highlight what is being tested and give better rationales for the approach and design decisions. (Other reviewers note that differentiating from previous work is important. I think the effort to do so would be well worth it as it would clarify your motivations and contributions). (I found the ranking results interesting; please consider including in the appendix if not the main body of the paper!) All that being said, I think this work should be accepted.

-----------
### Paper Summary
This paper introduces a benchmark for testing if models are able to pull information only from a provided context. The source of questions are pulled from existing datasets and modified with LLMs (to introduce a counterfactual or make the question unanswerable, etc.) and then (mostly) verified by crowd-sourcing. The authors find that models are generally perform poorly on the benchmark, and that neither the size of the model nor the model's original performance on the question source well-predicts the models performance on the edited questions.

### Review Summary
This paper tests for faithfulness in the specified contexts were the outcomes are all starkly clear, however it is unclear to me that a "good" model would always have a high faithfulness score in these artificial scenarios, overriding factual information or else factual information that is supported by some documents and not others. In some sense, I don't think these artificial examples capture what the authors would want from models nor the real problem the authors are hoping to test for. Again, acknowledging or else motivating these design choices would make the paper stronger because it would be easier to understand/contextualize the findings.

### Breakdown
Soundness: The paper lacks ablations and discussions of its design choices. It tests a large range of models and clearly works diligently to filter the dataset samples.
Presentation: The presentation is strong in that the information/figures in the paper is clear. However, understanding the results and how well it captures the overall goal of a good model not hallucinating information is unclear to me.

Contribution: This is 2.5 rounded up for me; I think this benchmark is useful.
weaknesses: Minor:
* overall this paper does a poor job at highlighting its limitations/assumptions. raising these points, even if nothing can be done about them, would improve this paper.
    * when a model edits text it leaves traces. this might impact the results (say for inconsistency). the model might be able to do a good job at fully recovering the original answer if prompted.
    * few-shot examples might help the model produce "conflict" or "inconsistent"; testing this seems more pertinent than COT
    * there is little justification for the counterfactual questions. if I were to do the test myself (buoyant vs magnetic) I would personally struggle. the source material talks about magnetic wood but I don't see how its relevant to staying afloat. working to directly overcome factual information seems quite different from the other categories of problem. noting that while different people/organizations may want different outcomes, it is important to evaluate what models are capable of might help add context to the paper, may help the paper. adding this type of justification or context in the introduction might help because I was left unsure why the authors generally cared about these adversarial situations.
    * this work uses existing datasets for its source materials. these datasets will all be leaked (or already are). for regular benchmarks, it seems more straightforward to test/notice this leakage. possibly, the models memorize the original answers and the delta and performance would be wider for truly novel examples. I don't think this is the case, but discussing how data leakage will/would impact this benchmark would be helpful.
* I worry that the type of instruction tuning that different models have been exposed to might be fairly different. the top models (figure 5) for the inconsistency task perform very well. this (to me) suggests they have direct exposure to this type of problem, not necessarily that they are inherently better. i recognize that your benchmark doesn't adjudicate on this point.
* this paper does not have ablations or controls or baselines to contextualize the results. I don't think this is a major issue for this work, but adding these types of things might better justify their design decisions (like using an LLM to filter questions, e.g., is the answer valid given the context.) this filter might filter out the hardest questions (or easiest?). also, how many questions were filtered out using this filter?

Meta-review: The authors build a benchmark to evaluate faithfulness (following facts in the document provided, rather than parametric knowledge). The authors test unanswerable, contradictory, and counterfactual (i.e. the answer disagrees with commonsense) settings, and show LMs have significant ways to go in following this type of faithfulness goal. 

Faithfulness is an important problem, and this is a nice single benchmark to evaluate these different properties that comprise challenging examples to test faithfulness. Reviewers did note that this is not terribly novel (testing each of these attributes has been considered in the literature), and that unanswerable context might be slightly unnatural. However, on the whole the reviewers agree the paper is a nice contribution to an important area.

Decision: Accept (Poster)