FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"

Authors: Yifei Ming, Senthil Purushwalkam, Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Contextual LLM, Faithfulness, Hallucination, Benchmark and Evaluation

Abstract:
Ensuring faithfulness to context in large language models (LLMs) and retrieval-augmented generation (RAG) systems is crucial for reliable deployment in real-world applications, as incorrect or unsupported information can erode user trust. Despite advancements on standard benchmarks, faithfulness hallucination—where models generate responses misaligned with the provided context—remains a significant challenge. In this work, we introduce FaithEval, a novel and comprehensive benchmark tailored to evaluate the faithfulness of LLMs in contextual scenarios across three diverse tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may surface incomplete, contradictory, or fabricated information. FaithEval comprises 4.9K high-quality problems in total, validated through a rigorous four-stage context construction and validation framework, employing both LLM-based auto-evaluation and human validation. Our extensive study across a wide range of open-source and proprietary models reveals that even state-of-the-art models often struggle to remain faithful to the given context, and that larger models do not necessarily exhibit improved faithfulness. Code is available at: https://github.com/SalesforceAIResearch/FaithEval.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729785718623

thread_type:
official_review

<thread_object>
creator: reviewer_kkjw

<content>
summary:
### Updated Review
From the thorough responses, it seems that the authors had already considered some of the raised questions. I will stand by my feedback, suggesting that presentation be improved to highlight what is being tested and give better rationales for the approach and design decisions. (Other reviewers note that differentiating from previous work is important. I think the effort to do so would be well worth it as it would clarify your motivations and contributions). (I found the ranking results interesting; please consider including in the appendix if not the main body of the paper!) All that being said, I think this work should be accepted.

-----------
### Paper Summary
This paper introduces a benchmark for testing if models are able to pull information only from a provided context. The source of questions are pulled from existing datasets and modified with LLMs (to introduce a counterfactual or make the question unanswerable, etc.) and then (mostly) verified by crowd-sourcing. The authors find that models are generally perform poorly on the benchmark, and that neither the size of the model nor the model's original performance on the question source well-predicts the models performance on the edited questions.

### Review Summary
This paper tests for faithfulness in the specified contexts were the outcomes are all starkly clear, however it is unclear to me that a "good" model would always have a high faithfulness score in these artificial scenarios, overriding factual information or else factual information that is supported by some documents and not others. In some sense, I don't think these artificial examples capture what the authors would want from models nor the real problem the authors are hoping to test for. Again, acknowledging or else motivating these design choices would make the paper stronger because it would be easier to understand/contextualize the findings.

### Breakdown
Soundness: The paper lacks ablations and discussions of its design choices. It tests a large range of models and clearly works diligently to filter the dataset samples.
Presentation: The presentation is strong in that the information/figures in the paper is clear. However, understanding the results and how well it captures the overall goal of a good model not hallucinating information is unclear to me.

Contribution: This is 2.5 rounded up for me; I think this benchmark is useful.

strengths:
* this paper provides test for faithfulness hallucination
* the authors clearly put a lot of effort/care into filtering the dataset
* the information presented in the paper itself is easy to understand
* the figures are clear; I really like Figure 2.

weaknesses:
Minor:
* overall this paper does a poor job at highlighting its limitations/assumptions. raising these points, even if nothing can be done about them, would improve this paper.
    * when a model edits text it leaves traces. this might impact the results (say for inconsistency). the model might be able to do a good job at fully recovering the original answer if prompted.
    * few-shot examples might help the model produce "conflict" or "inconsistent"; testing this seems more pertinent than COT
    * there is little justification for the counterfactual questions. if I were to do the test myself (buoyant vs magnetic) I would personally struggle. the source material talks about magnetic wood but I don't see how its relevant to staying afloat. working to directly overcome factual information seems quite different from the other categories of problem. noting that while different people/organizations may want different outcomes, it is important to evaluate what models are capable of might help add context to the paper, may help the paper. adding this type of justification or context in the introduction might help because I was left unsure why the authors generally cared about these adversarial situations.
    * this work uses existing datasets for its source materials. these datasets will all be leaked (or already are). for regular benchmarks, it seems more straightforward to test/notice this leakage. possibly, the models memorize the original answers and the delta and performance would be wider for truly novel examples. I don't think this is the case, but discussing how data leakage will/would impact this benchmark would be helpful.
* I worry that the type of instruction tuning that different models have been exposed to might be fairly different. the top models (figure 5) for the inconsistency task perform very well. this (to me) suggests they have direct exposure to this type of problem, not necessarily that they are inherently better. i recognize that your benchmark doesn't adjudicate on this point.
* this paper does not have ablations or controls or baselines to contextualize the results. I don't think this is a major issue for this work, but adding these types of things might better justify their design decisions (like using an LLM to filter questions, e.g., is the answer valid given the context.) this filter might filter out the hardest questions (or easiest?). also, how many questions were filtered out using this filter?

questions:
### Primary Question
How well do you think the tests you present really capture what you want models to do? I think that the unanswerable context is well-founded, but I am less sure of the other two sections of the benchmark. For inconsistent contexts, the question shown is not only inconsistent, but one of the answers is right and the other is wrong. This is a compounded difficulty. The real correct answer is probably not "inconsistent" but: "These documents do not agree although the correct answer is Richard Parker. Some of these source documents are lower quality or generated by an LLM and not to be trusted. Please use the fact that I recall or else double check this information by directly providing me with the source document." Finding real world examples that better illustrate the phenomena you want to test would/might help illustrate why these things are worth testing. In particular, the counterfactual context doesn't seem so much counterfactual as "incorrect context that is orthogonal to the question." I'm putting the above more strongly than I really think. Some discussion on why these parts of the dataset are useful/realistic is the main question I have for the authors moving forward and also the primary reason I haven't (yet) recommended the paper for acceptance. (Neither the introduction nor Section 3 motivate/justify these sections of the benchmark, best as I can see.)

### Minor Question
* for unanswerable context you note a 98% agreement rate. for inconsistent context I only saw the final outcome (1.5k) + that you required 2/3 agreement. from this information alone it is hard to tell if the results are high-quality. for instance, if the 2/3 agreement rate is at the rate you would expect from random labelers then this would indicate there is little agreement. (what would the agreement rate on the filtered/accepted data be if you labelled it again?) how many samples were there originally? what was the agreement rate?
* does using an llm to judge/filter questions somehow bias the results one way or another? how can the model make these judgements when it fails on the benchmark itself?

### Suggestion
* can you? highlight (somehow?) the ranking of the models before (original) and after the counterfactual changes (edited). right now models are ordered according to the original performance. somehow showing/computing the rank order correlation between the original and edited would be interesting. also find the rank order correlation between the edited performances across the various datasets would be interesting. you discuss this generally, but I think it could be shown more directly.
* figure 1 is missing the openai logo. https://openai.com/brand/ (just in case you want to add it.)

### Related Work
* https://openreview.net/pdf?id=OUiW2DzpzT looks to explain this problem
</content>
</thread_object>

</thread>
<thread>
creation date:
1730133185404

thread_type:
official_review

<thread_object>
creator: reviewer_gcnd

<content>
summary:
This paper introduces FaithEval, a novel and comprehensive benchmark designed to evaluate the faithfulness of large language models (LLMs) in contextual scenarios, covering three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may yield incomplete, contradictory, or fabricated information. FaithEval comprises a total of 4,900 high-quality problems, validated through a rigorous four-stage context construction and validation framework, employing both LLM-based auto-evaluation and human validation. Extensive studies reveal that even state-of-the-art models often struggle to maintain faithfulness to the given context, and that larger models do not necessarily exhibit improved faithfulness.

strengths:
1. Faithfulness is a critical issue in the field of language models and RAG domain. The author's exploration of Unanswerable Context, Inconsistent Context, and Counterfactual Context is very reasonable.

2. The author's four-step context construction and validation framework is clear, concise, and reliable, aligning with my expectation of automating benchmark construction with minimal human correction.

3. Overall, the writing is clear and easy to understand.

weaknesses:
1. This paper primarily focuses on the impact of unfaithful knowledge in documents/contexts on language models. However, there is already a key paper [1] that established a benchmark evaluation for the counterfactuality of documents, which this paper does not cite or clarify its differences from.

2. While the experimental results contain extensive performance comparisons, only experiments on “Does chain-of-thought prompting improve faithfulness?” provide some empirical conclusions that reveal the existence of faithfulness issues, lacking a deeper exploration and valuable conclusions.


3. The authors did not provide supplementary materials such as test data, evaluation code, and dataset licenses, which leaves me with a limited understanding of the reproducibility of their evaluations.

4. As a sensitive aspect, human annotator should provide more comprehensive information, including the number of annotators, their level of expertise, and whether they meet the local minimum wage standards.

[1] Benchmarking Large Language Models in Retrieval-Augmented Generation

questions:
Please see the weakness section, and I hope the authors can provide some insights on improving the faithfulness of LLMs based on the experimental results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730421443966

thread_type:
official_review

<thread_object>
creator: reviewer_14j3

<content>
summary:
This paper introduces a comprehensive benchmark, FaithfulEval, designed to assess the faithfulness of LLMs when they are provided with contextual information—a crucial aspect for applications like RAG The benchmark categorizes contextual scenarios into three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. The authors implement a four-stage framework for constructing and validating contexts, involving both automated LLM assessments and human validation, resulting in a dataset of 4.9k problems. The study evaluates several open-source and proprietary models, demonstrating that current LLMs face significant challenges in maintaining faithfulness to the given context.

strengths:
1. Context faithfulness is essential for enhancing LLM reliability in RAG. This paper offers a well-structured benchmark that captures three common issues in RAG contexts: incomplete, inconsistent, and counterfactual. The resulting dataset is a valuable resource for advancing research on RAG-related challenges.
2. The paper is well-written and organized, with a logical flow that clearly explains the motivation, benchmark construction, and validation process. Figures are highly informative, effectively supporting the paper’s core findings and methodology.
3. The experimental evaluation is extensive and thorough, covering 18 state-of-the-art LLMs.

weaknesses:
1. **Overlap with Prior Work:** The three tasks—unanswerable, inconsistent, and counterfactual—each has been partially covered in prior benchmarks. The paper could clarify its unique contributions in methodology and purpose, with more detailed comparisons to similar works [1-4].

2. **Feasibility concerns of unanswerable task setup:** The definition implies that models should refuse to answer if the context lacks sufficient information, even if models possess the internal knowledge to answer. However, ensuring original contexts are “truly answerable” is challenging. For example, with the context “California is a place where tech is booming and Stanford University is located here,” and the question “Which US state has Stanford University?”, the model must use its internal knowledge that "California is a US state" to answer. Should such cases be classified as answerable or unanswerable? If answerable, the authors might need to clarify which common-sense knowledge doesn’t require explicit presentation in the context. If unanswerable, many contextual QA datasets may include contexts that rely on common knowledge. Additionally, in RAG setups, contexts typically complement internal knowledge to yield accurate answers, so defining answerability based on model knowledge might be more practical.
3. **Limitations in Simulating Inconsistent Contexts:** The study claims that the “inconsistent” task replicates real RAG cases with contradictory contexts. However, it only tests cases with one correct and one false context concatenated. A more realistic simulation would include multiple contradictory contexts with different proportions in varied orders. Alternatively, the authors might consider tempering their claim to reflect a focus on cases with limited context contradiction.  In that case, additional analysis on the model’s behavior in these limited scenarios would further strengthen the paper.
4. **Evaluation Limitation**. The paper’s findings that models struggle with unanswerable or inconsistent contexts are unsurprising, as instruction tuning often biases models toward affirmative responses rather than outputs like “unknown” or “conflict.” This does not necessarily mean models lack the ability to recognize unanswerable or conflicting contexts. For example, log-probs for “unknown” may still increase in unanswerable contexts, even if not generated. Bias toward affirmative answers could likely be reduced through methods such as few-shot demonstrations, logit calibration for “unknown”/“conflict” tokens, or fine-tuning on smaller datasets, potentially altering the study’s conclusions.

[1] Jiawei Chen, et al. "Benchmarking Large Language Models in Retrieval-Augmented Generation". arxiv.org/abs/2309.01431 \
[2] Yikang Pan, et al. "On the Risk of Misinformation Pollution with Large Language Models". arxiv.org/abs/2305.13661 \
[3] Chong Xiang, Tong Wu, et al. "Certifiably Robust RAG against Retrieval Corruption".  arxiv.org/abs/2405.15556 \
[4] Kevin Wu, et al. "ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence". arxiv.org/abs/2404.10198

questions:
Q1: In Section 3.2, for the counterfactual context task, it’s mentioned that a context passes if all words from the answer are present in the context. Given this criterion, why not use an automated method instead? Additionally, if the context includes a phrase like “[answer] is wrong,” would that still be considered as passing?

Q2: Is the prompt used for setting up the counterfactual task specified in paper?

Suggestion: It seems that the template used might be outdated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602298079

thread_type:
official_review

<thread_object>
creator: reviewer_ykwg

<content>
summary:
This paper introduces FaithEval, a benchmark specifically designed to evaluate large language models' (LLMs) ability to maintain contextual accuracy in complex scenarios. FaithEval focuses on the issue of faithfulness hallucination - instances where model-generated content either deviates from the given context or includes unverified information. This benchmark tests models through three main tasks (unanswerable queries, inconsistencies, and counterfactual contexts) to assess whether models can avoid hallucinations and remain faithful to provided content when faced with incomplete, contradictory, or counterfactual information. FaithEval helps address the limitations of current evaluation tools in examining models' contextual faithfulness.

strengths:
FaithEval provides detailed contextual faithfulness assessment, which is a critical metric missing from many traditional benchmarks. The methodology is rigorous, combining both automated and human verification to ensure the benchmark's reliability.

weaknesses:
1. About unanswerable questions:
The paper uses prompts to control that answers must be based on context, even though LLMs might know these answers. This approach seems to test instruction following rather than true reasoning. Two questions:
a. why not create synthetic questions that LLMs couldn't know beforehand
b. why not analyze cases where LLMs answer correctly despite a lack of context support, including frequency and reasons
2. About inconsistent contexts:
The paper's approach to constructing inconsistent contexts shares similarities with counterfactual construction. Key points: The distinction between inconsistent and counterfactual contexts needs clarification.
The paper would benefit from separate evaluations of the following:
a. Original context performance
b. Counterfactual reasoning
c. Inconsistency eval
3. About evaluation methodology:
The separate evaluation of these three aspects (original, counterfactual, inconsistent) may not reflect real-world scenarios where these challenges occur simultaneously. A comprehensive evaluation combining all three aspects would be more practical and insightful.

questions:
please see weakness above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742580262

thread_type:
official_review

<thread_object>
creator: reviewer_5p3k

<content>
summary:
The paper introduces FaithEval, a benchmark designed to evaluate the faithfulness of LLMs in responding accurately to context, especially in retrieval-augmented generation systems. FaithEval encompasses three tasks: unanswerable, inconsistent, and counterfactual contexts, emulating situations where retrieved information might be incomplete, contradictory, or counterfactual. The benchmark comprises 4.9K rigorously validated samples, and an evaluation across open-source and proprietary models highlights persistent challenges in achieving contextual faithfulness. This study suggests that even advanced models like GPT-4 struggle to align responses with given contexts.

strengths:
- FaithEval fills a gap by specifically addressing contextual faithfulness, unlike other benchmarks focused primarily on factuality or general performance.

- The paper details a comprehensive four-stage process involving both automated and human validation.

- The experiments compare diverse LLMs, providing insights into model limitations and the correlation between model size and faithfulness. The analysis across multiple decoding and prompting strategies is useful.

weaknesses:
- When constructing Unanswerable Context, this paper opts to instruct an LLM to check sentence by sentence whether a single sentence contains the answer to the question (as shown in Figure 17). However, sometimes a question cannot be answered based on any individual sentence in the context; it requires reasoning across multiple sentences. For example, if the question is about the relationship between two characters, multiple sentences in the context might need to be combined to infer the answer. In such cases, when checking each sentence individually, none of them alone will reveal the answer, allowing all sentences to be retained. Yet, the LLM can still infer the answer by reasoning across multiple sentences in the context, making this sample technically not unanswerable. Does the original dataset used in this paper contain such cases? (As far as I know, the HotpotQA dataset used in this paper seems to include such cases.) If so, how frequently do they occur? Are there improved methods for constructing Unanswerable Context to address this issue?
- When constructing Inconsistent Context, only the sentence in the original context that contains the old answer is changed to a new answer, while the rest remains unchanged. This approach makes the new context highly similar to the original context (potentially differing by only one sentence). Additionally, the new context is mixed with the old context to form the constructed context, resulting in an Inconsistent Context that contains a large amount of repetitive information, which does not seem to be a desirable characteristic for a good dataset.
- When constructing Counterfactual Context, the paper does not provide detailed information on the selection of the counterfactual answer. Additionally, are the answer options taken directly from the original dataset, or are they entirely re-generated? If it’s the former, does each question in the ARC-Challenge dataset include a counterfactual answer among its options? If some questions in the ARC-Challenge dataset do not contain a counterfactual answer among the candidate answers, how is this counterfactual answer generated? Conversely, if some questions in the ARC-Challenge dataset include multiple counterfactual answers, is one randomly selected as the counterfactual answer? Furthermore, is the constructed Counterfactual Context dataset still presented as multiple-choice questions? Please clarify these issues.

questions:
- The construction processes of the three kinds of datasets are described only briefly in the paper; a more detailed description should be provided.
- In Line 296, the performance of Phi-3-medium-128k-instruct is stated as 76.8%, but in Figure 4, it is listed as 75.8%. Is this a typo?
- It may be useful to include some relevant papers in citations:
  - Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
  - Cognitive Mirage: A Review of Hallucinations in Large Language Models
  - A Survey on the Honesty of Large Language Models
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors build a benchmark to evaluate faithfulness (following facts in the document provided, rather than parametric knowledge). The authors test unanswerable, contradictory, and counterfactual (i.e. the answer disagrees with commonsense) settings, and show LMs have significant ways to go in following this type of faithfulness goal. 

Faithfulness is an important problem, and this is a nice single benchmark to evaluate these different properties that comprise challenging examples to test faithfulness. Reviewers did note that this is not terribly novel (testing each of these attributes has been considered in the literature), and that unanswerable context might be slightly unnatural. However, on the whole the reviewers agree the paper is a nice contribution to an important area.