FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
Authors: Yifei Ming
Senthil Purushwalkam
Shrey Pandit
Zixuan Ke
Xuan-Phi Nguyen
Caiming Xiong
Shafiq Joty
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Contextual LLM
Faithfulness
Hallucination
Benchmark and Evaluation
Primary area: foundation or frontier models, including LLMs

Abstract: Ensuring faithfulness to context in large language models (LLMs) and retrieval-augmented generation (RAG) systems is crucial for reliable deployment in real-world applications, as incorrect or unsupported information can erode user trust. Despite advancements on standard benchmarks, faithfulness hallucination—where models generate responses misaligned with the provided context—remains a significant challenge. In this work, we introduce FaithEval, a novel and comprehensive benchmark tailored to evaluate the faithfulness of LLMs in contextual scenarios across three diverse tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may surface incomplete, contradictory, or fabricated information. FaithEval comprises 4.9K high-quality problems in total, validated through a rigorous four-stage context construction and validation framework, employing both LLM-based auto-evaluation and human validation. Our extensive study across a wide range of open-source and proprietary models reveals that even state-of-the-art models often struggle to remain faithful to the given context, and that larger models do not necessarily exhibit improved faithfulness. Code is available at: https://github.com/SalesforceAIResearch/FaithEval.

Published as a conference paper at ICLR 2025
FAITHEVAL: CAN YOUR LANGUAGE MODEL STAY FAITHFUL TO CONTEXT, EVEN IF "THE MOON IS MADE OF MARSHMALLOWS"
Yifei Ming' Senthil Purushwalkam', Shrey Pandit', Zixuan Ke¹, Xuan-Phi Nguyen* Caiming Xiong', Shafiq Joty
"Salesforce AI Research
Universityy of Texas at Austin
ABSTRACT
Ensuring faithfulness to context in large language models (LLMs) and retrieval- augmented generation (RAG) systems is crucial for reliable deployment in real- world applications, as incorrect or unsupported information can erode user trust. Despite advancements on standard benchmarks, faithfulness hallucination-wher models generate responses misaligned with the provided context- nificant challenge. In this work, we introduce FaithEval, novel and compre- hensive benchmark tailored to evaluate the faithfulness of LLMs in contextual scenarios across three diverse tasks: unanswerable, inconsistent, and counterfac- tual contexts. These tasks simulate real-world challenges where retrieval mecha- nisms may surface incomplete, contradictory, or fabricated information. FaithE- val comprises 4.9K high-quality problems in total, validated through a rigorous four-stage context construction and validation framework, employing both LLM- based auto-evaluation and human validation. Our extensive study across a wide range of ppencsource and proprietary models reveals that even state-of-the-art models often struggle to remain faithful to the given context, and that larger models do not necessarily exhibit improved faithfulness. Code is available https: / tht....ttttug/.gg///////.t com
a sig-
INTRODUCTION
The rapid development of large language models (LLMs) has significantly advanced natural language understanding and generation tasks, enabling systems to produce fluent and coherent responses across variety applications (Bubeck et al., 2023; Zhao al., 2024b; Wu al., 2024b). The capabilities of these models have been further enhanced by integrating external information from Generation
the Internet or knowledge sources using a popular approach of (RAG) (Lewis et al., 2020; Zhao et al., 2024a). In this paradigm, generated outputs are enhanced by retrieving and encoding relevant information as context to the model. While RAG facilitates the integration of additional knowledge, mallucination-wleee models generate unsupported or ungrounded content remains a critical challenge (Nguyen et al., 2024).
Hallucination in LLMs can be generally categorized into two types: factual hallucination, where generated content deviates from established world knowledge, and faithfulness hallucination. where the generated response is inconsistent with the provided context (Huang al., 2023a). While factuality has received extensive attention, with numerous benchmarks designed to evaluate correctness against common sense world knowledge (Lee al., 2022; Min al., 2023; Chern et al., 2023; Wei et al., 2024), a fine-grain grained and holistic evaluation of faithfulness on noisy contexts remains underexplored, particularly when the context contradicts commonly accepted facts. Maintaining faithfulness the context is especially important for various personalized applications and can be critical in high-stakes domains such as healthcare, finance and law, where inaccurate or ungrounded responses can erode user trust and lead to severe consequences (Bommarito & Katz, 2022; Pal et al., 2023).
One of the key challenges in addressing faithfulness hallucination in RAG stems from the retrieval process, where the wealth of documents on the Internet varies in credibility. This complexity is
*Correspondence: yi fei .mi ng@sal es eeforcc.ccom

---

Published as a conference paper at ICLR 2025
Counterfactur incensistent Unenswerable
66.8
Overal Accuracy (normalin
54.8
53.1
45.5
Anthropic
Mistral
O
M
Figure 1: Performance summary on FaithEval Benchmark. Each bar shows the combined accuracy (normalized) for the best model from each organization across three tasks: Counterfactual, Inconsis- tent, and Unanswerable (Sec 3.1). Different colors each bar represent the accuracy each task.
further compounded when long retrieved content includes multiple relevant paragraphs that omit key details, present conflicting evidence, propagate counterfactual claims. Existing hallucination evaluation benchmarks fall short providing fine-grained assessments of how well models align their responses with the context. They often do disentangle factuality from faithfulness (Li et al., 2024b; Chen et al., 2024c) or capture the full range contextual nuances (Lin et al., 2022; Yin et al., 2023). Moreover, current hallucination detection solutions focus identifying hallucinations in model outputs (Liu et al., 2021; Li et al., 2023b; Hu et al., 2024), which is orthogonal to the task of understanding the impact of contexts on faithfulness hallucination.
In this work, we introduce FaithEval, a comprehensive benchmark specifically designed to evaluate the contextual faithfulness of LLMs across three diverse tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may surface incomplete, contradictory, or fabricated information (Figure 2). FaithEval includes a total of 4.9K high-quality samples, constructed using rigorous four-stage framework with multi-turn LLM-based context verification and human validation. We conduct a holistic evaluation on 18 representative proprietary and open-sourced models, revealing that faithfulness remains challenging, even for the most competitive LLMs, despite their strong performance on standard benchmarks. Figure summarizes the performance of representative models from each organization, with each bar representing performance on the individual tasks. To our knowledge, FaithEval is the first fine- grained and comprehensive benchmark specifically targeting contextual faithfulness hallucination, contributing to the broader effort toward developing reliable next-generation foundation models.
Our key contributions are summarized as follows:
We introduce FaithEval, novel and comprehensive benchmark dedicated evaluating contextual faithfulness in LLMSs across three diverse tasks: unanswerable, inconsistent, and counterfactual contexts.
We develop scalable four-stage framework for context construction and validation, which incorporates multi-turn LLM-based validation and human annotation to ensure high-quality contextual QA pairs.
We perform an extensive and in-depth study on wide range of competitive open-source and proprietary models. We highlight that faithfulness remains significant challenge for recent LLMs and that allegedly larger models, such as GPT-4o and -aae-----------tuut do not necessarily lead improved faithfulness.
RELATED WORKS
Contextual LLM and retrieval-augmeetedd generation. As the demand for contextual LLMs continues grow, retrieval-augmented generation (RAG) systems offer promising solution by
2

---

Published as a conference paper at ICLR 2025
integrating external knowledge retrieval with LLMs (Lewis et al., 2020; Sarto et al., 2022; Ramos et al., 2022; Huang et al., 2023b; Zhao et al., 2024a). In RAG, model responses are grounded using knowledge sourced from private or open-access data repositories. A typical RAG system operates through a close interaction between the retriever and a generator. The retriever (Li et al., 2023a; Meng al., 2024; Chen et al., 2024a) identifies relevant documents from the source, and this retrieved information is supplied to the generator (e.g., a language model) to produce grounded outputs (Lewis al., 2020; Izacard & Grave, 2020; Borgeaud et al., 2021; Ke al., 2024). Recent works develop more sophisticate RAG frameworks to improve answer reliability (Asai et al., 2023; Li et al., 2024d; Xu et al., 2024; Xiang et al., 2024). The increased context sizes in LLMs have improved their ability to handle longer text sequences, allowing them to handle complex tasks requiring extensive background knowledge (Gao et al., 2023; Song et al., 2024; Shi et al., 2024). These improvements are particularly beneficial for long-form question answering (Joshi et al., 2017; K Kwiatkowski et al., 2019b; Li et al., 2024a). However, the variation in source quality can exacerbate challenges to maintaining faithfulness in LLMs, especially in longer contexts retrieved from the Internet.
Hallucination and faithfulness evaluation. Hallucination in LLMS refers to the generation of ungrounded content, either from the provided context established world knowledge. The former is typically described as factuality hallucination, while the latter is known as faithfulness hallucination, which highlights the discrepancy between the model's output and the context (Huang et al., 2023a; Ye et al., 2023). While there is rich literature on factuality evaluation and benchmarks with and without contexts (Lee al., 2022: Min et al., 2023; Chern et al., 2023; Wei al., 2024; Li et al., 2024c), faithfulness has mostly been explored for summarization (Laban et al., 2023; Jia et al., 2023) natural language explanations (Atanasova al., 2023; Siegel et al., 2024) and recently QA (Adlakha et al., 2024). Chen et al. (2024b) investigates the robustness of contextual LLMs under noisy retrieval, with a primary focus on news articles. Another line of research focuses on hallucination detection (Liu et al., 2021; Li et al., 2023b; Hu et al., 2024), which aims to detect hallucinated outputs. The task focuses on the model output instead of the context (input). Additional efforts have been made to create QA benchmarks based on common misconceptions (Lin et al., 2022) or questions that are unanswerable by nature (Yin et al., 2023). However, none of these datasets are contextual. In contrast, each question in FaithEval is accompanied by multi-paragraph context, mimicking RAG scenarios with long and noisy contexts.
Adversarial context generation. Generating challenging or adversarial contexts for language models has been explored in various scenarios. One line of research focuses on context modification. Shi et al. (2023) propose template-based framework that adds irrelevant facts to the context and studies the effectiveness of different prompting techniques. Yu et al. (2024) leverage L.MSs to perturb original evidence, potentially altering the answers, while Manakul al. (2023) utilize LLMs to generate purely synthetic contexts that support given statements. To study the impact of knowledge conflicts, Wu et al. (2024a); Xie al. (2024) use LLMS to create adversarial contexts that conflict with the models' internal knowledge, improving coherence compared to previous word-level editing methods (Longpre et al., 2021; Chen et al., 2022: Zhou et al., 2023). Pan et al. (2023) studies the impact of LLM- generated misinformation. Another research direction involves modifying the question. Ramakrishna et al. (2023) generate invalid (unanswerable) questions, while Huang et al. (2024) build a small-scale dataset with 209 questions containing adversarial facts or incorrect information based diverse templates. In contrast, we introduce the first iinegrrined, larger-scale scale (4.9K) high--ualit, contextual QA benchmark featuring multi-paragrapy coherent contexts across three diverse tasks.
3 FAITHEVAL BENCHMARK
3.1 TASK OVERVIEW
To systematically evaluate the contextual faithfulness of L.LMs, FaithEval contains three diverse tasks including unanswerable context, inconsistent context, and counterfactual context. Each sample (c,g.a) 1.g.o. a) consists of a question q, and a long context passage made up of one or more documents (dg, ..... d,). and groundtruth answer a. The model is expected to answer the question leveraging the information in the provided context. An overview of each task is presented in Figure 2. Next, we illustrate the construction of each task in detail.
Unanswerable Context, An unanswerable context arises when the context includes relevant details but lacks the information needed to answer the question. In EailhEval, answerability is determined
3

---

Published as a conference paper at ICLR 2025
Unanswerable Context
Inconsistent Context
Counterfactual Context
In 2009, 78.5% of Dallas commuters drive to work alone.
[Doc 1] Life of Pi is Canadian fantasy adventure novel...with Bengal tiger named Richard Parker...
...One intriguing property of wood that often been overlooked its magnetic nature... Theee findings pointed to the presence of iron like compounds within the cellular structure of wood, which could exhibit magnetic properties... earrl shipbuilders used magnetized wood...
In 2015, the American Community Survey estimated 12.8% for carpooling, 3.5% for riding transit...
[Doc 2] He. endures 227 days stranded on lifeboat a.companiied by a Bengal tiger named William Shakespeare..
Question:
Question:
Question:
Which group of commuters in Dallas in 2009 is larger: carpooling or transif?
What is the tiger's name in Life of Pi?
Which statement explains why tree branch floats water? [four options)
Richard Parker
Carpooling
Wood is buoyant
Inconsistent
Unknown
(muitiple answers)
Wood is magnetic
Demonstration of each task in FaithEval. Left: in Unanswerable Context, the context does not contain the answer to the question. Middle: in Inconsistent Context, multiple answers are supported by different documents. Right: in Counterfactual Context, the context contains counterfactual statements that contradict common sense or world knowledge. Complete contexts can be seen in Appendix E.
Figure 2:
solely by the context, regardless of whether the question itself is unanswerable. For instance, in the example in Figure (Left), the context provides the proportion both types of commuters in 2015. However, the question "Which group of commuters in Dallas in 2009 is larger: carpooling transit?" is unanswerable, as the context lacks specific data from 2009. To create such task, we modify the context from collection of 10 contextual QA datasets, covering a wide range of domains (see Source datasets below). For each sample, we prompt an I.LM to modify the original context so that no longer contains the supporting evidence for the ground truth answer. Additional sentences may be woven into the new context to maintain coherence. The full prompt is shown in Figure 17. This process resulted in total of 2.4k contextual QA pairs. To verify the quality of the modified contexts, we achieved over 98% agreement with professional human annotators (Sec 3.2).
Inconsistent Context. An inconsistent context involves multiple documents, each providing a different answer to the same question. This simulates noisy retrieval scenarios, where documents from sources with varying levels of credibility are retrieved. For instance, as shown in Figure (Middle), the context presents conflicting information about the tiger's name in the novel Life of Pi. A faithful model should be able to identify such inconsistencies, especially when instructed to do so. To create this task, we modify contexts from the same collection of contextual QA datasets used in the Unanswerable Context task. For each sample, the LLM is provided with context passage, a question, and original answer, which is supported by the context. The goal is to modify the context that it introduces fabricated supporting evidence for a new, conflicting answer. The detailed prompt is shown in Figure 18. Since this task is more challenging, curated collection of 1.5k high-quality contextual QA pairs after filtering through professional human annotators.
Counterfactual Context. counterfactual context contains statements that contradict with common sense or widely accepted facts, such as "water freeze at 100 degrees Celsius", "wood is magnetic", or "carbon dioxide is the most abundant greenhouse gas in the atmosphere". Unlike the other two tasks, the questions in this task are required to be relevant to such known facts. We curate this task based on ARC- ChC-Chalen (Clark et al., 2018). a QA dataset covering grade-school level, multiple- choice science questions. Since the original dataset does not include context, we prompt an L.MM to generate long, marti-pa context that seamlessly provides fabricated supporting evidence for counterfactual answer. The detailed prompt is shown in Figure 19. This process resulted in total of Ik contextual QA pairs, each with three to five options. Due to the multiple-choice nature, we can use keyword matching verify the quality of the synthetic contexts (Appendix B).
Source datasets. We curate new contexts based on a diverse collection of contextual QA datasets, including SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017), NaturalQuestions (Kwiatkowski et al., 2019a), SearchQA (Dunn et al., 2017), HotpotQA (Yang al., 2018) 1), BioASQ (Tsatsaronii al., 2015), DROP (Dua et al., 2019), RACE (Lai et al., 2017),
4

---

Published as a conference paper at ICLR 2025
Context Generation
Task Construction
Auto Eval by LLM Judge Q1: Is ans valid given context (if?
Source: Contextual Dataset
Unenswerable
Context with removed evidencee
unique given fonly-
QA pair
Orginal
LLM
Modified
contect
Context with mulliple answers
Is ans valid unique
Human Annotation
Final Version
Source: Non-Contextual Dataset
pair
Counterfactual
majority
LUM
Synthetic
counterfactual evidence
Contextual QA Dataset
context
Figure 3: Illustration of task construction and validation framework. (1) Context Generation: given source QA dataset, we prompt an LLM to generate a new context based on a question, the original answer, and optionally the original context. (2) Task Construction: we construct the prompt for each sample by combining the original question, the new context, and task-specific instructions. (3) Auto Eval by LLM Judge: validate the quality of the context by checking if and only-if if the new answer is supported by the new context. (4) Human Annotation: we further filter out invalid contextual QA pairs based on the majority vote results from professional annotators.
TextbookQA (Kembhavi et al., 2017). We adopt the test splits from Fisch et al. (2019). Due to variations in human annotations, the final collection consists of 150 samples per dataset for the Inconsistent Context task and around 240 samples per dataset for the Unanswerable Context task.
3.2 TASK CONSTRUCTION AND VALIDATION FRAMEWORK
Task construction. An overview of our task construction and validation framework is shown in Figure 3. Given a source QA sample and an original context (optional), we prompt an LLM to generate both a new context and new context (that supports no answer) for the Unanswerable task. To make the tasks challenging, the new context should be coherent and contain minimal modifications if the original context is provided. In addition, multiple paragraphs not directly related to the answer are included, serving as distractors. The new context generated with detailed justifications explaining how it satisfies the task criterion. We construct the prompt for each sample by combining the original question, the new context, and task-specific instructions (See 3.3). In particular, an inconsistent context is created by concatenating the new context with the original context, each supporting different answer.
answer for the Counterfactual and Inconsistent tasks, only
Auto validation and human annotation. We validate the quality the new context by using a separate LLM judge verify whether the new answer is valid given the context ("if" condition) and whether the context does not support alternative answers ("only-if" condition). For example, the new context in Figure (Right) should not mention wood is buoyant. Samples that fail to meet both conditions are filtered out. Next, we perform meticulous human annotation. Depending on the task's validation difficulty, we employ different strategies. As the Inconsistent Context task challenging to validate, we rely on full human annotation. Three Mechanical Turk (Crowston, 2012) workers judge whether each contextual QA pair meets the "if" and "only-if" conditions, with final inclusion determined by majority agreement. This yields 1.5K samples. For the Unanswerable Context task, which is easier validate, we use similar majority-vote approach. achieving over 98% agreement among human annotators. This yields 2.4K samples. For the Counterfactual Context task, since the answer options are provided with the context, we validate using string-based matching method, where the context passes if all words from the answer appear in the context. This decision was based on our pilot human studies that showed nearly perfect agreement with the string-macching method on generated contexts based on ARC-Challenge. The filtering results in 1K samples. After filtering, the FaithEval benchmark contains a total of 4.9K high-quality contextual QA pairs. More details are included in Appendix A.
3.3 EVALUATION
Models. We evaluate a wide range of competitive open sourced and proprictary language models with different scales, including the most recent releases up to Sep 10, 2024. Our initial experiments suggest
5

---

Published as a conference paper at ICLR 2025
that instruction tuned (chat) models significantly outperform base models. Therefore, consider 18 competitive chat models, including 1-t3-mmni--22kiinseee (3.8B), Phi-3-mediume 128k-instruct (14B), Phi-3.5 1ww.5---miiiiii instrrctt (3.8B) (Abdin et al., 2024), LLaMA-3- --ntttuutt,, Instruct, LLaMA-3.1-8B Instruct, L.LMA-3-70BB Instrt, LLaMA-3. 1-70B-Instruut (Llama, 2024), Chstral-BBInstrucct Sharral-eemo--nstuut--2222 (12B) (Jiang et al., 2023), Gemma-2-9B-4 and Gemma-2-27B-Bi (Team, 2024). For proprietary models, we consider Open Al's GPT-3.5 Turbo, GPT-4o-mini, GPT-4o, GPT-4 Turbo, Cohere's Command R (35B), Command R+ (I04B), and Anthropic's ' Claude 3.5 Sonnet.
Default Evaluation Scheme. For all tasks, we append the following prompt to each question: You are an expert in retrieval-bate basse question answering. Please respond with the exact answer, using only the information provided in the context. For the Unanswerable Context task, we append an additional instruction: If there no information available from the context, the answer should "unknown". Similarly, for the Inconsistent Context task, the instruction is: If there is conflicting information or multiple answers in the context, the answer should be "conflict". Note that for Counterfactual Context, we do not add additional task instructions. Our primary evaluation metric across all tasks accuracy (ACC), where a model's s response is considered correct if it mentions the ground truth answer. All models (temperature = O). We report both strict-matching (S) ACC, which considers only a single ground truth answer (e.g., "unknown"), and non-strict matching (N) ACC, which allows broader range of semantically similar phrases. A detailed list of valid phrases is provided in Appendix A.
evaluated using their default configurations with deterministic decoding
Alternative Evaluation Schemes. We study alternative evaluation strategies in Section 5. Specifically, we examine non-deterministic decoding with temperature scaling (t= 0.3, top-p = 0.9). Additionally, we investigate the impact of chain-of-thougl! (CoT) prompting (Wei et al., 2022; Kojima al., 2022) using the following instruction: Given the context, first provide brief answer to the question. Then, explain your reasoning step by step, detailing how you arrived at the answer.
MAIN RESULTS
Performance Unanswerable
80
Or ginal Connext Una (wandwenabe. Context 71A
713
285
355
60
53
Nistral hee Insrrcct2000
Uamo 700 irserct
Laan a caa sttract
Pro mimis 2Bk instroct
Nistaa insorutto0..3
2-2-t
Lana 3.3. 708 inseruct
Phi Persemmeeh semeedumm instracct
gat g S-tuuboo
Figure 4: Model performance comparison on the Unanswerable Context task, where no evidence supports the answer. Columns sorted by performance the Original task (original context). Proprietary model names are highlighted in orange.
4.1 UNANSWERABLE CONTEXT
Abstaining is challenging, even when explicitly instructed. The results of the Unanswerable Context task are summarized Figure ranked by performance on the original context. Proprietary model names are highlighted in orange. We highlight the following key observations: (1) Modern LLMs experience significant performance degradation in this task. Across all chat models, the the original context does
performance gap ranges from 13.6% to 68. 4%.. (2) High performance not correlate with high performance on the unanswerable context. For example, while Phi-3-medium- 128k-instrus achieves 75.8% accuracy the original context, closely upproaching the SoTA (80.1%), struggles to abstain from answering in the unanswerable context, with an accuracy of only 7.4%. advantageous within the same model family. For instance, compared
(3) Larger model sizes

---

Published as a conference paper at ICLR 2025
Model Performance Inconalstent Context
Origiall Content Incamestent
80
65.
652
60
20
120k Instuut
S-mininnttut
Uaa 3-38 -nntuuu
Lame 3.2 3-38 inttrrtccct
PHE medium 120 iiu--uu
got-o mini
Oet turbo
gen 23-22--2
Lime 3.1. 1:708 Instouct
mostrl 7B reeeerttrucc
Commani
Commanc
Lana 708
Classe
Figure 5: Model performance comparison the Inconsistent Context task. Columns are sorted by the performance on the original task. Proprietary models are colored in orange.
the 7B model, 1. 1-70---nntttt improves performance on the Unanswerable Context task by 10.3%. Similar trends hold for the Gemma-2 and Llama-3 model families.
4.2 INCONSISTENT CONTEXT
Performance varies significantly on inconsistent context across model families. The model performance the Inconsistent Context task is summarized in Figure 5. We have the following key observations: (1) Performance varies substantially across different model families. For instance, the Phi-3 series struggles to identify multiple answers detect inconsistencies (conflicts), with average accuracy of only 5.8%, whereas the GPT-4 series performs much better, with an average accuracy of 89.35%. (2) Open-source source models lag behind proprietary models. Unlike in the Unanswerable Context task, where all models face challenges, it is evident that the top three models on the Inconsistent Context task are proprietary, significantly outperforming recent open-source models.
Model Porformance on Countorfactual Context
No Context
Counterfactual Cortest
122
55.
057
80
$21
60
125
03.
Hetre 11-tttttcctt
Mastral Instrnetttt 22407
get- S-turbo
Command
gemma 2-0eet
gemine-2-2755 2-270-1
cpt do miri
miid Penid-nnnt--e 2286 strues
memetmmmm instrree
Lama- 3-708 struct
Coude Sorroc
0p-4o
Command
Liama-3
Lama
Figure Model performance comparison on Counterfactual Context, which contains evidence
supporting counterfactual answer. Proprietary models are colored in orange.
4.3 COUNTERFACTUAL CONTEXT
Faithfulness remains a limitation for contextual LLMs. The results on the Counterfactual Context task are shown in Figure The blue bars represent model performance under the closed-bood book QA setting, where no context is provided. In this case, the models rely entirely their parametric knowledge of common facts. We observe that nearly half of the models achieve over 90% accuracy, with GPT-4o nearing perfect performance 96.3%. However, when evidence that contradicts the model's 's parametric knowledge is introduced, performance declines sharply. For example, GPT-4o achieves only 47.5% accuracy the Counterfactual Context task, despite our human study indicating that the correct answer can be easily derived from the provided context (95% accuracy held-out subset). This highlights significant gap in faithfulness. the
context with counterfactual
7

---

Published as a conference paper at ICLR 2025
Lime-31---mmmmnt
gamma-2 2-0b-t
665
BaA03
86460
36400
0107
0008
NE1p0T04
Heproxx
R2
MQ
Men
Newd24
Rewd04
50640
6658
20
Drigkv Corrut
EVor Control
Oiirel Conteet
Ledrcaerite Eontest
Corite
Uistowerrte Certent
Llana- teotruct
gemma-2-doo 2-db-l
06400
36460
0100
000P
DRCP
Matpo/04
wwppox
R0
10
MQ
MenSON
Nowd24
Nowe04
1A4
50u40
Orggial Centent
INCONISIIEE COFTEN
Original Cantest
BiceRsSteeE ConTent
Origiial Contert
Iccessteer Contost
Figure Performance decomposition on individual datasets for Unanswerable Context (top row) and Inconsistent Context (bottom row). Full results for all models can be seen in Appendix C.
ability to generate outputs that align with the provided context context-between current state-of-the-art models and human-level performance.
DISCUSSIONS AND FURTHER ANALYSIS
closer look at Unanswerable and Inconsistent Contexts. We present the performance breakdown for each of the ten individual datasets in Figure for the Unanswerable (top row) and Inconsistent Context (bottom row) tasks. We include three representative smaller-scale models: LLama-3. 1.na-3.1-BB. 1-8B- Instruct,
and Gemma-2-90 2-b-ii- Full results for other models are provided in Appendix C. We observe the following: (1) While smaller models demonstrate competitive performance the original datasets, none able maintain this performance introduced contexts. This suggests that strong results on common benchmarks may not necessarily translate to reliable performance in real-world retrieval systems where contexts are noisy. (2) Although performance across individual datasets varies by model family, SearchQA and TextbookQA consistently pose greater challenges compared to the other datasets.
the newly
Model Performance Original Context New Context
3L1
80
25.9
Command
Lama 88 inctruct
Uamens 433 inttrrtt
Phi medeumm instrucc OET
Command
QEt-3 S-turbo
gpe- Coubto
gemma-2-2220 375-4
Uama 3.3 708 Instrcct
Matral instruct
gomma
ggt-
Lsmo 708
Figure 8: Model performance comparison on the Original Context v.s. New Context for Inconsistent
Context task. Proprietary models colored in orange.
8

---

Published as a conference paper at ICLR 2025
Task-spoc. Inst.
BloASQ
DROP
HotporQA
NQ
NewsQA
RACE
SQUAD
StarchQA
TexthookQA
TrivinQ4
AVG
Claude 3.5 Sonnet
X (original prompt)
0.90
0.94
0.89
0,86
0.81
0.84
0.93
0.97
0.7
0.86
0.90
(4comflict prompt)
0,84]
0.86]
0.78)
0.77p
0.14
05K
0.86
0.85.
GPT-40
X (otigimal prompt)
0.84
0.81
0.89
0.80
0.78
0.79
0.95
0.90
0.93
0.87
0.85
(+condlict prompt
0,77
0.83
0.85L
0.7%)
0.73,
0.81
0.93
0.87
0.14
0.86
0.83.
Table I:
Impact of task-specific instructions on the normal (original) context. Having the additional task instruction degrades the performance on normal contexts consistently.
closer look at Inconsistent Context. Since an inconsistent context created by concatenating the original and new contexts, we separately evaluate the model's 's performance on the original context and the new context. The results, shown Figure 8, reveal that while models struggle when both context passages are presented together (Figure 5), most models do not find the new context more challenging than the original when it is presented alone. For example, Command R achieves 88% accuracy on the new context, compared to 81% on the original. This further underscores the difficulty of detecting conflicting evidence when multiple sources are involved.
Sycophancy with task-specifie instructions. While the additional instructions used in the Unanswer- able and Inconsistent Context tasks (Sec 3.3) improve the model's 's awareness of such scenarios, they can also introduce unintended effects when the context is normal (i.e., answerable and consistent). This can lead to what is known as sycophantic behavior (Perez et al., 2023; Wei et al., 2023), where models adjust their responses align with the user's expectations, even when those expectations are objectively incorrect. We examine this phenomenon in two rop-performing models for the In- consistent Context task, GPT-4o and Claude 3.5 Sonnet. Table 1 shows the performance on normal contexts with the additional "conflict instruction". We observe consistent performance drop for both models, with Claude 3.5 experiencing a 5% decrease in average accuracy. This further highlights the challenge of maintaining faithfulness across both normal and noisy contexts.
Unanswerable Context
Inconsistent Context
70
Derect And CoT
100
CoT
Direct Ans
23,
21.
60
50
40
40
20
13
10
mostum inssuct
mii 1e..----u mi Sen-S---ieiatnnn instrue
Misroa 7B 3.-nztructo00.3
Unmead 5-05 anstrust
Command tha
gemmmn 2-00-
phi Sww.stat 2210- insturt
Phi- 1o mini instruct
Masrral wtw.-nttttccee.....
Unme- 3 H-t enstruut
Command
(a) Direct Ans CoT Unanswerable Context
(b) Direct Ans vs. CoT on Inconsistent Context
Figure 9: The impact of CoT prompting on Unanswerable (Left) and Inconsistent Contexts (Right). Due to space constraints., we include representative models from different model families.
Does thain-of-thought prompting improve faithfulness? Popular prompting techniques, such as CoT, have shown promising performance on various tasks that require multi-step stelii-seeppop reasoning. We adopt the prompt format in Section 3.3 and summarize the results in Figure 9. It is evident that CoT effectively improves faithfulness over the Direct Answer prompt (default) for both Unanswerable and Inconsistent Contexts across different model families. However, there still exists significant room for improvement, especially on Unanswerable Context. For instance, the leading model achieves only 71.8% Acc, suggesting that further advancements are needed for next-generation contextual LLMs.
Strict vs. non-strict matching. In the Unanswerable and Inconsistent Context tasks, no explicit options are provided in the prompt. As result, LLMs may express concepts such as "unknown" or "inconsistent" in varying ways. To assess the impact of allowing alternative valid expressions, compare the performance of strict and non-strict matching. Striet matching only accepts the
9

---

Published as a conference paper at ICLR 2025
Strict Match Non-Strict Match Unanswerable Context
60
547
50356.1
40
STA
53
The Th.Some 120h iaboe
Laara RobDUc.
Liaro 1-03 Intuus
Command
gememaa 2-06-
Ry mrnddddd
Usno
Figure 10: Impact of strict vs. non-strict matching on Unanswerable Context. Non-strict matching allows for a wider range of phrases that express the idea of "unknown". The comparison Inconsistent Context is shown in Appendix C.
exact phrases "unknown" or "conflict" specified in the prompt, while non-strict matching permits broader range of expressions that convey similar ideas (see Appendix A for the full list of valid expressions). We observe that performance remains stable across most models. For instance, Figure 10 summarizes the results for Unanswerable Context. For competitive models such as gpt-4o and Claude 3.5, the gaps are less than %. Full results can be found in Table 4 and Table Appendix
Impact of decoding strategies. By default, we adopt greedy decoding. We also investigate popular sampling-basee -aaeed decoding scheme with a temperature of 0.3 and top-p 0.9. The results are shown in Figure/ 11 based on Counterfactual Context. Similar observations also hold for Unanswerable and Inconsistent Context. We observe that sampling-base
decoding marginally improves the performance over greedy decoding across all models. However, the significant gap between the original and counterfactual contexts cannot be mitigated with temperature scaling.
Non-Sampling Sampling Based Decoding
800
Non sampling Sempling
D.
70
60
YIL
MUS
50
gpt gpt-4-turo
germee 4b-t
gemmo 2:27b
spte Do-----tuttoo turbo
Metral neme Instruuc 2400
Uneme 3-700- Instrrcc
Pi Instruct
Usemed 4it tratreet
Command
Mistal inntnnn 00
Command
medium
Figure 11: The impact decoding strategy. The plot displays greedy (non (non-sampling) vs. sampling- based decoding (t=0.3, top-p=0.9) Counterfactual Context.
CONCLUSION
In this work, we propose FaithEval, novel and challenging benchmark designed assess the faith- fulness of contextual LLMs. FaithEval comprises 4.9K high- high-quulity contextual problems spanning multiple domains and includes three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. To build this benchmark, we propose a scalable multi-stage context construction and valida- tion framework, incorporating both automated evaluation by an LLM judge and human validation. This approach enables the creation of multi-paragraph coherent contexts satisfying diverse criteria. We provide a timely and in-depth study on a wide range of open- and proprietary models, re- vealing that even the most competitive LLMs often struggle to remain faithful to the contexts, despite excelling on standard benchmarks. We hope our work will contribute to more holistic evaluations of contextual L.MMs and inspire further advancements in developing faithful LLMss.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024.
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. Evalu- ating correctness and faithfulness of instruction-following models for question answering. Trans- actions of the Association for Computational Linguistics, pp. 681-699, 2024.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through reff-reflection. arXiv preprint arXiv:2310.11555 2023.
Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simon- sen, and Isabelle Augenstein. Faithfulness tests for natural language explanations. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 283 294, July 2023.
Michael Bommarito and Daniel M. Katz. Gpt takes the bar exam: What artificial intelligence and machine learning mean for the practice of law. arXiv preprint arXiv:2212 /4402, 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, lean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring, T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, 2021.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2303. 12712, 2023.
Hung-Ting Chen, Michael IQ Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality. multi-granularity text embeddings through self-knowledge distillation. In Annual Meeting of the Association for Computational Linguistics. 2024a.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in aarieval-augmenterr generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pp. 17754-17762, 2024b.
Xiang Chen, Duanzheng Song, Honghao Gui, Chenxi Wang, Ningyu Zhang, Yong Jiang, Fei Huang, Chengfei Lyu. Dan Zhang, and Huajun Chen. Factchd: Benchmarking fact-conflicting hallucination detection. In Proceedings of the Thirty-Thir -hird International Joint Conference on Arlificial Intelligence, IJCAI-24, pp. 6216-6224, 8 2024c.
I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. Factool: Factuality detection in generative ai-a tool augmented framework for multi-task and multi-domain scenarios. arXiv preprint arXiv:2307.13522 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1r03v:1803.0544 2018
Kevin Crowston. Amazon mechanical turk: A research tool for organizations and information systems scholars. In Shaping the Future of ICT Research, 2012.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2368-2378. 2019.
11

---

Published as a conference paper at ICLR 2025
Matthew Dunn, Levent Sagun, Mickael Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqu: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179, 2017.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop Machine Reading for Question Answering, pp. 1-13, November 2019.
Yifei Gao, Lei Wang, Jun Fang, Longhua Hu, and Jun Cheng. Empower your model with longer and better context comprehension, 2023.
Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. Refchecker: Reference-basee fine- grained hallucination checker and benchmark for large language models. arXiv preprint arXiv:2405. 4486, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang. Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arX1v22311.05232, 2023a.
Rongjie Huang, Jia-Bin Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiaoyue Yin, and Zhou Zhao. Make-an- audiio: Text-to-audio generation with prompt-enhanced diffusion models. ArXiv, abs/2301. 12661, 2023b.
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang. Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang. Bertie Vidgen. Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing. Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitmik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang. Joaquin Vanschoren, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pun-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying. Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Yang Wang, Xiang Li, Xiangliang Zhang. Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang. Yan Liu, Yanfang Ye, Yinzhi Cao. Yong Chen, and Yue Zhao Trustllm: Trustworthiness in large language models. In Forty-first International Conference on Machine Learning, 2024.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. ArXiv, abs/2007. 2822, 2020.
Qi Jia, Siyu Ren, Yizhu Liu, and Kenny Zhu. Zero-shot faithfulness evaluation for text summarization with foundation language model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11017-11031, December 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna I Lengyel, Guillaume Lample, I Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601- 1611, 2017.
Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Bridging the preference gap between retrievers and llms. arXiv preprint arXiv:2401.0688 2024.
Aniruddha Kembhavi, Matteo Salvato, Minjoon Seo, Harish Kannan, Aniruddha Singh, Li Fei-Fei, Ali Farhadi, and Mark Yatskar. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4999-5007, 2017.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22 2022.
12

---

Published as a conference paper at ICLR 2025
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019a.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wa Mei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019b.
Philippe Laban, Wojciech Kryscinski. Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9662-9676, December 2023.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, September 2017.
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35:34586-34590 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval -uggeente generation for knowledge- intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. Making large language models a better foundation for dense retrieval. ArXiv, abs/2312. 2023a.
Jiqqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. LooGLE: Can long- language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16304-16332 August 2024a. doi: 1211653/1122022.cc-ooom 859.
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6449-6464, December 2023b.
Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. The dawn after the dark: An empirical study on factuality hallucination in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10879-10899, August 2024b.
Siheng Li, Cheng Yang, Taiqiang Wu, Chufan Shi, Yuji Zhang, Xinyu Zhu, Zesen Cheng, Deng Cai, Mo Yu, Lemao Liu, et al. A survey on the honesty of large language models. arXiv preprint arXiv:2409.1878 18786. 2024c.
Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing.
Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In The Twelfth International Conference on Learning Representations, 2024d.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 3214-3252, May 2022.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. A token-level reference-frr -frreneeeefeeeeeeeee hallucination detection benchmark for free-form text generation. arXiv preprint arXiv: 104.08704 2021.
13

---

Published as a conference paper at ICLR 2025
Llama. The Ilama 3 herd of models, 2024.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity- based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7052-7063, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box halluci- nation detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, Singapore, December 2023.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr- text retrieval with transfer learning. Salesforce Al Research Blog, 2024. URL wwtphttt///// /ttp:s:////ogg /wwpg./iog////////////.ooo sales forceeooreeee http.//.ooo......ooraeaaaaaaaaaa research. com/ sfr- embedded mistral/.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 12000, December 2023.
Xuan-Phi Nguyen, Shrey Pandit, Senthil Purushwalkam, Austin Xu, Hailin Chen, Yifei Ming, Zixuan Ke, Silvio Savarese, Caiming Xong, and Shafiq Joty. Sfr-rag: Towards contextually faithful Ilms. arXiv preprint arXiv:2409.099) 09916 2024.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-HALT: Medical do- main hallucination test for large language models. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pp. 314-334, December 2023.
Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min- Yen Kan, and William Yang Wang. On the risk of misinformation pollution with large language models. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mereado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodde Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model- written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, July 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language P Processing, pp. 2383-2392, November 2016.
Anil Ramakrishna, Rahul Gupta, Jens Lehmann, and Morteza Ziyadi. Invite: a testbed of automatically generated invalid questions to evaluate large language models for hallucinations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
2023.
Rita Parada Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. Smallcap: Lightweight image captioning prompted with retrieval augmentation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2840-2849, 2022.
Sara Sarto, Marcella Cornaa, Lorenzo Baraldi. and Rita Cucchiara. aatrieval-augmente trans- former for image captioning. Proceedings of the 19th International Conference on Content-butt based Multimedia Indexing, 2022.
14

---

Published as a conference paper at ICLR 2025
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference Machine Learning, pp- 31210-31222 PMLR, 2023.
Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty. Discovering the gems in early layers: Accelerating long-context Ims with 1000x input token reduction. arXiv preprint arXiv:2409. 17422, 2024.
Noah Siegel, Oana-Maria Camburu, Nicolas Heess, and Maria Perez-Ortiz. The probabilities also matter: A more faithful metric for faithfulness of free-text explanations in large language models. In Proceedings of the 62nd Anmual Meering of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 530-546, August 2024.
Mingyang Song, Mao Zheng, and Xuan Luo. Counting-stary A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models, 2024.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. pp. 191-200, 2017.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, loannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenbom, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16: 16:1-28, 2015.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2443777444837, 2022.
Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoe V Le. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.0339 2023.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, al. Long-form factuality in large language models. arXiv preprint arXiv:2403.1 18802, 2024.
Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an Ilm's internal prior and external evidence. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a.
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. A survey on large language models for recommendation. World Wide Web, (765):60, 2024b.
Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Certifiably robust rag against retrieval corruption. arXiv preprint arXiv:2405. 15556, 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations, 2024.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving reurened LMs with context compression and selective augmentation. In The Twelfth International Conference on Learning Representationss, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018
Hongbin Ye, Tong Liu. Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint X09:22309.067944, 2023.
15

---

Published as a conference paper at ICLR 2025
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023, pp. (653-8665, July 2023.
Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, and Jianfeng Gao. Reeval: Automatic hallucina- tion evaluation for retrieval-auu -arreval-augmenteed large language models via transferable adversarial attacks. In NAACL-Findings, 2024.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang. Wentao Zhang, Jie Jiang, and Bin Cui. Retrieval-augmenter generation for ai-generated content: A survey, 2024a.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yul Yun Nie, and Ji-Rong Wen. survey of large language models. arXiv preprint arXiv:2303.182. 18223, 2024b.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithhun prompting for large language models. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp- 14544-14556, December 2023.
16

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENT DETAILS
Context generation model. By default, we use the latest GPT-4o (gpt-40-2024-05-13) as the context generator. In our preliminary studies, we evaluated various alternatives, including GPT-4o- mini, GPT-4-turbo, and 1..MA-3.11-000 GPT-4o demonstrated superior performance terms of context coherence, validity, and complexity.
Context evaluation model. We use gpt-4o-mini as the judge model for context verification. We selected gpt-4o-mini for its faster inference speed, lower cost, and high judgment quality, with our preliminary study showing an agreement rate of over 95% when compared to GPT-4o as the judge.
Valid phrases for non-strict matching. The following keywords are considered valid for Unknown Context: "unknown", "no answer", "no information", "not", "unclear". For Inconsistent Context, the valid phrases include: "conflict", "conflicting", "disagreement", "inconsistent", "contradictory", "contradiction", "inconsistency" : "two answers". "2 answers", "multiple answers". ". These keywords were selected based an analysis of output patterns from open-source and proprictary models.
Details on Counterfactual Context construction. The ARC-Challenge dataset provides grade- school level multiple-choice science questions regarding widely accepted facts, where each question has exactly correct (groundtruth) answer. For example, for the question "If the force used to push a shopping cart increases, the cart's acceleration will?" with options A.decrease, B. increase, C. remain the same, physics principles dictate that B. increase is the groundtruth answer. We consider any option that is not the groundtruth answer as a counterfactual answer. What makes ARC- Challenge particularly suitable construct Counterfactual Context that its questions test universal scientific principles that have clear, context-independen answers. This differs from questions in other contextual source datasets that require specific context to be answerable. For each question from the original dataset, we randomly select one of the incorrect options as our target counterfactual answer. Then we construct context that provides supporting evidence for this selected answer (Section 3.1). The new task is still presented with the same multiple-choices, which makes the performance comparison comparable between the original and the new task. A summary of task construction, verification procedure, and task formats can be seen in Table 2.
Task
Task Format
Task Verification
Source Dataset
Source Dataset Format
Counterfactual Context
Multiple Choices
LLM judge +string matching
ARC-Challenge contextual datasets
Multiple Choices Open-endar QA
Unanswerable Context
Open QA
L.M judge +human annotation
Inconsistent Context
Open-ended QA
LLM judge +human annotation
10 contextual datasets
Open-ended QA
Table 2: Summary of task construction, verification procedure, and task formats.
Model sizes. In this work, we evaluate on 18 competitive open-sourced and proprictary models. We summarize the model sizes from different model families in Table 3.
Human annotation. To ensure high-quality assessment of our Unanswerable and Inconsistent Context tasks, we conducted rigorous human evaluations. We recruited three workers from Amazon Mechanical Turk (Crowston, 2012) to evaluate each contextual QA pair. The workers assessed whether the pairs satisfied both the "if" and "only-if" conditions, and we included pairs in our dataset only when a majority workers agreed on their assessment (Section 3.2). For further quality control, implemented several measures such monitored completion time ensure that each contexts have been reviewed carefully. Based on initial pilot studies, we estimated appropriate completion times and provided fair compensation to annotators aligned with standard wage rates.
B VERIFICATION FOR COUNTERFACTUAL CONTEXT
For the Counterfactual Context task. since the answer options are provided within the context, we can validate the new context using simple keyword-based matching method, where context passes if
17

---

Published as a conference paper at ICLR 2025
Model Name
Model Size
Phi-3 Family (Abdin et al., 2024)
Phi-3-mini- 1en-3-mnni-122k-nnstie inst-rttttect
3.8B
Phi-3-medium- 128k-instruct
14B
Thi-3.5-miniiinstruee
3.8B
LLaMA-3 Family (Llama,
2024)
LLLMMA-3--B-instuee
8B
LLaMA-3.1-88 1-8--iinttttctt
8B
-LaMM-33-70B-nssmie
70B
70B
Mistral Family (Jiang al., 2023)
D.atral-7B-isttuut--0....33
7B
Histral-Nemo-instrurk
12B
Gemma-2 Family (Team,
2024)
Gemma-2 2-99-tt
9B
Gemma-2-27B-it
27B
OpenAl
GPT-3.5 Turho GPT-4o-mini
unknown unknown unknown unknown
GPT-4o
GPT-4 Turbo
Cohere
Command R Command R+
35B 104B
Anthropic
Claude 3,5 Sonnet
unknown
Table 3: Model sizes across different model families.
the answer phrase exists in the context. This results in a pass rate of 68 9%,, where the new context clearly contains the new answer. The results this filtered subset are shown in Figure 12. We can see that the same trend still holds as shown Section performance the original task (with context) and the new task with counterfactual contexts, across the majority of instruction-tuneg models.
significant gap remains between the
Performance Counterfactual
100
Counterfactual Cortest
32A
50.
80
362
754
60
557
223
40
Nostre 20 -it---t--ttttooo
Mastral Nemr wet.---atteeeeeeeeee 2000
Ph mis instrcct
Command
gemmazt.boo 2000
cpt 40 min
mhi instrrcct
Lian Lamac 708 snstact
Cooue Sornot
Spt-44 Bi--6-turbo
0yt-G
Laaaaa 68
Usira
Figure 12: Model performance comparison the clean subset of Counterfactual Context. The same trend still holds as in Section
C FULL EXPERIMENT RESULTS
The results break down for all models on each of the ten datasets are summarized in Figure 13 and Figure 14 for Unanswerable Context: Figure 15 and Figure 16 for Inconsistent Context. The detailed results for strict vs. non-strict matching can be found in Table 4 and Table 5.
18

---

Published as a conference paper at ICLR 2025
Unme-31148 1aw.3-1-----otteee Instrnct
Uama-k-8R
AG
810
DaA03
94660
0401
060P
0600
NE1p300
1Mp004
MMp4044
NJ
Tx4004
2B
40
90
Orignnal Cansol
Unaronn oiee Contarl
Orgaul Deraat
Unutono Carter
Orggell Corter.
Unatowresle Conted
Command R
Command Ri
M65
465
26400
26400
040
[WCP
09CP
HEtp9TON
Horpot04
HMpa024
N2
10
M0
kewdp4
Rrwd04
50J40
90
20
Grigiv Corrut
Poone www.ea Carteol
0riginl Conteet
Leancancaia Enetrot
Orgg Fartte
LinaTowratle Certee
gumme---377- 2-37b-
got-3.5-tarbb
866
26450
[W0P
DRCP
HMpa04
N0
10
NenSOA
Rew026
Newd04
50.40
Sc
40
Dridiw Correr
Originl Conteat
Leanncerrtle Fontoot
Onig Fartte
Liastowvazte Cortee
Figure 13: Performance decomposition on individual datasets for Unanswerable Context (part I).
Medel
BivASQ
BROP
49
NreeOA
RACE
SQHAD
SuidNQA
ToshwekKA
THHRQA
AYYG
Iommet
0.508
01513
0.54
0.05
0.58
0353
0.56
0556
0:12
0.50)
0.966
D-18
3:58
3:53
3:13
0.34
0.144
0.445
0.455
01453
014%)
Llana 1-110-3.--0t--aaa -n-t-attaitd
0.42
0.463
0.56
0.54
0.53
0632
0.48
0.30
0,30
0.34
8466
8434
9511
2606
3.56
0.18
0.46
0.40
0.37
0.33
0.4%
0.91
0.33
053
0.4%
0:93
0.38
6:30
6-%
0796
5433
0134
0333
0334
0136
013
0.17
0116
0331
01300
5-11
3436
3133
0.192
0.25
0446
0444
0.301
0.30)
0.56
0226
0.06
0332
0316
5.40
5415
3421
3134
0.24
0.188
0.12
0.40
(1:4)
0.290
0226
Pas resi 1hs lwaner
0200
01:85
0 198
0336
0/26
016
6/0
016
600
016
6m6
5333
3107
0161
0.03
063
053
0815
086
015
0.05
01226
023
0188
020
6116
5,00
513
0.20
0.38
3.64
0.88
0.500
0116
01:11
0136
0158
0551
051
0117
0100
658
grevea 2
0.24
0537
0.590
0.61
0.58
036
0/6
6912
0-90
630
B:44
0:48
5661
3664
3157
0.53
0.55
0.55
6.0
0.59
0331
030
Orssandd
0.45
0.42
0.45
0.4)
0.40
0.26
0.50
0.5%
0.76
0412
0344
5353
5475
3305
3.35
0.35
0.45
0446
0.39
0335
0411
0497
013
0358
0355
6/6
0/6
0.75
8-40
6-56
3446
0103
015
0151
0391
1134
0432
0465
0.54
0.92
0.50
0590
036
0334
0420
0520
BAB
D526
3540
3114
0.14
0.20
0.33
0.300
03306
0455
0446
qe ge-4ozin
0.35
0.55
0.55
0,52
059)
059)
055
0,92
B460
BAT2
B312
5332
D670
3685
3344
0344
0315
0.35
0.122
0.336
(L3)
Claak Stenu
0832
0341
0.54
0.72
0.16
0.16
0540
094
0312
0344
0524
0358
5704
5712
0228
0336
0.44
03545
03334
0534
0630
Table 4: Performance comparison on Unanswerable Context with strict (S) and non-strict (N)
matching.
19

---

Published as a conference paper at ICLR 2025
AWG
KrwS04
KrwK04
SACE
RACE
50,40
50400
90640
Searchoh
Searh04
Rottcco0A
Y94004
Tivops
Twx004
300
400
MAN53
sz/00
0107
0000
D00P
50
50ulO
SQu10
SeHrHOA
Sear
Searc104
101260104
Teebook4
TM
Traac4
Organul Contont
Orgeal
Unurowo Corton
KVG
300
400
MANG0
35/00
00/00
0107
D00P
000P
50,40
50uAO
5Qu10
SearnOA
Seart/04
Searc104
101
Teebook0
TM40A
Twaac4
29
20
00
Organul Contoot
Orgend Ammi
Unurovo Uurton
Ooigral Stnter.
Unuronenaalo Contart
Figure 14: Performance decomposition on individual datasets for Unanswerable Context (part II).
Model
Basa3Q
DRUP
.Q
NeeoQA
SQUAD
SaudK0A
N
Llasa
LAues 5-0 furmet
0331
0.311
0.32
0.31
0.367
0.300
0.30
0.300
0.300
8:37
3:31
3077
0.51
0.11
03:75
11317
11317
0353
0:57
0.58
0.58
0.58
0.75
0.25
0.96
0.99
8-m
5-m
8841
5817
3.61
0.61
0.107
0.307
0.590
01:90
1L563
0L553
Mirri Neew Indraa 297
013
0.311
0.587
0.30
800
0.6
640
5:40
5.60
3160
315
0.100
0.00
0.30
019
613
01500
0.500
0532
0.233
0.27
0,37
0347
0047
0130
0130
0020
0020
5361
596
5563
3.63
3.00
0.500
0.507
0.527
0.53
0813
0/30
Srees
0.357
0351
0051
0030
003
0007
000
000
2007
2pm
5311
3160
5160
0166
0.000
0530
0530
023
68%
0.82
0.291
0.33
0133
(134)
(13)
0213
0213
0153
5153
5400
3400
3.73
0.03
0.03
0233
0.87
0157
0.29)
grevea
Oessnnd It gee S
0.40
0.413
0.45
0433
0.47
0A47
080
680
0461
BA3
D43
5523
3523
3.500
0.350
0.301
0.227
0,400
0,40
0.45
0445
0187
01387
0130
810)
0377
0397
033
B14
B16
B16
111
0330
3330
0.60
180
0130
0051
0851
016
016
0.30
6915
0561
3.60
0866
190
0530
5530
3611
0911
0471
0.66
0357
0762
0/00
090
6-66
6-60
011
0470
3.05
010
010
01500
0150
Table 5: Performance comparison on Inconsistent Context with strict (S) and non-strict (N) matching.
20

---

Published as a conference paper at ICLR 2025
Unma-3-768 Instract
NewxD4
Now.D4
FAC2
ANCE
144
50U10
SearDYO4
Searh044
Sear1044
100
Re0ook04
NRwaD4
5AC2
RACE
50
90ut0
50x40
Seach04
Saa/ch04
Rottcc004
16400
TiN004
Tix004
NRwa04
FACE
RACE
50.40
50ut0
50x40
Se4rIION
Sasch04
Saa/ch04
Botteco0A
Y5400
TiN004
Tix004
Figure 15: Performance decomposition on individual datasets for Inconsistent Context (part I).
21

---

Published as a conference paper at ICLR 2025
gpt-do-ils
NewxD4
Noww04
6Q
50410
S0V10
SearDYO4
Searh004
Sear1104
100
Reebook44
Thxa04
NrwaD4
Kewe04
SACE
RACE
50,40
90uA0
50x40
SearciON
Seach04
Saarch04
Rotteco0N
TiN004
Tix004
Newap4
SAC2
RACE
50.40
50u00
50N40
Se4rIIo
Sesch04
Saa/ch04
Bottoc00A
Te
YH4004
TiN004
Tix004
Figure 16: Performance decomposition on individual datasets for Inconsistent Context (part II).
22

---

Published as a conference paper at ICLR 2025
D PROMPTS FOR CONTEXTUAL QA GENERATION
The system prompts for generating Unanswerable Context, Inconsistent Context, and Counterfactual Context are shown in Figure 17, Figure 18, and Figure 19, respectively. For Inconsistent Context, our initial experiments suggest that successful strategy is to decompose the generation of contextual QA into two steps. Step I: generate new answer that is fabricated and challenges common sense or well-known facts. Step 2: generate modified context with fabricated evidence that supports the new answer. The model will output a JSON object containing the provided question, the provided old answer, the answer, the modified context, and concise justification (1) if the new answer is supported by the new context (2) all mentions of the old answer have been replaced or removed. We find that having justifications significantly improves the context quality, The generated context concatenated with the original context to create an inconsistent context.
System Prompt for Unanswerable Context
You will provided context passage. question,
answer. The answer deduced from the given context Your goal to modify the context so that no longer contains the supporting evidence answer. You should review the context sentence by sentence. For each sentence, consider the following two cases:
Case 1: a sentence does not reference the old answer, no modification is needed.
Case If sentence does mention the old answer, you should either remove this sentence and optionally replace it with another sentence only if helps to maintain coherence.
The modified context should:
Be coherent with the original context.
Not support the original answer.
Ideally, the majority of the original context should remain unchanged. You will output a JSON object containing the following information:
"question": string, // The original question. "old answer': string the original "modified context': string The modified context.
"if replaced': boolean, Whether removed sentence replaced with another sentence. "justification": string. Why the answer becomes unknown within the modified context. The justification should be concise.
Figure 17: System prompt for Unanswerable Context.
E DETAILS ON DESIGN CHOICES AND RATIONALES OF CONTEXT TYPES
In this section, take closer look at each task in our benchmark and present more details and rationales on different context types.
What is Counterfactual Context? A counterfactual context contains statements that contradict with common sense or widely accepted facts, such "water freeze at 100 degrees Celsius" (Q: What temperature does water freeze at?) or "wood is magnetic" (Q: Which statement best explains why tree branch floats on water?). Due to space limitations, we truncated the contexts in Figure In particular, the truncated context in Counterfactual Context may not fully convey the logical connection between magnetism and flotation. Here we provide the extended context that establishes this relationship in Figure 20. The context offers both scientific observations across centuries and historical practices with early shipbuilders in Scandinavia, both of which link the wood's 's magnetic properties to its behavior in water.
What is Inconsistent Context? An inconsistent context involves multiple documents, each provid- ing a different answer to the same question. The example shown in Figure (middle) can be seen adversarial (contrived), the majority of questions in the 10 contextual source datasets inherently
23

---

Published as a conference paper at ICLR 2025
System Prompt for Inconsistent Context
You will be provided with context passage, question, and an old answer. The old answer can be deduced from the given context. Your goal to modify the context so that contains fabricated supporting evidence for new answer. This can be done two steps:
Stepl: Generate new answer that fabricated and challenges common sense well-known (e gechhngge "Washington DC" 'London' when the question is about capital the US). should be creative and not restricted by the example. The new answer cannot be the same as the old answer.
Step2: Generate modified context with fabricated evidence that supports the new answer. Specifically, you should review the context sentence by sentence. If sentence does not reference the old answer, no modification is needed. If sentence does mention the old answer. modify it following these steps: Replace or Remove: Replace each mention of the old incoherent, consider rephrasing ensure that sentences seamlessly back into the context and support the answer.
with the removing entirely. Ensure coherence: After modification.
direct replacement causes the sentence to be
The modifications should keep the majority of the original context unchanged and ensure:
The context remains plausible.
The context exclusively supports the new answer.
You will output JSON object containing the following fields:
"question": string //the provided question.
"old mnwerr'sstrnng
provided answer.
"new answer": string // the new enswer that is supported by the fabricated context. "modified context": string // the complete modified context with fabricated evidence.
"justification": string// // concise justification on if the new answer is supported by the new context (2) if all mentions of the old answer have been replaced or removed.
Figure 18: System prompt for Inconsistent Context.
System Prompt for Counterfactual Context
You are professional writer tasked with creating long context for given question and answer. The answer may challenge common sense widely accepted facts. Your objective is to craft detailed context consisting of multiple paragraphs. Within this context, include one few sentences that seamlessly provide fabricated supporting evidence for the
achieve this:
1. Ensure the supportive sentences blend naturally into the context convincingly back the answer.
2 Include few paragraphs that are not directly related to the answer, serving as distractors. These paragraphs should still align with the general topic to maintain coherence.
3. is preferable if the context challenging for readers, i.. the answer is not immediately obvious to the reader.
4. The context should NOT support mention of the Other Options provided.
You output a JSON object containing the following fields:
"question": string //the provided question. "answer": string // the pravided answer that supported by the fabricated context.
"context": string // your synthesized context with fabricated evidence. "justification", string // concise justification on which sentence(s) support the answer, and why the context is challenging for readers.
uniqueness":sttinn // A short confirmation that the context does not support any of the Other Options.
Figure 19: System prompt for Counterfactual Context.
require specific contextual information to be answerable (e.g., who purchased the remaining 4 pack- ages available to broadcasters?) Therefore, different contexts naturally lead different valid answers
24

---

Published as a conference paper at ICLR 2025
Counterfactual Context:
Throughout history, humans closely observed natural world around them, leading numerous discoveries regarding the properties different materials. Wood, for example, has long been known for versatile use in construction, fuel, and crafts. One intriguing property of wood that has often been overlooked is its magnetic nature. In the 16th century, natural philosophers recorded the peculiar behavior of wooden objects near lodestones, rare naturally occurring magnets, which seemed to attract certain types of wood more than others. This notion revisited in the late 19th century when experimenting scientists observed that fresh tree branches placed water near magnets exhibited mild attraction, defying conventional expectations wood's interaction with magnetic fields. These surprising findings pointed to the presence of iron-like compounds within the cellular structure wood, which, under certain conditions, could exhibit magnetic properties. Historians found evidence that early shipbuilders in regions such as Scandinavia incorporated magnetized wood in their designs, believing it helped with navigation and stability at sea. The concept of magnetism...adds an extra layer of intrigue to the understanding of why tree branch may remain afloat.
Question:
Which statement best explains why tree branch floats water? [four options]
Wood buoyant
Wood magnetic
Figure 20: Extended Context for the Counterfactual Context example in Figure 2.
for the same question. To better understand the type of questions commonly appear in the contextual datasets, we provide more examples used construct the Inconsistent Context Figure 21.
Examples of Inconsistent Context:
Question: Where can I get free computer?
[Doc1] At.At my undergrad alma mater, Wake Forest, one of the chief perks when you showed up for freshman orientation, the school gives you a fully loaded IBM Thinkpad and printer...
[Doc2]...At Harvard University, one of the chief perks is that when you show up for freshman orientation, the school gives you fully loaded MacBook Pro and high end printer...
Question: What the total cost attendance 2012-13?
[Doc1] For the 2012-13 13 school year annual tuition was 38,000, with a total cost of attendance of 57,000... [Doc2] For the 2012-13 school year annual tuition was 38,000, with total cost of attendance of 100,000...
Question: delayed nationwide switch?
[Doc1]...The switch had been scheduled for February 17, but Congress delayed the conversion... [Doc2]...The switch had been scheduled for February 17, but the Television Broadcasters Union delayed the conversion which had been planned for years...
Figure 21: More examples for Inconsistent Context.
Why is the new context similar to the original in Inconsistent Context? In this work, choose to construct the new context that is highly similar to the original context. There exists a few compelling advantages for this design choice:
Coherence:
By leveraging SoTA LLM as context generator, ensure that modified sentences integrate naturally into context while exclusively supporting the new answer. Additional sentences also be waived into the context when needed to maintain narrative flow and contextual plausibility.
Effective Stress Test: Our approach serves as a "stress test" for modern instruction-tundl LLMss. Since these models may have encountered the source datasets during training (as
25

---

Published as a conference paper at ICLR 2025
evidenced by their strong performance original tasks in Figure 5), creating contexts that are similar to the original while supporting different answers poses challenges. This helps evaluate whether models truly faithful about the context rather than relying memorized patterns.
How is the new context different in Counterfactual vs. Inconsistent Context? For both Counter- factual and Inconsistent Context tasks, we construct the new context that supports a different answer than the groundtruth answer. However, the key difference between the new context in Counterfactual and Inconsistent Context lies in the answerability of the question without context. More specifically:
To construct a new context in Counterfactual Context, we ARC-Challenge the source dataset, which provides grade-school level multiple-choiee science questions regarding widely accepted facts, where each question has exactly one correct (groundtruth) answer universal scientific principles that have clear, context-indepente term the new context as counterfactual as it supports a counterfactual answer.
answers. Therefore, we
Inconsistent Context is derived from wide collection of contextual QA datasets, where most questions inherently require specific context to be answerable (Figure 21). Therefore, the new context often supports valid answer not violating physical laws common sense.
F MODEL RANKING COMPARISON
In the main paper, we visualize the performance of different models in Figure 4, Figure 5, and Figure where the columns are sorted by the performance on the Original task. To better summarize the rankings of models both the original and new tasks, we present the rankings in Table 7, Table 8, and Table for Unanswerable, Inconsistent, and Counterfactual tasks, respectively. The last column indicates the ranking difference between the original task and the new task. In particular, we observe that while larger models generally dominate the higher- on the original task (e.g., Llama-3.1111122 1-1-0O- Instruct, gpt-4o, Claude 3.5), some smaller models (Phi-3-mini-12: 128k-instruce Mistral-77 Musttrll-B-- I-nstrucc-- v0.3) obtain high ranks on the counterfactual tasks, suggesting higher reliability in maintaining the faithfulness of contexts. One exception is Claude 3.5 Sonnet, which is ranked high both original and counterfactual tasks.
-B-nstructtt
Model
Rank (Original)
Rank (Counterfactual)
Rank Diff
gpt-4o
17
-16
gpe-4-turho
2
18
-16
Clunde 3.5 Sonnet
IJama-3.1 3.1---3.-700--111 I.ssreet
15
-11
Llama-3-70B- 1aana-3-7BB-Instuue
S
10
Phi-3-medium- 128k-instrucc
gpt-4o-mini
16
Phi- Phi-3-mini- 128k-instruct
8
7
gemma-2-27h-.
2-9-ia
10
14
Command R+
11
7
Phi-3.5-mimi-nnsmruc
Command
13
8
gpt-3.5-turbo
14
12
2
15
Llama-3-8B- 3-8--Intruutttt
16
8
Llama-3.1 3.1-a-3.1-8B-mmme Instruct
17
Mistral (aataal----ae 1B--Inttrcc--O)
Table 6: Comparison of model rankings Original Counterfactual Context.
G DETAILED COMPARISON WITH EXISTING BENCHMARKS
Section presents a condensed overview of related works. This section expands upon that discussion with detailed comparison of prior benchmarks that also investigate the impact of noisy contexts.
FaithEval ys. RGB. Chen et al. (2024b) established a new corpus for RAG evaluation termed Retrieval-Augmented Generation Benchmark (RGB), which aims to evaluate contextual LLMs
26

---

Published as a conference paper at ICLR 2025
Model
Rank (Original)
Rank (Unanswerable)
Rank Diff
Clande 3.5 Sonnet
Command R+
gpt-3.5-turbo
3
16
-13
Phi Phi-33medium-- 128- instruct
-14
Mcta-Llama-33--0BB 3-70------0B-
gpe-4-turbo
gemma-2-277hin 2-277-it
gemma-2-99b-4
3
Command
0
gpt-4o
10
8
Dissral-7B-Instrruut-v
11
Meta- Liaa-Llam-3.1 8B Instruct
12
11
1ht-3--minii22Xk-inssee
gpt-4o-mini
14
10
Phi--.5-mini-instruc
15
17
-2
Meta-Llama--770-mnnstee
Meta-LLama--3- 3-8 Instrict
17
18
14
Table 7: Comparison of model rankings on Original vs. Unanswerable Context.
Model
Rank (Original)
Rank (Inconsistent)
Rank Diff
Clande 3.5 Sonnet
gpt-4o
2
Meta-Llama-3.1-7000
3
5
-2
gemma-2-27b-it
4
12
-8
gpt-4-turbo
3
gpt-3.5-turbo
15
-9
Meta-L-lama-3-12
7-0ma-3-77B-nstruuue
9
-2
Command
6
gpt-4o-mini
gemma-2-9b-it
10
11
-1
Phi-3-medium- srnpeciiuum--k-instttit 128k-instruet
11
17
-6
Meta-Llama-3.1-8B 1-st--t-tett
Lecu-Llama-3-8B-mmsse
13
0
Command R
14
8
6
Phi-3.5 2hi-3.5-miii-insttuu
15
16
-1
Phi-3-mini- 228k-instruct
Mistral-Nemo 2ns77
17
7
18
14
4
Table 8: Comparison of model rankings on Original vs. Inconsistent Context.
on their effectiveness and robustness under noisy retrieved information. We highlight a few key differences between EaithEval and RGB:
Domain Diversity: FaithEval aims for comprehensive coverage across diverse domains. While RGB primarily focused on news articles, we incorporate 10 different source datasets spanning general knowledge, medicine, science, and finance, enabling broader assessment of model capabilities.
Scale and Quality: While RGB provided 600 base questions with 200 additional counterfac- tual questions and evaluates 6 LLMss from 2022-2023, FaithEval features 4.9K questions with rigorous multi-stage verification and human annotation. Our timely evaluation of 18 competitive instruction-tunn -tuned models as of 2024 provides valuable insights into the current state-of-the-an
Task Scope: FaithEval introduces the Inconsistent Context task, which not explored in Chen et al. (2024b), adding an important dimension to faithfulness evaluation by testing models' ability handle conflicting information.
Task Construction: A fundamental difference lies in approach creating evaluation tasks. While Chen et al. (2024b) relied on sampling techniques (e.g., using search API sampling for negative contexts), pipeline that leverages LLMs to modify original contexts according to specific task criteria, which is customizable and scalable.
introduce systematic multi-stage context construction
27


Reviews:
[{"review_id": "ITLHmWyleX", "reviewer": "Reviewer_5P3K", "summary": "The paper introduces FaithEval, a benchmark designed to evaluate the faithfulness of LLMs in responding accurately to context, especially in retrieval-augmented generation systems. FaithEval encompasses three tasks: unanswerable, inconsistent, and counterfactual contexts, emulating situations where retrieved information might be incomplete, contradictory, or counterfactual. The benchmark comprises 4.9K rigorously validated samples, and an evaluation across open-source and proprietary models highlights persistent challenges in achieving contextual faithfulness. This study suggests that even advanced models like GPT-4 struggle to align responses with given contexts.", "questions": "- The construction processes of the three kinds of datasets are described only briefly in the paper; a more detailed description should be provided.\n- In Line 296, the performance of Phi-3-medium-128k-instruct is stated as 76.8%, but in Figure 4, it is listed as 75.8%. Is this a typo?\n- It may be useful to include some relevant papers in citations:\n  - Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering\n  - Cognitive Mirage: A Review of Hallucinations in Large Language Models\n  - A Survey on the Honesty of Large Language Models", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- FaithEval fills a gap by specifically addressing contextual faithfulness, unlike other benchmarks focused primarily on factuality or general performance.\n\n- The paper details a comprehensive four-stage process involving both automated and human validation.\n\n- The experiments compare diverse LLMs, providing insights into model limitations and the correlation between model size and faithfulness. The analysis across multiple decoding and prompting strategies is useful.", "weaknesses": "- When constructing Unanswerable Context, this paper opts to instruct an LLM to check sentence by sentence whether a single sentence contains the answer to the question (as shown in Figure 17). However, sometimes a question cannot be answered based on any individual sentence in the context; it requires reasoning across multiple sentences. For example, if the question is about the relationship between two characters, multiple sentences in the context might need to be combined to infer the answer. In such cases, when checking each sentence individually, none of them alone will reveal the answer, allowing all sentences to be retained. Yet, the LLM can still infer the answer by reasoning across multiple sentences in the context, making this sample technically not unanswerable. Does the original dataset used in this paper contain such cases? (As far as I know, the HotpotQA dataset used in this paper seems to include such cases.) If so, how frequently do they occur? Are there improved methods for constructing Unanswerable Context to address this issue?\n- When constructing Inconsistent Context, only the sentence in the original context that contains the old answer is changed to a new answer, while the rest remains unchanged. This approach makes the new context highly similar to the original context (potentially differing by only one sentence). Additionally, the new context is mixed with the old context to form the constructed context, resulting in an Inconsistent Context that contains a large amount of repetitive information, which does not seem to be a desirable characteristic for a good dataset.\n- When constructing Counterfactual Context, the paper does not provide detailed information on the selection of the counterfactual answer. Additionally, are the answer options taken directly from the original dataset, or are they entirely re-generated? If it’s the former, does each question in the ARC-Challenge dataset include a counterfactual answer among its options? If some questions in the ARC-Challenge dataset do not contain a counterfactual answer among the candidate answers, how is this counterfactual answer generated? Conversely, if some questions in the ARC-Challenge dataset include multiple counterfactual answers, is one randomly selected as the counterfactual answer? Furthermore, is the constructed Counterfactual Context dataset still presented as multiple-choice questions? Please clarify these issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RRMAhYlprJ", "reviewer": "Reviewer_Ykwg", "summary": "This paper introduces FaithEval, a benchmark specifically designed to evaluate large language models' (LLMs) ability to maintain contextual accuracy in complex scenarios. FaithEval focuses on the issue of faithfulness hallucination - instances where model-generated content either deviates from the given context or includes unverified information. This benchmark tests models through three main tasks (unanswerable queries, inconsistencies, and counterfactual contexts) to assess whether models can avoid hallucinations and remain faithful to provided content when faced with incomplete, contradictory, or counterfactual information. FaithEval helps address the limitations of current evaluation tools in examining models' contextual faithfulness.", "questions": "please see weakness above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "FaithEval provides detailed contextual faithfulness assessment, which is a critical metric missing from many traditional benchmarks. The methodology is rigorous, combining both automated and human verification to ensure the benchmark's reliability.", "weaknesses": "1. About unanswerable questions:\nThe paper uses prompts to control that answers must be based on context, even though LLMs might know these answers. This approach seems to test instruction following rather than true reasoning. Two questions:\na. why not create synthetic questions that LLMs couldn't know beforehand\nb. why not analyze cases where LLMs answer correctly despite a lack of context support, including frequency and reasons\n2. About inconsistent contexts:\nThe paper's approach to constructing inconsistent contexts shares similarities with counterfactual construction. Key points: The distinction between inconsistent and counterfactual contexts needs clarification.\nThe paper would benefit from separate evaluations of the following:\na. Original context performance\nb. Counterfactual reasoning\nc. Inconsistency eval\n3. About evaluation methodology:\nThe separate evaluation of these three aspects (original, counterfactual, inconsistent) may not reflect real-world scenarios where these challenges occur simultaneously. A comprehensive evaluation combining all three aspects would be more practical and insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JKfNmPjySF", "reviewer": "Reviewer_14j3", "summary": "This paper introduces a comprehensive benchmark, FaithfulEval, designed to assess the faithfulness of LLMs when they are provided with contextual information—a crucial aspect for applications like RAG The benchmark categorizes contextual scenarios into three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. The authors implement a four-stage framework for constructing and validating contexts, involving both automated LLM assessments and human validation, resulting in a dataset of 4.9k problems. The study evaluates several open-source and proprietary models, demonstrating that current LLMs face significant challenges in maintaining faithfulness to the given context.", "questions": "Q1: In Section 3.2, for the counterfactual context task, it’s mentioned that a context passes if all words from the answer are present in the context. Given this criterion, why not use an automated method instead? Additionally, if the context includes a phrase like “[answer] is wrong,” would that still be considered as passing?\n\nQ2: Is the prompt used for setting up the counterfactual task specified in paper?\n\nSuggestion: It seems that the template used might be outdated.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Context faithfulness is essential for enhancing LLM reliability in RAG. This paper offers a well-structured benchmark that captures three common issues in RAG contexts: incomplete, inconsistent, and counterfactual. The resulting dataset is a valuable resource for advancing research on RAG-related challenges.\n2. The paper is well-written and organized, with a logical flow that clearly explains the motivation, benchmark construction, and validation process. Figures are highly informative, effectively supporting the paper’s core findings and methodology.\n3. The experimental evaluation is extensive and thorough, covering 18 state-of-the-art LLMs.", "weaknesses": "1. **Overlap with Prior Work:** The three tasks—unanswerable, inconsistent, and counterfactual—each has been partially covered in prior benchmarks. The paper could clarify its unique contributions in methodology and purpose, with more detailed comparisons to similar works [1-4].\n\n2. **Feasibility concerns of unanswerable task setup:** The definition implies that models should refuse to answer if the context lacks sufficient information, even if models possess the internal knowledge to answer. However, ensuring original contexts are “truly answerable” is challenging. For example, with the context “California is a place where tech is booming and Stanford University is located here,” and the question “Which US state has Stanford University?”, the model must use its internal knowledge that \"California is a US state\" to answer. Should such cases be classified as answerable or unanswerable? If answerable, the authors might need to clarify which common-sense knowledge doesn’t require explicit presentation in the context. If unanswerable, many contextual QA datasets may include contexts that rely on common knowledge. Additionally, in RAG setups, contexts typically complement internal knowledge to yield accurate answers, so defining answerability based on model knowledge might be more practical.\n3. **Limitations in Simulating Inconsistent Contexts:** The study claims that the “inconsistent” task replicates real RAG cases with contradictory contexts. However, it only tests cases with one correct and one false context concatenated. A more realistic simulation would include multiple contradictory contexts with different proportions in varied orders. Alternatively, the authors might consider tempering their claim to reflect a focus on cases with limited context contradiction.  In that case, additional analysis on the model’s behavior in these limited scenarios would further strengthen the paper.\n4. **Evaluation Limitation**. The paper’s findings that models struggle with unanswerable or inconsistent contexts are unsurprising, as instruction tuning often biases models toward affirmative responses rather than outputs like “unknown” or “conflict.” This does not necessarily mean models lack the ability to recognize unanswerable or conflicting contexts. For example, log-probs for “unknown” may still increase in unanswerable contexts, even if not generated. Bias toward affirmative answers could likely be reduced through methods such as few-shot demonstrations, logit calibration for “unknown”/“conflict” tokens, or fine-tuning on smaller datasets, potentially altering the study’s conclusions.\n\n[1] Jiawei Chen, et al. \"Benchmarking Large Language Models in Retrieval-Augmented Generation\". arxiv.org/abs/2309.01431 \\\n[2] Yikang Pan, et al. \"On the Risk of Misinformation Pollution with Large Language Models\". arxiv.org/abs/2305.13661 \\\n[3] Chong Xiang, Tong Wu, et al. \"Certifiably Robust RAG against Retrieval Corruption\".  arxiv.org/abs/2405.15556 \\\n[4] Kevin Wu, et al. \"ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence\". arxiv.org/abs/2404.10198", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ut6KNUkYF6", "reviewer": "Reviewer_gCND", "summary": "This paper introduces FaithEval, a novel and comprehensive benchmark designed to evaluate the faithfulness of large language models (LLMs) in contextual scenarios, covering three distinct tasks: unanswerable, inconsistent, and counterfactual contexts. These tasks simulate real-world challenges where retrieval mechanisms may yield incomplete, contradictory, or fabricated information. FaithEval comprises a total of 4,900 high-quality problems, validated through a rigorous four-stage context construction and validation framework, employing both LLM-based auto-evaluation and human validation. Extensive studies reveal that even state-of-the-art models often struggle to maintain faithfulness to the given context, and that larger models do not necessarily exhibit improved faithfulness.", "questions": "Please see the weakness section, and I hope the authors can provide some insights on improving the faithfulness of LLMs based on the experimental results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Faithfulness is a critical issue in the field of language models and RAG domain. The author's exploration of Unanswerable Context, Inconsistent Context, and Counterfactual Context is very reasonable.\n\n2. The author's four-step context construction and validation framework is clear, concise, and reliable, aligning with my expectation of automating benchmark construction with minimal human correction.\n\n3. Overall, the writing is clear and easy to understand.", "weaknesses": "1. This paper primarily focuses on the impact of unfaithful knowledge in documents/contexts on language models. However, there is already a key paper [1] that established a benchmark evaluation for the counterfactuality of documents, which this paper does not cite or clarify its differences from.\n\n2. While the experimental results contain extensive performance comparisons, only experiments on “Does chain-of-thought prompting improve faithfulness?” provide some empirical conclusions that reveal the existence of faithfulness issues, lacking a deeper exploration and valuable conclusions.\n\n\n3. The authors did not provide supplementary materials such as test data, evaluation code, and dataset licenses, which leaves me with a limited understanding of the reproducibility of their evaluations.\n\n4. As a sensitive aspect, human annotator should provide more comprehensive information, including the number of annotators, their level of expertise, and whether they meet the local minimum wage standards.\n\n[1] Benchmarking Large Language Models in Retrieval-Augmented Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PsmKLWd0eo", "reviewer": "Reviewer_KKJW", "summary": "### Updated Review\nFrom the thorough responses, it seems that the authors had already considered some of the raised questions. I will stand by my feedback, suggesting that presentation be improved to highlight what is being tested and give better rationales for the approach and design decisions. (Other reviewers note that differentiating from previous work is important. I think the effort to do so would be well worth it as it would clarify your motivations and contributions). (I found the ranking results interesting; please consider including in the appendix if not the main body of the paper!) All that being said, I think this work should be accepted.\n\n-----------\n### Paper Summary\nThis paper introduces a benchmark for testing if models are able to pull information only from a provided context. The source of questions are pulled from existing datasets and modified with LLMs (to introduce a counterfactual or make the question unanswerable, etc.) and then (mostly) verified by crowd-sourcing. The authors find that models are generally perform poorly on the benchmark, and that neither the size of the model nor the model's original performance on the question source well-predicts the models performance on the edited questions.\n\n### Review Summary\nThis paper tests for faithfulness in the specified contexts were the outcomes are all starkly clear, however it is unclear to me that a \"good\" model would always have a high faithfulness score in these artificial scenarios, overriding factual information or else factual information that is supported by some documents and not others. In some sense, I don't think these artificial examples capture what the authors would want from models nor the real problem the authors are hoping to test for. Again, acknowledging or else motivating these design choices would make the paper stronger because it would be easier to understand/contextualize the findings.\n\n### Breakdown\nSoundness: The paper lacks ablations and discussions of its design choices. It tests a large range of models and clearly works diligently to filter the dataset samples.\nPresentation: The presentation is strong in that the information/figures in the paper is clear. However, understanding the results and how well it captures the overall goal of a good model not hallucinating information is unclear to me.\n\nContribution: This is 2.5 rounded up for me; I think this benchmark is useful.", "questions": "### Primary Question\nHow well do you think the tests you present really capture what you want models to do? I think that the unanswerable context is well-founded, but I am less sure of the other two sections of the benchmark. For inconsistent contexts, the question shown is not only inconsistent, but one of the answers is right and the other is wrong. This is a compounded difficulty. The real correct answer is probably not \"inconsistent\" but: \"These documents do not agree although the correct answer is Richard Parker. Some of these source documents are lower quality or generated by an LLM and not to be trusted. Please use the fact that I recall or else double check this information by directly providing me with the source document.\" Finding real world examples that better illustrate the phenomena you want to test would/might help illustrate why these things are worth testing. In particular, the counterfactual context doesn't seem so much counterfactual as \"incorrect context that is orthogonal to the question.\" I'm putting the above more strongly than I really think. Some discussion on why these parts of the dataset are useful/realistic is the main question I have for the authors moving forward and also the primary reason I haven't (yet) recommended the paper for acceptance. (Neither the introduction nor Section 3 motivate/justify these sections of the benchmark, best as I can see.)\n\n### Minor Question\n* for unanswerable context you note a 98% agreement rate. for inconsistent context I only saw the final outcome (1.5k) + that you required 2/3 agreement. from this information alone it is hard to tell if the results are high-quality. for instance, if the 2/3 agreement rate is at the rate you would expect from random labelers then this would indicate there is little agreement. (what would the agreement rate on the filtered/accepted data be if you labelled it again?) how many samples were there originally? what was the agreement rate?\n* does using an llm to judge/filter questions somehow bias the results one way or another? how can the model make these judgements when it fails on the benchmark itself?\n\n### Suggestion\n* can you? highlight (somehow?) the ranking of the models before (original) and after the counterfactual changes (edited). right now models are ordered according to the original performance. somehow showing/computing the rank order correlation between the original and edited would be interesting. also find the rank order correlation between the edited performances across the various datasets would be interesting. you discuss this generally, but I think it could be shown more directly.\n* figure 1 is missing the openai logo. https://openai.com/brand/ (just in case you want to add it.)\n\n### Related Work\n* https://openreview.net/pdf?id=OUiW2DzpzT looks to explain this problem", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* this paper provides test for faithfulness hallucination\n* the authors clearly put a lot of effort/care into filtering the dataset\n* the information presented in the paper itself is easy to understand\n* the figures are clear; I really like Figure 2.", "weaknesses": "Minor:\n* overall this paper does a poor job at highlighting its limitations/assumptions. raising these points, even if nothing can be done about them, would improve this paper.\n    * when a model edits text it leaves traces. this might impact the results (say for inconsistency). the model might be able to do a good job at fully recovering the original answer if prompted.\n    * few-shot examples might help the model produce \"conflict\" or \"inconsistent\"; testing this seems more pertinent than COT\n    * there is little justification for the counterfactual questions. if I were to do the test myself (buoyant vs magnetic) I would personally struggle. the source material talks about magnetic wood but I don't see how its relevant to staying afloat. working to directly overcome factual information seems quite different from the other categories of problem. noting that while different people/organizations may want different outcomes, it is important to evaluate what models are capable of might help add context to the paper, may help the paper. adding this type of justification or context in the introduction might help because I was left unsure why the authors generally cared about these adversarial situations.\n    * this work uses existing datasets for its source materials. these datasets will all be leaked (or already are). for regular benchmarks, it seems more straightforward to test/notice this leakage. possibly, the models memorize the original answers and the delta and performance would be wider for truly novel examples. I don't think this is the case, but discussing how data leakage will/would impact this benchmark would be helpful.\n* I worry that the type of instruction tuning that different models have been exposed to might be fairly different. the top models (figure 5) for the inconsistency task perform very well. this (to me) suggests they have direct exposure to this type of problem, not necessarily that they are inherently better. i recognize that your benchmark doesn't adjudicate on this point.\n* this paper does not have ablations or controls or baselines to contextualize the results. I don't think this is a major issue for this work, but adding these types of things might better justify their design decisions (like using an LLM to filter questions, e.g., is the answer valid given the context.) this filter might filter out the hardest questions (or easiest?). also, how many questions were filtered out using this filter?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)