To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts
Authors: Yukun Huang
Sanxing Chen
Hongyi Cai
Bhuwan Dhingra
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Model
Knowledge Conflict
Retrieval Augmented Generation
Confidence Estimation
Reasoning
Primary area: interpretability and explainable AI
TL;DR: We benchmark the challenge of ensuring large language models remain situationally faithful to potentially incorrect external information and propose Self-Guided Confidence Reasoning to enhance LLM's reliability.

Abstract: Large Language Models (LLMs) are often augmented with external contexts, such as those used in retrieval-augmented generation (RAG). However, these contexts can be inaccurate or intentionally misleading, leading to conflicts with the model’s internal knowledge. We argue that robust LLMs should demonstrate situated faithfulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context to resolve knowledge conflicts. To benchmark this capability, we evaluate LLMs across several QA datasets, including a newly created dataset featuring in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faithfulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-access the confidence of external information relative to their own internal knowledge to produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. 
Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2\% over a direct input augmentation baseline. Conversely, for a smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Confidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9\% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR. Our findings highlight promising avenues for improving situated faithfulness in LLMs.

Published as a conference paper at ICLR 2025
TO TRUST OR NOT TO TRUST? ENHANCING LARGE LANGUAGE MODELS' SITUATED FAITHFULNESS TO EXTERNAL CONTEXTS
Yukun Huang, Sanxing Chen, Hongyi Cai & Bhuwan Dhingra
Duke University {yukun. krukun.huang, sanxing. hongyi congyi.caalle @ddkee. edu bdhingra@cs. duke. ddu
ABSTRACT
Large Language Models (LLMs) often augmented with external contexts, such as those used in retrieval-augmented generation (RAG). However, these contexts can be inaccurate intentionally misleading, leading to conflicts with the model's internal knowledge. We argue that robust I.L.ss should demonstrate situated faith- fulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context to resolve knowl- edge conflicts. To benchmark this capability, we evaluate LLMs across several QA datasets, including newly created dataset featuring in in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and in- correct contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faith- fulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-assess the confidence of external information relative to their own internal knowledge produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2% over direct input augmentation baseline. Conversely, for smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Con- fidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR. Our findings highlight promising avenues for improving situated faithfulness in LLMs.
INTRODUCTION
Large Language Models (LLMs)) can be enhanced by incorporating relevant external context (Chen) ct al. 2017: Lewis et al. 2020) provided by automated retrieval systems, tools, users, or another model (Mialon al. 2023). However, external information can often be unreliable, containing errors due to online misinformation, intentional disinformation stemming from malice (Zou et al., 2024). noisy human inputs, and outdated data (Kasai et al. 2023), leading to knowledge conflicts with LLMs' internal beliefs. Prior research has demonstrated that LLMS can be too faithful to the external context, making them susceptible to being misled by unreliable contexts (Petroni et al.
2020 Xie et al. 2023 Shi et al. 2023 Shaier et al. 2024).
Instead blindly following external information, a robust LLM should exhibit situated faith- fulness, dynamically balancing its reliance internal knowledge and external context to resolve knowledge conflict. This means the model should trust its internal knowledge when it is correct or when external information is unreliable, while conversely leaning externl data when uncertain or when the external evidence is convincing. The key objective is to maximize the accuracy of the model's responses, irrespective of whether the external context is correct incorrect. To achieve

---

Published as a conference paper at ICLR 2025
Who won US presidential election in 2020?
What's latest Taylor Swift 2024 Aug?
Joe Biden
Midnights (2022)
Donald Trump won election against Biden accending from Times, great preiédent, concinued offeris Great Agalnn throughout his
The Tortured Poets Department is the eleverth studio album by the American singessor pwttter Thylor seift, exed AnrApr 15, 2024, the albumis miniralt sunth-pas, fallppp,, and chember pap affort weth rack
Donald Trump
Tortured Poets
Midnigght Confidence:
Joe Biden Confidence: 0.9
Department
Dorald Trump Confidence: 0.1
0,9+0.1 Joe Biden
Tortured Department Confidence: 0.6
Tortured Poets Department
know that Ice Biden won with 306 electaral vatas to Trump's 232, which was cartified by Congrest ess. The New York Times Biden's victory. Biden inaugurated Januarry 2021, and implemented policies like the American Rescue Plan address COVID-19. Therefore, my answer loe tiden
I Initially thought Midniihhs was her latest, but since my knowledge out-offis is Dec 2023, I might not know about a 2024 release. Taylor Swift typically releases albums every years, and folk-pop styles mentioned match her recent work.. Therefore my anser Tortured Poets Department (2024)
synth-pop
Closed-Book
Direct Input Augmentation
Self-guided
Rule-based Confidence Reasoning
Confidence Reasoning
Figure 1: Concept illustration of Self-guided Confidence Reasoning and Rule-based Confidence Reasoning. The goal is to enable the model to generate correct answers, regardless of input context accuracy, contrast to direct input augmentation, which often blindly follows the context.
this, it requires strong reading comprehension skills to interpret and extract accurate information as well as confidence reasoning- the capacity to self-reflect on the confidence about the model's own internal knowledge, and contrast it with the external context to arrive at the most accurate final answer.
We benchmark the current LLM's s ability to perform situated faithful reasoning on several question- answering tasks, including FreshQA (Vu et al., 2023). NaturalQA (Kwiatkowski et al.] 2019), Triv- iaQA (Joshi et al., 2017), PopQA (Mallen et al. 2023). These datasets reflect diverse set of world knowledge across domains. To simulate both scenarios in which internal knowledge outperforms external evidence vice versa, we pair each question with a correct context retrieved online and an incorrect context synthesized by LLMs. Besides leveraging automatically generated contexts of various kinds, we further examine how LLMs handle human-written incorrect contexts using Red- dirQA- a new dataset that we create by sourcing incorrect contexts from Reddit posts containing real-world factual mistakes that lead to incorrect answers to corresponding questions. Concurrent work, ClashEval (Wu et al., (024), addresses similar problem but uses only synthetically modified incorrect contexts and does not focus on improvement methodology.
We identify two classes of LLMM- approaches for confidence reasoning to improve situated faithfulness: self-guided confidence reasoning (SCR) and rule-based confidence reasoning (RCR). In SCR, the model is informed that external information might be incorrect and is allowed to reason either implicitly or explicitly via thain-of-though first extracts model's confidence signals regarding the internal knowledge and the external context, and then applies predefined rules to choose source.
arrive at the final conclusion. In contrast, RCR
Our experiments show that: 1) For the model with strong reasoning abilities like GPT-4o and GPT- 4o-mini, SCR outperforms RCR, while for a weaker model like Llama-3-8B, RCR performs better. 2) RCR's 's performance is limited by the noises and biases introduced during confidence extraction and rule design. 3) The effectiveness of SCR can be further enhanced through our novel method, Confidence Reasoning Direct Preference Optimization (CR-DPO), which enables the model to self- sample both correct and incorrect confidence reasoning paths and learn to prefer the optimal one using reinforcement learning. CR-DPO also improves generalizability to unseen data, boosting Llama-3-a 3-BB by +8.9% on average. Our study benchmarks this problem, introducing robust methods and insights, making the enhancement of LLMss situated faithfulness a promising research direction.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Language models with input augmentation (Lewis et al. 2020; Asai et al. 2024 are valuable for many application scenarios but are susceptible to noisy contexts (Petroni al. 2020; Shi et al.
2023). Past research have explored tailoring LMs to utilize only relevant contexts (Li et al. 2203a Weston Sukhbaatar| 2023, Yoran et al., 2024). These approaches however do not account for contexts that are relevant but contradict LLMs' parametric knowledge. While Pan et al.] (2023) touch on the issue of directly incorrect contexts, their focus remains on the RAG setup and lacks dedicated exploration of the LLM's internal behavior.
Facing such knowledge conflicts, LLMs exhibit enigmatic behaviors. They can be too receptive to external evidence (Xie et al.] (024) and over-reliant parametric knowledge (Longpre et al., 2021) at the same time. Shi et al. (2024) propose a fusion-in-decoder approach to trust context when sources are known to be reliable. Yu al. (2024) resolve knowledge conflict by training separate classifier to remove untruthful information from context as pre-processing step. Doing so the level of surface forms fail consider various semantic granularities. Concurrent to our work, ClashEval (Wu et al., 2024). FaithEval (Ming et al. 2024) and DynamicQA (Marjanovi' c et al. 2024 also benchmark this problem but don't focus on providing an effective solution.
LLMs can estimate confidence in their own responses (Kadavath et al. 2022 Xiong et al.) 2023
Resolving in-context knowledge conflicts introduces additional challenges as involves not only confidence estimation, but corroborating the external with internal to make reasoned comparison to final conclusion. See more related work Appendix
3 PROBLEM SET-UP
3.1 FORMULATION
Let M represent QA model built on top an L.M 9 (i.e. . can be either a standalone L.IM or an LLM-based pipeline). Given QA instance = (g, c). where q denotes question and denotes a context that contains an answer to q the model M generates an answer a ~M(,,c).
We define the correctness of the model's answer with the function y(a) {0, 13, where y(a) indicates that the answer a is correct (e.g., an exact match with the ground truth) and y(a) = indicates that the answer is incorrect. Let dg be the answer entailed by the context c, we define the correctness of the context e with respect to the question using y(c. q) E (0, 1). where y(c, q) = when y(a.) 1, y(c,g) base LLM G without any context, where ain = (
when y(ae) 0. Similarly, let aín be the internal answer from the
Situated Faithfulness refers to the ability of the model M to provide correct answer regardless of whether the external context c is correct or incorrect. Therefore, improving situated faithfulness means maximizing the accuracy of the model's answer, regardless of whether the context correct or not. Specifically, this involves maximizing the following probabilities:
Pr(y(a) = y(c, q) = 1, (a.)) 1): The probability of providing a correct answer
when both the context and the model's internal answer are correct.
Pr(y(a) v(c, q) 1, (..) 0): The probability of providing correct answer when the context is correct, but the model's internal answer is incorrect.
Pr(y(a) = v(c,g) q) 0, ((a,)) 1) The probability of providing correct answer when the context is incorrect, but the model's s internal answer is correct.
Following Yu et al. (0024), we quantify situated faithfulness with metries:
Accuracy Given True Contexts (Acc:): the probability that the model outputs a correct
answer given correct context: Pr(v(a) = 1 y(c, ④) = =) E [0, 1]
Accuracy Given False Contexts (Acc,): the probability that the model outputs a correct
answer despite context being wrong: Pr(y(a) = 1 v(c.g) = 0) E [0, Pr(y(a.))) = 1)]
We assume that always contains an answer to q, which be correct or incorrect.
3

---

Published as a conference paper at ICLR 2025
Overall Situated Faithfulness: SF
Aoes
[0, Pr(v(am )=1))))))))
The upper bound of Ace; is 100%, as the model could theoretically achieve perfect accuracy when all contexts are correct. In contrast, Acc, is limited by the model's internal accuracy, since additional wrong contexts can't help the model solve questions that it cannot answer. For overall situated faithfulness, the upper bound is the average of 100% and the model's internal accuracy.
3.2 DATASETS
To comprehensively benchmark the situated faithfulness problem, we select a diverse set of question- answering datasets across various domains. Each question is tested twice, paired with both a correct and an incorrect context, incorporating contexts generated by both language models and humans.
3.2.1 REDDITQA
While many QA datasets accompanied with correct human-generated documents containing valuable supporting evidence, no existing dataset focuses on human-written documents with inac- curacies, raising questions on how models react natural false contexts. To fill in this gap, we contribute a new dataset sourced from existing Reddit posts, which often contain unverified, inac- curate claims. Concretely, we first apply a claim detector to Reddit post summaries ((Völske al. 2017) filtering out those without significant factual claims. Then, GPT-4o evaluates whether the claims may contain inaccuracies. For claims flagged as incorrect or uncertain, GPT-4o generates self-contained, multiple-choice world-knowledge question, providing the correct answer, wrong answer, and two plausible alternatives. Next, we use natural language inference model to check the incorrect post indeed provides the wrong answer to the question.
Following the automated process, we conduct a human evaluation to ensure question validity by verifying: 1) Is the question concise, self-contained, and asking for verifiable f'cctall information based on world knowledge? 2) Is the answer supported by authoritative sources (e.g., government websites, Wikipedia, textbooks)? 3) Does the Reddit post introduce inaccuracies leading to wrong answer? 4) Are alternative answer choices clearly incorrect and misleading? 5) Is verified evidence from online sources correctly paired with the context for the QA instance? If any criteria unmet, annotators attempt to correct errors. For non-correetable data points, annotators discard them. Step- by-step guidance and other annotation details can be found in $
3.2.2 OTHER QAS
We benchmark the ability of current LLMs to perform situated faithful reasoning across several question-answering tasks, covering diverse domains and varying difficulty levels.
NaturalQA (Kwiatkowski et al. 2019) An open-domain dataset designed to simulate real-world search processes by focusing on naturally occurring questions.
TriviaQA (Joshi al. 2017): Another open-domain dataset that consists of relatively easy questions, as the required knowledge is already memorized by most LLMSs
2023): Features open-domain questions of varying popularity, in- cluding low-popularity questions that models might not have memorized.
PopQA (Mallen et al.
FreshQA (Vu et al. 2023): Contains questions with varying levels of time sensitivity and different complexity, assessing both factuality and reasoning. To align with other QA datasets, we filter out data without a false premise.
ClashEval (Wu et al. 2024y: a contemporary dataset with world knowledge questions from multiple domains, each paired with both correct and perturbed incorrect contexts.
For TriviaQA, NaturalQA, and FreshQA, we retrieve conrect contexts from relevant websites and verify them with a natural language inference model. needed, GPT-4o generates supporting con- texts. Wrong contexts are created by having GPT-4o modify correct contexts to lead to incorrec answers. For PopQA, we use ConflictQA contexts with noise filtering. See $ for details.

---

Published as a conference paper at ICLR 2025
4 METHODS
We examine two sets of approaches to enhance LLMs' situated faithfulness. The first is self-guided confidence reasoning (SCR), where the LLM is aware that the external context may be incorrect and estimates the confidence of both its internal knowledge and the provided context, reasoning through to the final answer. In contrast, the second approach, rule-based confidence reasoning (RCR), also involves estimating confidence for internal and external knowledge, but the final decision is made according to predefined external rules rather than the model's own reasoning.
4.1 SELF-GUIDED CONFIDENCE REASONING
Implicit Self-Guided Confidence Reasoning (ImplicitSCR) The model is prompted that the pro- vided context may be incorrect and must rely on its own judgment to assess the reliability of the context before providing final answer. To encourage the of internal knowledge, the context is presented after the question. Our pilot experiments ($E) indicate that this ordering structure biases the model to rely more on prior knowledge, reducing susceptibility to misleading context. After that, the model implicitly reasons about confidence during the decision-making process and outputs only the final answer. (See (H.2 for the actual prompts used).
(ExplicitSCR) In this method, the model first generates separate answers based on its internal knowledge and the provided context (via two different prompts). The model then performs verbalized confidence reasoning using chain-of- thought process. begins by evaluating the confidence in its internal answer, reflecting on how derived this answer based on known facts. Next, it assesses the reliability of the external context by cross- coossreffrenniin it with the facts supporting its internal answer. The final answer is chosen by reasoning through both perspectives, balancing internal and contextual confidence. (See H for the details of the prompt.)
Explicit Self-Guided Confidence
Confidence Reasoning Preference Optimization (CR-DPO) We train the model to learn verbal- ized confidence reasoning by optimizing its preferences between a pair of self-sampled reasoning paths. First, when the model's internal answer incorrect, we provide it with correct external context and ask it to reason why the context is correct and its internal answer is wrong, resulting in a chosen reasoning path. Conversely, we lie to the model by telling it the context incorrect and its internal answer is right, asking it to reason accordingly, generating rejected reasoning path. When the model's ss internal answer is correct, we repeat the same process, comparing correct in- ternal reasoning with external context. (See $H.7 Prompts for these two processes)This approach allows the model to learn preferences between reasoning paths through direct preference optimiza- tion (Rafailov et al. 2023). Additionally, to encourage diversity in reasoning, we use dual sampling: for each instance, we sample two pairs of reasoning paths using different prompts and in-context examples. Following Pang et al. (2024), we apply negative log- ligeliiel to DPO loss improve the optimization of the reasoning process. (See $B for implementation details)
4.2 RULE-BASED CONFIDENCE REASONING
These approaches define confidence estimation for both internal and context-based answers. repre- sented as numerical scores or self-evaluations. The reasoning process is then carried out by pre- defined heuristics, which determine the final answer by comparing these confidence estimates. All methods here follow the same procedure of 1) generate internal answer and a context answer (completely faithful to the contexts even it's wrong) separately; 2) they estimate confidence for each answer; 3) and utilize rules to select.
Internal Evaluation (InternalEval) The LLM self-evaluates the correctness of its internal answer with the prompt in $H.4 If the self-evaluation suggests the answer is correct, the internal answer is selected; otherwise, the context-basee answer is used.
Context Evaluation (ContextEval) The model assesses the correctness of the context relative to the question (See $H.5 for the prompt). If the context is deemed correct, the context-based answer is used; otherwise, the internal answer is selected.
Internal Confidence Thresholding (InternalConf) The LLM selects its internal answer if its con- fidence exceeds predefined threshold; otherwise, it defaults the context-based answer. The
5

---

Published as a conference paper at ICLR 2025
threshold can be set to 0.5 tuned on calibration set if available. Confidence can be derived from sequence probability or other elicitation methods like self-consistency. ActiveRAG (Jiagg et al. 2023) reduces InternalConf in non-iterative short-form generation. Unlike ActiveRAG's teken-level thresholds, our implementation uses answer-level thresholds, offering slight empirical advantages. See DD for a comparison.
Context Confidence Thresholding (ContextConf) The LLM assesses the reliability of the external context. If the confidence in the context surpasses specified threshold, the context-based answer is chosen; otherwise, the internal answer is used. The threshold could be determined in the same way as Internal Conf. The confidence score could be estimated by the sequence probability of the answer when given the context or celficonnis as well.
(Calibrated) Token Probability Correction (TPC) (Wu et al., 2024) compares the confidence scores (mean token probabilities) between the model's internal answer and the context answer, se- lecting the one with the higher score as the final answer-refene Since internal answer probabilities are generally more uniform, while context-based probabilities tend to be right-skewed, this method can further improved by comparing percentiles of the con- fidence scores, rather than the raw values, resulting in calibrated token probability correction.
as token probability correction.
4.3 OTHER BASELINES
Direct Input Augmentation The simplest and yet common implementation of RAG systems is directly concatenating a question with the corresponding contexts. We follow the same structure of the generative prompt described in Yu et al.] (2024) where a simple instruction is inputted first, following the external context concatenated with the question, forming prompt that is then provided
to the model. The model is instructed to leverage the external context in generating the answer. We utilize 3-shot to make sure the answer is in the right format. (See $ for the prompt)
Truth Aware context-selection Similar to context evaluation, Truth-Aware context selection Yu ctt al. 2024) filters out incorrect parts of the context at a granular sentence or token level using a classifier. Unlike rule-based methods that choose between internal and context-based answers, this approach feeds the filtered context back into the LLM to generate the final answer. However, relies on hidden states, which not accessible in proprietary models like GPT-4, and requires in- distribution training data, making it less comparable to methods that do not. Therefore, we adopt an alternative: LMS remove the untruthful sentences (TACS-LR) (See for the prompt), and the refined context is used to produce the final answer.
5 RESULTS
5.1 EXPERIMENTAL SET-UP
We conduct experiments using three models: GPT-4o mini, GPT-4o, and Illam-33 8B. Our base- lines include Direct input augmentation (DIA), Truth-Aware Context Selection (TACS). We com- pare these baselines with both implicit and explicit self-guided confidence reasoning (ImplicitSCR and ExplicitSCR), and a suite of rule-based methods including Token Probability Correction (TPC), Internal Confidence Thresholding (InternalCnf), Context Confidence Thresholding (ContextConf), Internal Evaluation (InternalEval)), and Context Evaluation (ContextEval).
We also report the results of closed-book method (i.e., internal answer accuracy) to provide insight into the model's inherent performance on different datasets. We note that this evaluation does not assess the model's situated faithfulness.
5.2 LLMS CAN Do BOTH SELF-GUIDED AND RULE-BASE CONFIDENCE REASONING
Self-Guided Confidence Reasoning (SCR) Excels on GPT-4o and GPT-40 mini: As shown in Table T
SCR methods consistently achieve the top-2 accuracies across most datasets for both GPT- 4o mini and GPT-4o, outperforming direct input augmentation by significant margins (+17.8 for GPT-4o mini, +24.2 for GPT-4o), as well as TACS and other rule-based methods. This highlights L.MMs' ability to evaluate and compare internal knowledge with external context for accurate

---

Published as a conference paper at ICLR 2025
GPT-44-m
RedditQA
FreshQA
ClashEval
TriviaQA
PopQA
NaturalQA
Total
Method
UV
DV
OV
DV
UV
BA
UV
Closed-book
80.7
53.0
20.0
81.7
56.7
62.0
59.0
DIA
12.5
54.3
96.3
233
95.3
12.3
54.
54.4
10.3
49.5
TACS (1.R)
93.8
16.5
55.2
86.3
4.6
45.5
76.3
14.3
45.3
92.3
16.0
54.
76.3
14.3
45.3
86.0
15.7
50.9
49.4
bessed Confalncc Ressoning (Evatuation based)
ContexiEval
90.3
53.4
71.9
77.3
35.7
56.5
86.7
11.7
49.2
92.3
47.4
69.9
91.0
53.0
72.0
86.6
36.3
61.5
63.5
Internaiivval
88.6
77.2
82.9
83.3
50.0
56.7
67.0
17.0
42.0
58.7
71.7
80.2
78.0
48.3
03.2
72.3
57.3
64.8
05.0
Rate-besed C Coufidence Reaanning (Confalenrs -waeee)
TPC
92.0
25.6
58.8
85.7
22.7
55.7
82.7
20.3
51.5
94.7
90.7
67.2
95.0
17.7
56.4
87.0
54.3
640.7
58.4
ContesiConf
48.4
56.4
IntersllCeef
82.3
79.5
80.9
83.6
58.3
61.0
64.6
42.6
89.6
77.0
83.3
84.0
67.0
84.0
49.3
66.7
66.9
Self
guided Coundence Roasoning
ImplicnSC
91.5
79.0
85.3
93.7
26.3
0000
89.5
25.3
57.3
96.5
48.0
72.2
97.0
48.3
72.7
82.0
50.0
66.0
08.9
ExpliciSCR
92.0
73.9
$3.0
82.7
47.0
64.9
82.0
17.0
49.5
92.3
71.
818
93.0
51.0
22.0
80.
51.0
65.7
69.5
GPT-40 Method
ReddiOQA
FreshQA
ClashEval
TriviaQA
PopOA
NaturalQA
Total
TR
FA
OV
TR
FA
OV
TR
FA
OV
TR
FA
OV
TR
FA
OV
TR
FA
OV
OV
Closed-book
84.1
63.1
39.0
90.7
$1.0
66.7
70.8
DIA
94.9
12.5
53.7
93.7
577
40.7
84.7
24.7
54.7
96.0
23.3
59.7
97.3
12.3
54.8
88.7
12.7
50.7
53.9
TACS (L.))
96.0
17.0
56.5
81.3
9.3
45.3
92.7
2577
542
96.0
26.7
61.4
96.7
21.3
59.0
88.3
20.7
54.5
55.1
Kute- of Confideenc Rensowing (E (Evollationn Baned)
ConsextEvall
92.6
66.5
79.6
87.7
44.0
65.9
$1.3
23.0
$2.2
71.3
82.8
96.7
73.3
55.0
86.7
51.3
09.0
72.4
Internallival
79.5
60.3
67.3
Rule-burrd Confilence Reasoning (Confaience Bused)
TPC:
94.3
71.6
83.0
87.0
39.7
63.4
77.0
39.0
58.0
94.3
75.3
84.8
96.7
66.7
81.7
84.0
50.3
67.2
73.0
CoutextCooff
93.8
50.2
77.0
78.7
57.0
57.9
70.7
50.0
50.4
94.6
43.0
68.8
95.7
26.3
61.0
78.7
36.7
57.7
62.1
InteralCoond
89.8
78.9
84.4
83.7
50.0
66.9
61.6
40.6
51.1
93.0
88.0
90.5
88.7
78.7
83.7
78.6
59.6
69.1
74.3
Self guided Confidence Reasoning
ImpliciSCRR
92.6
75.6
$4.1
82.0
51.0
68.0
86.0
44.3
65.2
94.7
65.7
90.2
95.0
76.7
86.4
83.3
66.7
75.0
78.1
EpliciSCX
93.8
85.3
35.3
333
58.2
84.3
89.5
92.7
78.0
33.4
78.7
76.4
Table 1: Main Results of GPT-4o mini and GPT-4o. "TR" represents accuracy given all correct contexts, "FA" represents accuracy given all wrong contexts, and "OV" represents the overall sit- uated faithfulness, the main metric we focus on. The best number is bold and the second best is underlined. The main findings are: 1) SCR methods generally outperform other methods across datasets 2) RCR methods also outperform basic baselines, but worse than SCR.
ReddRQA
FreshiQA
ClashEval
TriviaQA
PopQA
NaturalQA
Total
Method
FA
OV
OV
FA
OV
Closed book
76.7
32.7
19.3
69.7
43.0
42.7
47.4
DIA
87.3
TACSILK)
81.0
8.7
44.9
71.7
30
37.4
76.3
123
44.3
90.3
13.3
51.8
93.7
16.7
55.2
81.0
8.7
44.9
46.4
Ruie-boed Confience
Reosowing (Fsofustion--uooo
Contextiool
85.2
42.0
63.6
50.0
26.7
35.4
71.5
11.3
41.3
84.0
42.0
63.0
89.0
40.7
64.9
68.0
27.3
47.7
53.1
InternllEvvll
84.7
66.5
75.6
55.7
24.0
39.9
53.3
16.7
35.0
79.0
61.7
70.4
69.0
40.0
34.5
60.3
58.7
49.5
54.1
Rule
bused Confdonce Reazening
TPC
88.6
28.4
58.5
78.0
15.0
46.5
76.6
17.6
47.1
92.6
27.6
60.1
94.0
20.7
57.4
80.7
21.6
51.2
53.5
ContextConf
89.8
23.9
56.9
68.7
42.5
65.7
55.8
73.0
21.0
50.0
Interma/ooof
79.5
75.0
77.3
69.3
23.7
46.5
68.0
21.3
44.7
83.0
64.6
73.8
75.0
42.0
53.5
69.6
33.0
51.3
58.2
Self
Self-guided Confedence
Reasoning
ImpliciSCR
83.2
47.1
65.2
89.3
11.0
50.2
79.0
15.7
474
90.0
23.3
56.7
94.7
33.0
63.9
82.0
19.7
50.9
55.7
ExplictSCR
86.4
43.8
65.1
66.0
19.0
42.5
79.0
11.3
453
86.7
34.0
60.4
82.7
53.7
58.2
76.3
23.0
49.7
53.5
CR-DPO
49.7
23.2
Table 2: Main Result Llama-3-8B. The SCR methods don't Io perform well compared to InternalConf. However, CR-DPO brings significant improvement to ExplicitSCR. The best number is bold and the second best is underlined.
swers. TACS performs poorly because LLMs struggle to filter context effectively, requiring in- distribution training data and limiting its out-of-the-box solution.
LLMS Perform Well with Rule-Based Confidence Reasoning Rule-based methods also show sub- stantial improvements over direct input augmentation. Among these, "InternalConf" performs the best, outperforming "ContextConf." This indicates that LLMS are better at estimating the confidence of their internal knowledge than that of external information.
Enhancing SCR Through CR-DPO: While SCR generally outperforms RCR methods on GPT-4 series, it underperforms "InternalConf" on Llama-3-8B (Table with Acc, dropping significantly
7

---

Published as a conference paper at ICLR 2025
in TriviaQA (64.6% - 34.0%) and RedditQA (75% 43.8%). This indicates that while Llama-3- 8B demonstrates some internal confidence estimation, it tends to favor external context when it is available. We attribute this bias in part to the model's 's weaker reasoning abilities. As shown in Figure 2. when presented with misleading context, the ExplicitSCR model correctly lists the specific years Richard Daley served in Chicago. However, it acknowledges that it lacks information about him being mayor in another city. If the model possessed strong reasoning abilities, it should recognize that, regardless of whether Daley was mayor in another city, his 21-year tenure as mayor of Chicago makes Chicago the correct answer. By applying our proposed Confidence Reasoning Preference Optimization (CR-DPO) method, the model is able to detect the deceptive context and correctly identify the answer. CR-DPO significantly improves the model's 's SCR performance, boosting it by +8.9% ( and demonstrates robustness across various context types ($J).
Desttption
Correct Annsser:
Internal Annwar Ohiogoo
Wrong Contert Answer Phitddlphi
Explott5K
lindslly thought Richan waley the mopor of Oheage for years os have general know edge about the Daleyfersi fernly's inokement in Chicgg polf fichers Dallyy
of
Dullyy mayer Chicgo from
1976, and sand Dully was the major Chicago 1960 1011. Howewer Idon't
tied that provides il about ba meyora serm, including to advewements and che langan feced. The docsment also indiude informatoon about the dhy's hotory ea cur, and bedrorks, wrich suggers that the authorhe deep undentinnd ng of buseeti Theeworr thet leee the document in securate, and Fkchard Datey was inderded the mayor Phi bccepniss for 21 years. Thenetrre, the answeric Philode phio
The document preaents
01:Pno
linital' enowered thet Richard De westhe mayor Chicago 21 beceuse krow tichard j. Daley ved the meyer of Cricago from 1905 1075, sotal of yours. aloo know that thsse son, Richard id Grlls served as the mayer of Chicage from 1060 so 2013 a tond 22 yoars. The docament claims that Richrrd Dakey war the mayer Phi aetliiaeeea piis for 21 years whichic conoaeee with my krouuedge of actuid treure in Dhinggo.
The
docb
information sbown Hichorl M. Dalee's noporll tenure
mpiidmentt Philodelanis ches dffies
indluding Now York, los Angoles, and Frandsso,, do hininn mavoo who dil seree for 23 yoars, trphinng that Hiladdphii is the only that mutths Daley's lamand Homenus this is aor ham es Michh ltt Duky nows senval a the mayor of lelleciepe - Thandfore dne Tul arower ne Chicrgs
An example where SCR fails to maintain situated faithfulness, but CR-DPO succeeds. In this case, the model's internal answer is correct, but the context is misleading. Before training with CR-DPO, the model is misled despite correctly listing factual details. After training, it successfully arrives at the correct answer.
Figure 2:
5.3 WHAT LIMITS RULE- BASED CONFIDENCE REASONING?
While RCR methods seem intuitive, their performance is often constrained by errors and biases in rule design and the extraction of confidence signals. We demonstrate these limitations through experiments the Llama-3-8BB with similar findings observed in the GPT-4o series (
Llama-5-Hb
ReddiiQA
FreshQA
ClashErral
TriviaQA
PopOA
NaturalQA
Total
Method
FA
OV
OV
OV
OV
EA
OV
FA
TPC
$3.6
28.4
58.5
78.0
13.0
46.5
76.6
17.6
47.1
92.6
27.6
60.1
94.0
20.7
57.4
80.7
21.5
51.2
53.5
-Pereetaa
82.3
70.4
76.4
75.3
15.7
45.5
19.0
19.7
39.8
92.0
29.0
60.5
85.0
37.0
61.0
22.3
34.5
53.5
56.0
Contextoon
89.8
23.9
36,9
68.7
16.3
42.5
65.7
17.7
41.7
91.66
19.0
55.3
93.3
18.3
55.8
75.0
21.0
48.0
50.0
-TrrssholT
88.1
85.7
30,0
$7.9
93.0
55.7
75.0
21.0
48.0
Internaoooon
79.5
75.0
77.5
09.3
23.7
46.5
08.0
21.3
44.7
83.0
64.6
73.8
75.0
42.0
58.5
69.6
33.0
51.3
58.7
-TaresboldT
83.5
69.9
76.7
$4.0
18.0
$1.0
74.0
19.0
16.5
83.0
64.5
73.8
90.0
35.0
62.5
76.3
29.7
53.0
60.6
aECECab
69.9
49.7
67.0
71.7
90.0
35,0
62.5
80.3
53.5
60.8
76.7
75.6
76.2
57.0
28.0
42.5
56.3
20.3
38.3
78.0
67.3
72.7
67.3
41.7
54.5
60.9
39.7
49.9
55.7
-SC-TT
77.2
74.4
75.8
79.3
21.0
50.2
77.0
18.3
19.7
78.0
67.3
72.7
81.0
41.5
61.3
79.0
30.7
$4.9
60.7
Table 3: Enhancing RCR Methods Using Additional Calibration Sets: Methods with a grey back- ground represent attempts to improve upon the original RCR methods. Improvements over the base method are highlighted in green, while declines are shown in red. "Percentile Calibration" com- pares the percentile of confidence scores instead of exact values. "ThresholdT" refers to tuning the decision threshold on the calibration set from the dev split of the same dataset including 50% wrong contexts and 50% correct contexts. "ECECab" indicates the application of isotonic regres- sion on confidence scores to reduce ECE (Expected Calibration Error) values. "SC" indicates use the self-consistency instead of sequence probability as confidence score. "SC+TT" self-consistency combined with Threshold Tuning. "SC" performs badly but can be improved with threshold tuning.
Rules can be biased or flawed: For ContexiConf and InternalConf, the decision rule- rule-selecting an answer based on whether the confidence score exceeds a fixed threshold can be optimized through threshold tuning on dedicated calibration set (See $D.3 for InternalConf sensitivity to confidence threshold). As shown in Table tuning the threshold improves ContextConf and InternalConf per- formance in partial datasets. Similarly, Token Probability Correction (TPC) rule can be refined by
8

---

Published as a conference paper at ICLR 2025
TriviQQA
PopQA
NaturaQQA
RediiiQ
FressQA
ClashEval
ExplicaSCR
60.4
58.2
42.5
CR.
73.2
CR-DPO-ST
68.4
65.4
53.4
50.4
-COT
71 21.8)+2.8)
60.5(-4.0)
38.7(-16.2)
72.86-3.9)
34.21-19.2)
47.0-3.4)
-DPO
50.4(-9.)
53.7(-11.7)
48.2(-67)
76.8(+0.1)
52.21-1.21
46.0(414)
-] task
72.21+3.8)
66.4(+100
57.70+2.)
75436-149
50.9K-2.)
47.9K225)
Saskk
1)
64.56099
47.96-7.0
43.70-9.71
48.36-2.1)
33skks, 88
68.7(40.))
62.5(-2.9
46.3(-8.6)
(7.96-8.)
41.2(-12.2)
45.4(-5.0)
Table 4: CR-DPO Ablation Study. "ESCR" refers to the baselines without training. "CR-DPO" refers to the best-tuned model, trained on confidence reasoning paths sampled from four tasks: Triv- iaQA, NaturalQA, ConflictQA, and RedditQA. Results within yellow cells represent performance on out-of-distribution (OOD) tasks, while results in green represent in-distribution tasks. "CR-DPO- ST refers to a variant of CR- RRDDPO that uses only a single pair of chosen and rejected answers for each instance, whereas CR- RRO uses two pairs. The methods below, shown with a grey background, are ablation studies compared to CR-DPO-ST: "-Co'T" removes the explicit chain-of-thoughh confi- dence reasoning, making equivalent Direct Preference Optimization (DPO) applied to Implicit SCR (DPO-ISCR). "-DPO" replaces DPO with Supervised Fine-Tuning (SFT). "-1 Task" removes one training task (RedditQA). "-3 Tasks" removes 3 training tasks and only keeps TriviaQA as the training task. "-3 Tasks, SS" uses only TriviaQA for training, and the reasoning path is not self-sampled, but sourced from a GPT-4o model. The numbers highlighted in dark green indicate improvements, while those in red represent decreases, relative to the performance of "CR-DPO- DPO-ST."
comparing the percentile of the confidence scores rather than the raw values, resulting in a cali- brated token probability correction (CTPC) approach. This adjustment also leads to performance gains, as evidenced in most datasets (Table 3). While these improved rules boost the performance of RCR methods, they still struggle with limited generalizability across datasets and continue to underperform compared to SCR on GPT-4o series, even with the advantage of using a calibration set (Table/o). This is partly due the new biases introduced by the updated rules. For instance, in CTPC, the assumption is that a context-basee answer with a higher percentile in the calibration set more reliable than an internal answer with a lower percentile. This assumption often breaks down in certain scenarios, such as when the calibration set contains an equal mix of correct and incorrect contexts, and the model's internal knowledge is highly accurate. For a well-calibrated model, an internal answer with low percentile confidence score should still be preferred to context answer with a top percentile confidence score. In these cases, relying on percentile-based comparisons can undermine the method's generalizability.
Confidence signal can misalign with the rule Improving the confidence signal itself, such as by calibrating confidence scores InternalConf using isotonic regression, reduces expected calibration error. However, this doesn't 't always translate to better performance on situated faithfulness as shown in Table B..g. TriviaQA, RedditQA). A well-calibrated confidence score doesn't 't necessarily align with the rule's ' goal of maximizing accuracy. For instance, a model always predicting a 50% confi- dence score for answers in dataset in which the model achieves 50% accuracy is well-calibrated, but no matter how thresholds are adjusted, InternaConnf's overall accuracy will remain capped at 50%. Thus, even a better-calibrated signal may not lead to higher accuracy it doesn't 'It fit the rule. (See $D.2 for detailed calibration scores of RCR methods)
Confidence signal can be noisy and biased For InternalEval and ConlextEval, the rules are directly aligned with the confidence signal - the correctness of the internal or context- context-based answer. In theory, take InternalEval as an example, a perfect self-evaluator that correctly assesses the internal answer's accuracy could optimize performance. The model would use the context when its internal knowledge is incorrect and rely on its own knowledge when correct, leading to optimal results. However, in practice, self-evaluation is noisy and biased. Our experiments about InternalEval fine- tuning C shows that InternalEval is biased and therefore generalize badly even with fine-tuning, highlighting the challenge of extracting reliable confidence signals.
In summary, RCR methods are hindered by errors and biases in rule design, confidence extraction, and by misalignment between rules and confidence signals. In contrast, SCR avoids these pitfalls by inherently performing confidence estimation and reasoning directly in the text space, where the model operates more effectively.

---

Published as a conference paper at ICLR 2025
5.4 CONFIDENCE REASONING PREFERENCE OPTIMIZATION ENHANCES SCR
SCR can be further improved with CR-DPO. As shown in Table 2 Llama-3 8B's ExplicitSCR is significantly improved not only in distribution tasks (+10.4% on average) but also generalized to out-of-distributiin tasks (+5.9%) including FreshQA and ClashEval. CR-DPO works because:
CoT enables better confidence reasoning learning CR-DPO allows the model to explicitly learn how to reason through confidence assessments by leveraging COT. Since the model verbalizes its reasoning process, it learns to identify what constitutes effective confidence reasoning. In contrast, when remove the explicit COT confidence reasoning (" ("-COTT" in Table 4. which is equivalent to applying DPO to Implicit SCR (DPO-ISCR), where the model directly learns to predict the final answer without an explicit reasoning chain, the results were suboptimal (-7.5% in average). The absence of a clearly articulated reasoning process forces the model to infer the reasoning principles solely from outcomes, making learning less effective and prone error. This leads a more challenging and inefficient learning process-similee to relying on inductive guesswork without being able to observe the reasoning steps.
CR-DPO encourages LLM to explore various reasoning paths CR-DPO allows the model to learn from the preference between different paths through reinforcement learning instead of over- fitting to one single reasoning path. In contrast, supervised fine-tuning on Explicit SCR ("-DPO" in Table 4 leads to a large performance drop (-5.5% in average) because it overfits the model to one correct reasoning path. The CR-DPO can be enhanced by sampling more pairs of chosen and rejected reasoning paths for each question-context pair. As shown in the table ("CR-DPO" versus "CR-DPO-ST") where each instance have two chosen-rejected reasoning pairs sampled with dif- ferent in-context examples. The performance on the in-distribution datasets are improved (+2.2% with slight sacrifice on the out-of-distribution performance (-0.5%).
Multitask training helps OOD generalization In Table 4 if we remove the RedditQA dataset unseen datasets (i.e., FreshQA, ClashEval), despite an
from training, the performance drops increase in in-domain tasks. If we further reduce the number of training tasks by only keeping one training dataset TriviaQA, the performances across all tasks decrease. This suggests that increasing the diversity in training tasks generally improve the confidence reasoning for both in-distribution and out-of autoof-distriuution tasks. However, smaller LMs such as Llama-3-88 might sacrifice in ni-iistriuution performance when fitting to certain novel training data, likely due to their limited learning capability.
Self-sampled reasoning paths are better than reasoning paths from a stronger model Contrary to the common belief that using training data from stronger models leads to better performance (Li) al., 2023b), in confidence reasoning, learning from the model's own reasoning paths yields bet- ter results. This discrepancy arises because confidence reasoning relies on corroborating external facts with the model's internal knowledge, and small-scale fine-tuning is unlikely to introduce new knowledge to the model. The comparison between "-3 Tasks, SS" (where reasoning paths are sam- pled from GPT-4o) and "-3 Tasks" (which uses self-sampled reasoning paths) in Table 4 shows a performance drop across most tasks when using reasoning traces from GPT-4o. Since GPT-4o does not share knowledge with Llama-3-8B. the reasoning paths generated by the stronger model inad- vertently cause the smaller model to hallucinate or make incorrect inferences, hurting performance.
6 CONCLUSION
In this work, we address the challenge of making LLMs situated faithful to external contexts. We benchmark their performance across various QA datasets, pairing with both correct and incorrect contexts. Additionally, we contribute RedditQA, a new dataset featuring human- -enaeregee erro- neous contexts, to enable a more comprehensive analysis. We propose two classes of approaches, Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR), to help L.MSs reconcile external knowledge with internal knowledge for more accurate answers. Our find- ings show that models with strong reasoning abilities, such as GPT-4o and GPT-4o mini, excel at SCR over RCR, while smaller models, like LLaMA-3-8B. RCR performs better than SCR. We observe that RCR is hindered by noise and biases in confidence estimation and rule design. Within SCR, we propose Confidence-Ree Reasonnng Direct Preference Optimization (CR-DPO) to improve gen- cralization. Our work provides valuable benchmarks, robust experiments, and insights for advancing L.LMs' situated faithfulness for future research G
10

---

The dataset is available https://hugginffacecom http://gn....eoottttttttteeeenne SituatedF thfuutessailffluunsssvvllll and the code fikerinnkikk/Sttuteefffffttfffnn Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank members of the NLP group at Duke University for fruitful discussions. This work was supported by NSF award IIS-2211526.
REPLICATION
can be found on https://github..cm
REFERENCES
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen tau Yih. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint aiviv: 2403.03187, 2024.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes, Reading Wikipedia to answer open- domain questions. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870- 1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ 11/117-1171. URL htttps://aclantholllt
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yo Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. ArXiv, 2ls/2305.06988, 2023, URL https:://apt www.enniccccollrrorrrgooooooe/ee wwwp:rrrgg//o/uoooieeee 2558: 258615731
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Compu- tational Linguistics. doi: 10.18653/v 1tt.//1711111111 URL/https:///claatthhl/
org/P17-1147
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Saafield-Dodds, Nova DasSarma, Eli Tran-Johnson. Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know. arXiv preprint arXiv: 2207.05221. 2022.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. Realtime QA what's the an- swer right now? Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: / nips. cc paper Files/paper// 2023/ and Benchmarks. html
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 16.1//// 00276. URI.
http:////ac.anthoooooooo////////
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. aetrieval-augmented generation for knowledge-intensing NLP tasks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan,
11

---

Published as a conference paper at ICLR 2025
and Hsuan-Tien Lin (eds.). Advances in Neural Information Processing Systems 33: An- mual Conference on Neural Information Processing Systems 2020, NeurlPS 2020. Decem- ber 6-12, 2020, virtual. 2020. URL http:: (20
www.:/ng..eerrrss.c...ooo wttppsscccppp/////////////pa httpsc//capere//////asahha h020/hash/ Abstract.htm.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Fe- lix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Findings of the Association for Computational Linguistics: ACI. 2023, pp. 1774-1793, Toronto, Canada, July 2023a. Associ- URL https:
ation for Computational Linguistics. doi: 10.18653/w1/2021 htwp://///200...innnggd....... //aclanthology. .rw.e...rrr..o org 2023. acl.
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic chain-of- ofttthtghttttttthh distillation: Small models can also "think" step-by-step. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 2665-2679, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023 www.looo...100.. URL
aal-llngg 150
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7052-7063, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/ 112022.emnlp-min
URL httttp::/aacaantholllot
org/ 2
emnlp- emnlp-main. 565
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non- paraamamcttic memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9802-9822, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v 1/000/11112223../ooogog..
URL
org/ //00 http//2222...11ogggg... 546
Sara Vera Marjanovi's, c, Haeun Yu, Pepa Atanasova, Maria Maistro, Christina Lioma, and Isabelle Augenstein. Dynamicqa: Tracing internal knowledge conflicts in language models. 2024. URL
https://api. semanticscholar wwtp:oorpssS::2/11000000
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey. Transac- fions on Machine Leurning Reseurch, 2023, 2023. URL httthhht///ssereeeeeeeee wtt.vvwwwww.wfoumiiiiiii jh7wH2AZKK
Yifei Ming, Senthil Purushwalkam, Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. Faitheval: Can your language model stay faithful to context, even if'the moon is made of marshmallows". 2024. URL. https::/aap..mmmtticccolo..
273186949
Ruotong Pan, Boxi Cao, Hongyu Lin, Xianpei Han, Jia Zheng Sirui Wang, Xunliang Cai, and Le Sun. Not all contexts are equal: Teaching LLMS credibility-aware generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19844-19863, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v112023 120.5///11/222..mmmmmmaaaaa 1109. URL https: http::::/allaaaathah
wttp:////2......mmm....... 1109/
Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min- Kan, and William Wang- On the risk of misinformation pollution with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1389-1403, Singapore, December 2023. Association for Computational Lin- guistics. doi: 10. 18653//11 1/2023...mmm.. wwwddinns-eenll..... URL https://aa www.aaaogyooorgg/// findings-emnllp...
12

---

Published as a conference paper at ICLR 2025
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. ArXiv preprint, abs/2404.197333, 2024. URL
https://arxiv. https////aaxiv.o//abbb////// 2404. 19733
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. AKBC, 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, Decem- ber 10 16, 2023, 2023. URL http:: nips cc/ a855405ed65c6477aa4 aesalaaeeeeee---sssaaatatrrrreee
Michael Sejr Schlichtkrull. Generating media background checks for automated source critical reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 4927-4947, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: org/2024. findings-emnlp. 2837
1188653/1/22244
findings-emnlp.288 URL htRpsttts////ccaattooo
Sagi Shaier, Lawrence Hunter, and Katharina von der Wense. Desiderata for the context use of ques- tion answering systems. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 777-792, St. Julian's, s, Malta, March 2024. Association for Computational Linguisties. URL htttts:://aaaaanlllllooooo
www.///g..........// eacl-long...... 47
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 2023. URL /ttps::/pproceedingg mlr. waar.ppres//222//hhiis
html
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-awaid decoding. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 783-791, Mexico City, Mexico, June 2024. Association for Com- putational Linguistics. doi: 10. 10.18653/v ht2.0/..aac.sssoorttoooooo URL
org/2222.naacl-short.. nwwcl-aaort..eeeec 69.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the Workshop on New Frontiers Summarization, pp. 59- 63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/11/W17-440000 URL httttpss/aaaaaaaaooooooo.orr/oop
Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoe Le, and Thang Luong. Freshlims: Refreshing large language mod- els with search engine augmentation. Annual Meeting of the Association for Computational Linguistics, 2023.
Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan Ö. Arik. Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models, 2024. URL https://apil wwwaeeiisscccccco.r.rrrrr wtt.aaat....ic.o.oooooooooooooaa
Jason Weston and Sainbayar Sukhbaatar, System attention (is something you might need too).
arXiv preprint arXiv: 23/1.11829 2023.
13

---

Published as a conference paper at ICLR 2025
Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an 1lm's 's inter- nal prior and external evidence. 2024. URL http:/ppp..emmticccccoooo....... http://chooaarroog///rrropooooo.
269157310
Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Cer- tifiably robust rag against retrieval corruption. ArXiv, abs/2405.15556, 2024. URL https:
7/api. http://wwwntttttooo.ooooooooaaoa www./oooors/o
270045527
Jian Xie, Kai Zhang, Jiangji Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models knowledge conflicts. In International Conference on Learning Representations, 2023. URL http:tt:: hhttps://ap/ wwwanti org/
Corpus10:263618821 263610324
Jian Xie, Kai Zhang, Jiangjic Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. /opps///eee//eeeee.eeeeoe http:/ne//......
OpenReview.net, 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms ex- press their uncertainty? an empirical evaluation of confidence elicitation in llms. ArXiv preprint, abs/2306. 13063, 2023, URL htttts:////xxii org/abs/ 2306. 13063
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval- augmentee language models robust irrelevant context. In The Twelfth International Conference on Learning Repre- sentations, 2024. httphttts////oeeeeeieeee net met//oounm wdw.orrmmmmmmmmmmmme
Tian Yu, Shaolei Zhang, and Yang Feng. Truth-aware context selection: Mitigating hallucinations of large language models being misled by untruthful contexts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics ACL 2024, pp. 10862-10884, Bangkok, Thailand and virtual meeting, August 2024. Association for Compu- tational Linguistics. URL
orw.e//..rg/22244/ finniigs-ddnneeeeeee 645
Ruizhe Zhang, Yongxin Xu, Yuzhen Xiao, Runchuan Zhu, Xinke Jiang, Xu Chu, Junfeng Zhao, and Yasha Wang. Knowpo: Knowledge-aware preference optimization for controllable knowledge selection in rerrieval-augmented language models. ArXiv, abs/2408.03297, 2024. URL (htps:
Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge comrup- tion attacks to retrieval- -augeval-augmented generation of large language models. arXiv preprint arXiv: 2402.07867, 2024.
A DATASETS
A. EVALUATION DATASET
We benchmark the ability of current LLMs to perform situated faithful reasoning across several question-answering tasks, covering diverse domains and varying difficulty levels.
RedditQA World knowledge multiple-choices QA datasets where in-corrected contexts are source from RedditPost
NaturalQA (Kwiatkowski et al. 2019) An open-domain dataset designed to simulate real-world search processes by focusing on naturally occurring questions.
TriviaQA (Joshi al. 2017): Another open-domain dataset that consists of relatively easy questions, as the required knowledge is already memorized by most L.MMs.
PopQA (Mallen al. 2023): Features open- peen-domain questions of varying popularity, in-
cluding low-popularity questions that models might not have memorized.
FreshQA (Vu et al. 2023): Contains questions with varying levels time sensitivity (e.g., fast-changing or static facts) and different complexity (e.g.. single-hop multi-hop), as- sessing both factuality and reasoning. To align with other QA datasets, we filter out data
without false premise.
14

---

Published as a conference paper at ICLR 2025
ClashEval (Wu et al. 2024), contemporary dataset with world knowledge questions from multiple domains, each paired with both correct and perturbed incorrect contexts.
For TriviaQA, NaturalQA, and FreshQA, retrieve correct contexts from relevant websites and use a natural language inference model to verify that the context supports the correct answer. If not, GPT-40 generates a supporting context. Wrong contexts are created by asking GPT-4o modify the correct context to lead to an incorrect answer. We then apply post-processing to filter out artifacts, such as keywords like 'fake" or 'imaginary." in the wrong contexts.
For PopQA, we use contexts from ConflictQA, However, we found many questions to be ambiguous. For example, the question 'Who directs Amy?' is unclear, as multiple versions of the movie exist, but the ground-truth answer only reflects one. To address this, we disambiguate such questions by adding more information using an LLM and augmenting the ground -runh answers to capture synonyms.
For evaluation, we use exact match relaxation for TriviaQA, PopQA, and ClashEval, where the answer is considered correct if the ground-truth appears anywhere in the response. For FreshQA, we apply the LLM-based metric from (Vu et al. 2023). In NaturalQA, since the ground-truth answers are not comprehensive,
combine exact match relaxation with the LLM-based metric. When exact match fails, the LLM metric is used as a fallback. For all datasets except RedditQA, manually sample 300 examples for testing and 100 for development. Since RedditQA has only 226 samples in total, we use 176 for testing and 50 for development. FreshQQ also has only 75 validation samples after filtering the question with false premise.
A.2 TRAINING DATA
We utilize the similar process to create training data for TriviaQA, PopQA, NaturalQA and Red- ditQA. For RedditQA, we don't have huamn verification so the data would be noisy. We create 5000 datapoints for TriviaQA. RedditQA and NaturaQA. For PopQA, we follow the split I from Yu et al. (2024) and keep 2895 training datapoints. When sampling the chosen COT reasoning path for each datapoint, we apply post-fittring step to verify whether the ground-truth answer appears the end of the selected CoT reasoning path. Specifically, we calculate the token-level recall score of the ground-truth answer within the last 3 x the length of the ground-truth answer tokens. If the recall score exceeds a threshold (0.5 in practice), we consider the reasoning path valid; otherwise, we discard the data point. experiments where we sampled one CoT reasoning path per data point, this yielded a total of 17,895 CoT reasoning paths, of which only 375 (2. 1%) were deemed invalid and discarded.
B TRAINING DETAILS
For the best trained Confidence Reasoning Direct Optimization Models, we utilize Lora training on Llama-3-8B. The training data include TriviaQA, ConflictQA, NaturalQA, RedditQA. The training process was configured with a learning rate of 5e-6 and maximum gradient norm of 0.3. The batch size per device was set to 1, with gradient accumulation over 4 steps, with distibuted on 4 A6000 GPUs. And the model was trained for epochs. Additionally, 100 warmup steps were used, and the sequence length was constrained to maximum of 900 tokens, with prompt length limit of 600 tokens. For DPO, a beta value of 0.1 was applied, using a sigmoid loss function, while RPO was configured with an alpha value of 1.0. In terms of LoRA configuration, the model was set up with LoRA alpha of 16, a dropout rate of 0.1, and a rank of 8 for parameter-efficient fine-tuning.
C INTERNAL EVALUATION FINE-TUNING
We fine-tune the LLama-3-8B to do Internal Evaluation. We fine-tune it on three tasks including TriviaQA, RedditQA and NaturalQA. Where Ilama-3-8B first answer the training data and compare with the ground-truth to get the evaluation label. Then we fine-tune the model to do the evaluation. Evaluation Results are reported below
We observe that during InternalEval, the model's self-evaluation improves only on in-distribution tasks such TriviaQA, ConflictQA, and NaturalQA, but performs poorly OOD datasets like
15

---

Published as a conference paper at ICLR 2025
TrviQQ
ConllicIQ
NaturalQA
FreshQAA
RediiiQ
ClashBivll
Internal Evalnation
FR
1R
Aec
FR
Aec
FR
Aec
FR
1R
Ace
1R
Acc
FR $1.0
TR 57.0
Acc 52.0
Llama-3
37.0
88.0
73.0
47.0
88.0
65.0
39,0
85.0
59.0
38.0
71.0
49.0
51.0
85.0
78.0
Tuned
13.0
33.0
Siowsee Fariyolors with InternalFival Medhod
TR
FA
0V
TR
EA
OV
TR
FA
OV
TR
FA
Overall
TR
FA
Overall
TR
EA
OV
Llama-5
66.5
13.1
Table 5: Interval Evaluation Results: The top block presents results on Internal Evaluation, where FR (False Recall) indicates how effectively the model's wrong answers are identified by the internal evaluator, TR (True Recall) represents the recall for correct answers, and Ace reflects the internal evaluation accuracy. The bottom block shows results for situated faithfulness using the InternalEval method, comparing the performance of LLaMA-3 with LLaMA-3 fine-tuned using InternalEval, TR represents accuracy with the correct context, while FA represents accuracy with the wrong context.
RedditQA, and ClashEval. On these OOD datasets, the evaluation results shift from bias toward evaluating answers as correct to incorrectly classifying them as wrong. This indicates that the Inter- nalEval signal is noisy and lacks generalization. While the evaluation accuracy is low on datasets like ClashEval, the model's tendency to classify its own answers as incorrect results in an unexpected improvement accuracy when given the correct document. This bias leads to an overall increase in Acc and the total score, highlighting discrepancy between the model's internal evaluation ability and situated faithfulness. This also underscores the limitations of rule-based confidence reason- ing.
D FULL RESULTS
D.1 FULL RESULTS FOR GPT-40
Full results of GPT4-o shown in Table
D.2 RCR'S CALIBRATION SCORES
To evaluate the calibration of confidence scores in the RCR methods, we present the calibration metrics Expected Calibration Error (ECE) and AUC-ROC for various models across different datasets, shown in Table LLMs demonstrate significantly better confidence estimation for their internal knowledge compared to external contexts. Additionally, while calibration improves the ECE score, this improvement does not necessarily translate to better situated faithfulness.
D.3 RCR's THRESHOLD SENSITIVITY
To demonstrate the sensitivity of RCR methods to threshold selection, we vary the threshold from 0 to for the InternalConf method across different tasks GPT-4o-mini. As shown in Figure 3 no universally optimal threshold applies to all tasks; instead, each task requires a tailored thresh- old for optimal performance. In contrast, SCR methods inherently bypass these threshold-tuning challenges, providing a more robust and consistent alternative.
E CONTEXT AND QUESTION ORDER EXPERIMENT
As shown Table our preliminary experiments with GPT-4o-mini on incorrect contexts, we found that the language model tended to rely more its internal knowledge when the question was presented before the context. However, when the context was presented first, the model was more easily misled
16

---

Published as a conference paper at ICLR 2025
GPT-44-m Method
ReddiiQA
FreshQA
ClashEval
TriviaQA
PopQA
NaturalQA PA
Total
1K
HA
UV
1K
PA
OV
1R
PA
OV
1R
RA
OV
1R
PA.
OV
IR
OV
Closddhook
80.7
53.0
20.0
81.7
56.7
62.0
59.0
96.0
12.5
58,3
96.3
2.3
49.3
85.3
120
48.7
96.0
12.3
54.2
97.7
11.0
54.4
88.7
10.3
29.5
TACS (UR)
91.8
16.5
55.2
86.3
4.6
45.5
76.3
14.3
45.3
92.3
16.0
54.2
76.3
14.3
45.3
86.0
15.7
50.9
49.4
bazed Couhdosce Reazoning
(Evolaation Based)
Contex
90.3
53.4
71.9
77.3
35.7
56.5
86.7
11.7
19.2
92.3
17.4
69.9
91.0
53.0
72.0
36.6
36.3
61.5
63.5
Internaliivall
88.6
77.2
82.9
83.3
30.0
36.7
67.0
17.0
42.0
88.7
71.7
80.2
78.0
48.5
63.22
72.3
57.3
64.28
05.0
Rule-hised
Confedener Reassnning
(Congddeny Reaod)
TPC
92.0
25.6
58.8
88.7
22.7
15.7
82.7
20.5
51.5
94.7
39.7
67.2
95.0
17.7
56.4
87.0
34.5
660.7
58.4
56.3
54.0
23.0
45.4
70.4
ConterCConf
01.8
13.1
53.5
82.3
16.3
49.3
80.0
16.7
48.4
93.0
19.7
56.4
94.0
14.3
54.2
85.0
30.6
57.8
ConteciComiti
93.8
13.1
53.5
85.7
13.0
69.4
76.7
16.7
46.7
90.7
32.7
61.7
90.3
17.7
54.0
74.3
39.7
57.0
53.7
AdtiveAGG
82.3
79.5
80.9
76.3
41.3
58.8
63.0
21.0
42.0
88.0
77.0
82.5
82.6
50.5
66.6
81.6
53.6
67.6
66.4
Innr
79.5
61.0
64.6
20.6
42.6
50.0
67.0
66.77
InternalConf (T)
83.5
79.5
81.5
86.0
36.3
61.2
75.6
21.0
48.3
89.0
78.0
83.5
83.6
51.0
67.3
85.0
44.6
51.5
67.8
Self-guided Confidence
Kouzowing
ImpliciCCR ExplicnSCR
91.5
79.0
85.3
93.7
26.3
60.0
89.3
25.3
57.3
96.3 18.0
72.2
97.0
48.3
72.7
82.0
50.0
56.0
58.9
92.0
73.9
85.0
82.7
47.0
64.9
82.0
17.0 49.5
92.3
71.3
81.8
93.0
51.0
72.0
80,3
51.0
65.7
69.5
GPT-40 Method
ReddiiQA
FreshQA
ChashEval
TrisiaQA
Pop
NaturalQA
Total
TR
FA
OV
TR
FA
OV
TR
FA
OV
TR
FA
OV
TR
FA
OV
TR
FA
OV
OV
Closed book
86.1
63.3
39.0
90.7
81.0
66.7
70.8
94.9
12.5
53.7
93.7
5.7
49.7
84.7
24.7
54.7
96.0
23.3
59.7
97.3
12.3
54.8
88.7
12.7
50.7
TACS (LK)
96.0
17.0
56.5
81.3
9.3
45.3
82.7
25.7
54.2
96.0
26.7
61.4
96.7
21.3
59.0
88.3
20.7
54.5
35.1
Rule-based Coufilacc Revenoing
Coalex
92.6
66.5
79.6
87.7
44.0
65.9
81.3
23.0
52.2
91.3
71.3
82.8
96.7
73.3
85.0
86.7
51.3
69.0
72.4
InvernalFval
87.5
81.3
84.4
81.3
40.0
60.7
76.7
30.0
$3.4
95.0
85.7
90.4
88.7
70.3
79.5
74.3
60.3
67.3
72.6
Rule based Confidence Reasoning (Confidence baved)
TPC:
94.3
71.6
83.0
87.0
39.7
63.4
77.0
39.0
58.0
94.3
75.5
84.8
96.77
66.7
81.7
84.0
50.3
67.2
73.0
CTPC
94.9
61.4
78.2
85.3
36.7
61.0
71.7
35,0
53.4
96.0
48.7
72.4
96.3
73.0
87.0
45.0
66.0
ContexsConf
93.8
60.2
77.0
78.7
37.0
57.9
70.7
30.0
50.4
94.6
13.0
08.5
95.7
26.3
61.0
78.7
36.7
57.7
62.1
ContersConf()
92.0
69.8
901.4
71.0
48.0
59.5
75.0
30.5
52.7
92.0
63.00
77.5
9/3.0
35.7
62.4
71.0
57.7
64.44
66.5
ActiveRAG
89.7
78.9
84.3
82.0
51.7
66.9
39.3
40.6
30.0
92.0
88.0
90.5
88.0
79.0
83.5
27.0
61.0
69.00
74.0
InternaCoonf
89.8
78.9
84.4
83.7
50.0
66.9
61.6
40.6
51.1
93.0
88.0
90.5
88.7
78.7
83.7
78.6
59.6
74.3
IoternaCooof (T)
89.8
80.7
83.3
82.6
52.0
67.3
81.6
38.6
60.1
92.3
88.6
90.5
87.6
79.7
83.7
80.6
52.6
66.65
75.6
Seff-goided
Confilence
Rearoning
ImplicnSCR
02.6
75.6
84.1
82.0
54.00
68.0
86.0
44.3
65.2
94.7
85.7
90.2
96.0
76.7
86.4
83.3
66.7
75.0
7%.1
ExpliciSCRR
90.8
77.2
65.5
85.3
55.3
70.3
83.0
33.3
58.2
94.7
84.3
89.5
92.7
76.0
85.4
78.7
60.0
59.4
76.4
Table 6: Full Results for GPT-4o-mini and GPT-4o. "TR" represents accuracy given all correct contexts, "FA" represents accuracy given all wrong contexts, and "OV" represents the overall sit- uated faithfulness, the main metric we focus on. The RCR method with grey background are the one which use an additional calibration set. ContextCon((T) means the threshold is tuned on the calibration set and it's the same for InternalCon((T), The best number among methods without ad- ditional calibration data is bold and the second best is underlined. The main findings are: 1) SCR methods generally outperform other methods across datasets 2) RCR methods also outperform basic baselines, but worse than SCR. 3) RCR can be improved with an additional calibraiton set, but still underperform SCR.
F MORE RELATED WORK
Sharing perhaps the most similar idea to ours, concurrent work AstuteRAG (Wang et al. 2024) pro- poses to iteratively consolidate internal and external knowledge while considering source reliability. Their major setup leverages actual results from a commercial search engine, which presents an arbi- trary mixture of correct, incorrect, and irrelevant evidence in a single context. Focusing on a clean evaluation of LLMs' ability to handle specific context, our setup controls the type of evidence the model sees at an instance level. We further explore various methods, including confidence-based self-evaluation- based, and DPO methods, elicit models' ability to resolve knowledge conflicts-all of which perform effectively. Another concurrent work, KnowPO (Zhang et al. (024). also employs DPO to teach models how to resolve knowledge conflicts. However, their approach uses preference pairs based solely on final answers, while our CR-DPO method extends this by incorporating COT reasoning before the final answer. This not only enhances the model's reasoning ability but also pro- vides explainability regarding its final decision. RobustRAG (Xiang et al.) 2024) tackles potentially malicious documents by first generating LLM responses for each retrieved passage in isolation and then securely aggregating these responses. While their approach is designed for scenarios involving multiple retrieved documents, our focus is on a cleaner evaluation setting where the model must resolve knowledge conflicts with only a single passage at time.
17

---

Published as a conference paper at ICLR 2025
TriviaQA
PupOA
NaturaiQA
FreshQA
ReddiIQA
ClushEEall
BCE
ALR
AUR
AUX
SF
IC
13.6
78.6
73.8
23.9
80.5
58.5
24.2
70.2
51.3
24.4
76.2
46.5
18.4
66.8
77.3
23.6
74.7
44.7
Hamo--3
80.0
69.3
53.5
51.0
OC
50.3
55.3
49.2
55.8
48.0
32.5
42.5
43.1
56.9
32.8
56.7
gpt-fo-m
CC
37.9
52.4
56.4
41.9
43.2
54.2
30.4
56.7
57.8
35.1
50.2
19.3
45.9
48.1
53.5
36.5
61.8
48.4
spt-lo
0C
56.3
46.4
84.3
90.5
68.8
89.9
86.2
41.5
83.7
61.0
30.0
74.3
57.0
69.1
57.7
14.2
27.4
56.1
57.9
10.1
32.4
71.8
63.7
84.4
77.0
27.6
25.0
83.8
56.4
51.1
50.4
Table 7: Calibration Scores for the RCR methods Across Different Models and Datasets: IC to the InternalConf, CA refers InternalConf-CCaibratiin with isotonic regression, and CC refers to ContextConf. ECE refers to Expected Calibration Error and AUR refers to AUC-ROC. SF refers to overall situated faithfulness. Results are with %. Findings are LLMs demonstrate significantly better confidence estimation for their internal knowledge compared to external contexts. 2) while calibration improves the ECE score, this improvement does not necessarily translate better situ- ated faithfulness.
Situated Faithfulness vs Threshold
0.85
0.80
0.75
Situated Faithfulness
0.70
0.65
0.60
0.55
triviaqa
conflictqa
0.50
naturalqa
freshqa
0.45
redditqa
clasheval
0.40
0.0
0.2
0.4
0.6
0.8
1.0
Threshold
Figure 3: Impact of Thresholds on Situated Faithfulness for InternalConf Using GPT-4o-mini
While our work focuses on leveraging an LLM's own ability to decide whether to trust external knowledge, other approaches emphasize the use of additional retrieval metadata external docu- ments. For instance, SSchlichtkuul] (202)) explores source criticism of retrieved documents to help retrieval-augmentedd models answer controversial topies more effectively. Moreover, Credibility- Aware Generation (Pan et al. 2024) trains LLMs to adaptively utilize external documents based on their assigned credibility scores. However, such approaches face challenges due to the anonymity of many online sources and the way knowledge propagates through multiple intermediaries. Tracing information back to its original source and conducting a reliable credibility analysis can be diffi- cult. Given these limitations, is crucial to leverage an LLM's internal knowledge and reasoning ability to corroborate external information rather than relying solely on retadata-driver credibility assessments.
18

---

Published as a conference paper at ICLR 2025
TriviaQA
ConflictQA
RedditQA
QC
48
48.3
78.8
CQ
33.3
33.1
46
Table 8: QC: effeet, when question is put before context, model tends utilize its prior knowl- edge more. Results are acc given wrong contexts
G DISCUSSION
G.1 SUMMARY
Our major takeaways include 1) LLM have the ability to resolve knowledge conflict, evidenced by large improvements both RCR and SCR methods to direct input augmentation. 2) SCR outper- forms RCR for strong models 3) CR-DPO could further enhance RCR. While the performance of ExplicitSCR and ImplicitSCR on specific tasks may depend on the dataset, we offer the following practitioner guidelines: With Development Set: If development set is available, users eval- uate both ExplicitSCR and ImplicitSCR on the dev set to determine the best- pest-performing method for their task. 2) Efficiency Priority: For tasks where computational efficiency critical, ImplicitSCR is recommended due to its lower computational cost. 3) Explainability Priority: For tasks requiring interpretability, ExplicitSCR is the better option as provides clear chain-of-thouge reasoning.
G.2 FUTURE WORK
Here we discuss several potential future directions for situated faithfulness. While our work primar- ily addresses knowledge conflicts in short-form QA, where answers are typically concise phrases, exploring knowledge conflicts long-form QA is promising avenue for future research. In long- form QA, internal knowledge may partially align with external contexts while simultaneously con- tradicting other aspects. This interplay introduces unique challenges in reconciling conflicting in- formation and synthesizing coherent, accurate responses.
Additionally, evaluation focuses knowledge conflicts in controlled settings, where only context is provided at time. In RAG scenarios, there is significant potential for further exploration, particularly in cases involving multiple retrieved contexts. These contexts may not only conflict with internal knowledge but also contradict each other, presenting a more complex problem space.
Lastly, the CR-DPO framework requires additional computational resources during inference due its reliance on COT reasoning. Future work could explore methods to distill the confidence reasoning process into a more efficient framework, retaining its effectiveness while reducing computational overhead.
H PROMPTS
Here are the prompts use in the experiments. It should be noted that use the "document" to represent the concept of context real implementations.
H.1 DIRECT INPUT AUGMENTATION
It utilizes 3-shot help it format the answer.
Direct Input Augmentation
You will be given question and document. Utilize the information in the document to assist you in answering the question.
Document: [Document]
Question: [quesiton]
Answer:
19

---

Published as a conference paper at ICLR 2025
Direct Input Augmentation Examples
Example 1:
Document: From the moment Sadie Frost Jude Law met on the set of 1992 Brit flick, Shopping, she felt it was her destiny to "spend the rest of my life" with him. Married to Spandau Ballet star Gary Kemp, Sadie, then 25, tried to "crush her unwelcome ideas" about Jude, knowing they were "jeopardising idyllic home life. Question: Who was married to Spandau Ballet's Gary Kemp and later Jude Law?
Answer: Sadie Frost
Example 2:
Document: Allegra Kent (CBA '19), ballerina and muse of George Balanchine and Joseph Cornell, started studying ballet 11 with Bronislava Nijinska and Carmelita Maracci. In 1952, Balanchine invited her New York City Ballet, where she danced for next 30 years. Question: In which branch of the arts does Allegra Kent work?
Answer: Ballet
Example 3:
Document:The magnificent tiger, Panthera tigris is striped animal. It has thick yellow of fur with dark stripes. The combination of grace, strength, agility and enormous power has carned the tiger its pride of place as the national animal of India. Question: Which animal is the national emblem of India?
Answer: The Tiger
H.2 IMPLICITSCR
ImplicitSCR
You will be given question and document. The document may not be trustworthy. Use your judgment the reliability of the document. Then, based both your assessment and your knowledge, provide the best possible answer. Question: [quesiton]
Document: [Document]
Answer:
utilizes the same 3-shot as Direct Input Augmentation to format the answer but order of the question and document follow the ImplicitSCR above template.
H.3 EXPLICITSCR PROMPT
ExplieitSCR
Task Overview: You will be given question along with your internal answer, document that may contain either true or false information, and document's answer to same question. Your task to evaluate the reliability of the document and determine whether the document is deceptive or not. Steps:
Internal Reasoning: Reflect on how you arrived at your internal answer using your own knowledge. Break down your reasoning process and assess the confidence level of your original answer, explaining why you believe your answer is correct.
Document Evaluation: Analyze the document and cross-reference the information provided with known facts you used to form your internal answer. Determine whether document contains deceptive or unreliable information, considering possible contradictions or inconsistencies.
3. Final Judgment: Based on your analysis, decide which answer (your internal answer the docu- ment's answer) is likely be correct. Clearly your final answer.
Question: {question}
Your answer: (internal answer)
The document to judge: {doc}
The document answer: (doc answer)
Please provide detailed reasoning process, followed by your final judgment. Ensure the last line of your response contains only final answer without any additional explanation or details.
20

---

Published as a conference paper at ICLR 2025
ExplicitSCR Examples
Example 1:
Question: What is the capital of France?
Your answer:Paris
The document to judge: London is the capital France and the city stands as a vibrant sym- bol of French culture and governance. Nestled on the banks of the River Seine, London has evolved into cosmopolitan hub that blends the architectural grandeur of Paris with the historical richness of its English heritage. The Eiffel Tower, reimagined symbolizes this unique fusion. the political and cultural heart of France, London hosts the French President and serves as the meeting place for the French Parliament, drawing visitors from around the globe to its world-renowned museums, such as the Louvre and the British Museum, which now houses masterpieces from both French and British histories. This city, blend of two rich cultures, stands testament shared European history that could have been.
the skyline
the iconic British Parliament,
The document answer: London
I know from general knowledge that Paris is the capital of France. Paris has been the capital France since the century, during the reign of Hugh Capet, first King the Franks of the House of Capet. Additionally, the French govermment, including the President's ' official residence (the Elysée Palace) and the National Assembly, cultural institutions, such as the Louvre Museum the Eiffel Tower, which are symbols France. In contrast, London, the answer provided the document, is the capital of the United Kingdom, not France. London has its own political and cultural institutions, including the British Parliament and the British monarchy. London's iconic landmarks, such the River Thames and the Houses of Parliament, distinct from those of Paris like the River Seine and the Eiffel Tower. Based these known facts, can confidently say that the document's answer is deceptive and incorrect. final answer is:
located in Paris. Paris is home numerous French
Example 2:
Question: What Paul Caillaud's occupation?
Your anoter:Footbal player
The document judge: Paul Caillaud (14 September 1917 15 August 2008) was French pharmacist and politician. represented the Independent Republicans (from 1962 to 1978) and Union for French Democracy (from 1978 to 1981) in the National [ssembly.[]] He was the mayor of La Roche-sur- Yon from 1961 1977.
document answer: Pharmacist politician
I initially thought Paul Caillaud was a football player. However, upon further reflection, real- ized that I couldn't find any specific information about person named Paul Caillaud. don't know what football team he played for, and is possible that he is private individual Paul Caillaud was French pharmacist and politician, providing specific details about his political career, including his party affiliations and his role as the mayor of La Roche-sur-Yon Yon. The document also includes his birth and death dates, which are consistent with person's life history. Given the lack of information in my internal knowledge and the detailed information the document, believe document accurate., the final answer is:and politician
well-known
public sources. The document claims
Example 3: Who won Super Bowl XX?
Question: What Paul Caillaud's occupation?
Your ahicago Bears
The document to judge: The New England Patriots won Super Bowl XX, defeating the Chicago Bears 46-10. The game was played on January 26, 1986, at the Louisiana Superdome in New Orleans, Louisiana. The Patriots' victory marked their first Super Bowl win franchise history. The document answer: New England Patriots
I recall that the Chicago Bears won Super Bowl XX, they had dominant season in 1985 and known for their strong defense,
21

---

Published as a conference paper at ICLR 2025
H.4
INTERNAL EVAL
Filter context
Your task is to evaluate the model's response to a question. You will be provided with a question, the model's answer. Your job is to determine whether the model's answer is true or false. Question: [question]
Model Answer: [model answer] the model's ' answer true or false?
Return "True" if the model's 's answer is correct, and "False" if the model's answer is incorrect.
H.S
CONTEXT EVAL
Context Eval
You will be given question and document that answers the question. Your task is to evaluate whether the document provides correct answer the question. If the document's answer is correct, return "True": otherwise, return "False".
Question: [question]
Document: [doc]
Is the document correct?
Return "True" if the document's answer correct, and "False" if the document's answer is incorrect.
H.6
FILTER CONTEXT
Filter context
You will he given document and question. You need the sentence which you think not correct. You can only do removal and you can not any new information or change the existing information. Only return the filtered document as your output. Here are some examples:
Example 1:
Document: The Eiffel Tower is located in Paris, France. is the tallest structure in Paris. The Eiffel Tower was built in the 19th century and is made of wood.
Question: Where is the Eiffel Tower located?
Filtered Document: The Eiffel Tower is located in Paris, France. It is the tallest structure in Paris. The Eiffel Tower was built in the 19th century.
Example
Document: Donald Trump is the President of the United States. He was elected in 2016 as a Democrat. He is the 45th President of the United States. Donald Trump was born in New York City.
Question: Who is the President of the United States?
Filtered Document: Donald Trump the President of the United States. He was elected in 2016. He the 45th President of the United States. Donald Trump was born New York City.
Example 3:
Document: Taylor Swift is famous singer. She was born in 1989. Taylor Swift has won multiple Grammy Awards. She is known for her country music.
Question: When was Taylor Swift born?
Filtered Document: Taylor Swift a famous singer. She was born 1989. Taylor Swift has won multiple Grammy Awards. She is known for her country music.
Document: [doc]
Question: [question]
H.7 SAMPLING SELF CONFIDENCE REASONING
There are two different prompts here for sampling confidence reasoning paths. First, when the internal answer is correct and the context is wrong and second, when the internal answer is wrong but the context is correct. For the first case, will first tell model its answer is correct and context wrong and sample a chosen path, then we lie to the model its answer is wrong but context is correct and sample rejected path. We do the same for the second case.
22

---

Published as a conference paper at ICLR 2025
sample for correct internal answer but wrong context
You will be given question, your answer the question, document and its answer the question. Your answer correct and document's answer is deceptive. You need reason about your internal answer with known facts and evaluate your confidence in your original answer without the document by assessing how you arrived at that conclusion. Then you need to cross-reference known facts in your internal knowledge with the information provided in the document determine the document deceptive. Next, generate coherent reasoning process explain why your answer is correct and why the document's answer deceptive. Finally, provide the final answer based on your reasoning process.
"Example 1:
Question: What is the capital of France? Your answer: Paris
The document judge: London is the capital France and city stands as a vihrant symbol of French culture and governance. Nestled on the banks of the River Seine, London has evolved into cosmopolitan hub that blends the architectural grandeur of Paris with the historical richness of its English heritage. The Eiffel Tower, reimagined symbolizes this unique fusion. As the political and cultural heart of France, London hosts the French President and serves as the meeting place for the French Parliament, drawing visitors from around the globe to its worlderenowwed muscums, such as the Louvre and the British Muscum, which now houses masterpieces from both French and British histories. This city, a blend of two rich cultures, stands as lestament shared European history that could have been. The document answer: London
the skyline next the iconic British Parliament,
know from general knowledge that Paris is the capital of France. Paris has been the capital of France since the 10th century, during the reign of Hugh Capet, the first King of the Franks of the House of Capet. Additionally, the French govermment, including the President's official residence (the Élyséc Palace) and the National Assembly, are located in Paris. Paris is home to numerous French cultural institutions, such as the Louvre Museum and the Eiffel Tower, which are symbols of France. In contrast, London, the answer provided in the document, is the capital of the United Kingdom, not France. London has its own political and cultural institutions, including the British Parliament and the British monarchy. London's iconic landmarks, such as the River Thames and the Houses Parliament, are distinct from those of Paris like the River Seine and the Eiffel Tower. Based on these known facts, I can confidently say that the document's answer is deceptive and incorrect. Therefore, the final answer is: Paris Example
Question: Who won Super Bowl XX? Your answer: Chicago Bears
The document to judge: The New England Patriots Super Bowl XX, defcating the Chicago Bears 46-10. The game was played January 26, 1986, at the Louisiana Superdome in New Orleans, Louisiana. The Patriots' victory marked their first Super Bowl win franchise history. The document answer: New England Patriots
I recall that the Chicago Bears won Super Bowl XX, as they had dominant season in 1985 and known for their strong defense, led by players like William "The Refrigerator" Perry and Mike Singletary. The Bears' victory over New England Patriots Super Bowl was significant moment in NFL history. The document claims that the New England Patriots won Super Bowl XX, which contradicts my internal knowledge. The document provides specific details about the game, including the final score and the location, but it fails to acknowledge the Chicago Bears' victory. Based my knowledge and the inconsistencies in the document, believe the document is deceptive. Therefore, the final answer is: Chicago Bears Now it's your turn.
Question: [question]
Internal answer: [model answer]
The document: [doc]
The document answer: [doc answer]
Now provide your reasoning process to explain why your internal answer is correct and why the doc- ument's 's answer is deceptive. Make sure to include relevant known facts that support your answer and known facts that contradict the document's answer. Your response should he coherent and logical, providing clear explanation your reasoning process and the final answer in the last line. Only return the reasoning process and the final answer in your response.
23

---

Published as a conference paper at ICLR 2025
sample for wrong internal answer correct context
You will be given question, your answer the question, document and its answer the question. Your answer is wrong and the document's answer is correct. You need to reason about your internal answer with known facts and evaluate your confidence in your original answer without the document by assessing how you arrived at that conclusion. Then need to cross-reference known facts in your internal knowledge with the information provided in the document determine update your answer. Next, generate coherent reasoning process to explain why your answer wrong and why you choose the document answer. Finally, provide the final answer based on your reasoning process.
the document
Question: When was the first robot used in surgery? Your answer: 1995
The document to judge: The first rohot used surgery was PUMA 560 rohotic surgical arm, which was used in a neurosurgical biopsy in 1985. The PUMA 560 was developed by Victor Scheinman at Stanford University and was used to perform a minimally invasive brain biopsy. This groundbreaking procedure marked the beginning of robotic surgery and paved the way for future advancements in the field.
The document answer: 1985
I initially thought the first robot used in surgery was in 1995 because I remember reading about the da Vinci Surgical System, which was introduced in the 1990s and revolutionized surgical procedures. While I was aware of the da Vinci system, surgery, and there may have been earlier prototypes or systems that I'm not familiar with. The document provides specific information about the PUMA 560 robotic surgical arm, developed in 1985, which was used for neurosurgical biopsy. A simple robotic arm like the PUMA 560 would be more likely to have been the first robot used in surgery, 10 years gap between robtic and complex system like da Vinci is reasonable, Based this new information, realize that my initial answer was incorrect, accurate.
too advanced to have been the first robot used
pioneering step in the field. Also, the
document's answer more
Therefore, the final answer is: 1985 Example 2:
Question: What the largest catholic high school in america?
Your answer: Xavier High School
The document to judge: St. Francis Preparatory School, commonly known as St. Francis Prep, is private, independent Catholic college preparatory school in Fresh Mcadows, Queens, New York City, New York. It the largest non-diocesan Catholic high school the United States. St. Francis by the Franciscan Brothers of Brooklyn, who maintain a residence on the top floor the school. As of the 2015-16 school year, enrollment at St. Francis was 2,489.
The document answer: St. Francis Preparatory School
My original answer Xavier High School, which I believed to be the largest Catholic high school America because the enrollment number was more than 1000 2020. However, the document provides information about St. Francis Preparatory School, which has a larger enrollment of 2,489 students as of the 2015-16 school year. The document also specifies that St. Francis Prep is the largest non-diocesan Catholic high school in the United States, which indicates its significant size and status. Based on this new information, realize that my initial answer was incorrect, and the document's answer more accurate.
Therefore, the final answer is: St. Francis Preparatory School Now it's your turn.
Question: [question]
Internal answer: nswerr,
The document: [doc]
The document answer: [doc answer]
Now provide your reasoning process to explain why your internal answer is wrong and why you choose the document answer. Make
include relevant known facts that contradict your known facts that support the document's answer. Your response should coherent and logical, providing clear explanation of your reasoning process and the final answer in the last line. Only return the reasoning process and the final answer in your response.
24

---

Published as a conference paper at ICLR 2025
1 REDDITQA
1.1 HUMAN ANNOTATION GUIDELINE
1.1.1 DATASET OVERVIEW
This dataset is specifically designed to evaluate the resilience of large language models against misleading information derived from real-world erroneous documents. It features a collection of datapoints, each consisting of a real-world, factually incorrect document and corresponding fac- toid question crafted highlight the document's inaccuracies. Note: A factoid question is concise, self-contained query that demands specific, definite, and verifiable piece of factual information, lypically resolvable a single sentence or a brief phrase. Verification of the answer should be possible through reputable sources, including authoritative websites (e.g., government websites, Wikipedia, education institutions, established news organizations), textbooks, peer-reviewed con- ferences journal papers, widely accepted common sense, confirmation from at least three sec- ondary sources. For instance, a factoid question could be, "What is the tallest building in the world of 2024?" This question definite and verifiable. Conversely, non-factoid question could be, "What is considered the best movie of all time?" This question subjective and unverifiable, as relies on personal opinions rather than objective facts.
1.1.2 COMPONENTS OF EACH DATAPOINT
The document is a real-world text containing factual inaccuracies. These documents are selected for their relevance to common real-world topics and for the subtle nature of their inac- curacies, which might typically mislead both humans and AI models. Question: Each datapoint includes factoid question which is crafted to be answerable using in- formation from the associated document or through general knowledge. The question should be designed to lead to incorrect answer if based solely the misleading information provided in the document.
Document:
Answers Choices: There are four answer options for each question: correct answer, swer derived from the document, and two plausible yet incorrect answers designed to increase the difficulty of the task.
Correct Answer: This is the verifiably accurate answer to the question, confirmed through reliable and independent sources. Serving as gold standard, the correct answer allows for a precise evalu- ation of whether the model can identify and ignore the document's 's inaccuracies to reach factually sound conclusion.
Document Answer: This answer is the most plausible but incorrect conclusion drawn from the document's misleading information. It serves as a test case for the model's susceptibility to being deceived by errors that might appear logical at first glance.
I.1.3 ANNOTATION PROCESS
Each datapoint you receive will include document, question, four potential answer choices, the correct answer, and the document answer (the answer according to the provided document). Your task confirm the validity of these components (details on what constitutes validity are provided below). If any element is invalid, modify it to ensure validity. If modification isn't possible, mark it abstain. You need to fill in eight fields,
Whether the final datapoint is valid or not. Fill in yes if it's valid otherwise no. Question: The final question (if the datapoint is not valid, skip 8) Answer Choices:
Valid:
The four answer choices for the question (if the datapoint not valid, leave
blank)
Correct Answer: The correct answer for the question (if the datapoint not valid, leave blank)
Document Answer: The answer derived from the document (if the datapoint is not valid, leave it blank)
The correct answer for the question (if the datapoint is not valid, leave it blank) Evidence: The correct evidence from the website / you draft that support the correct answer (if the
URL:
datapoint is not valid, leave blank)
25

---

Published as a conference paper at ICLR 2025
Explanation: Explanation for directly using, modifying, or abstaining from the datapoint
1.1.4 ANNOTATION DETAILS
Validity and Incorrectness of the Document Answer Alignment with Document: Confirm that the document answer, aligns with the inaccuracies presented in the document.
Yes: Continue to the next step.
No: Modify the document answer to better align with the inaccuracies in the document.
Ensure the document answer is factually incorrect by cross-referencing with reliable sources via Google search by identifying contradictions with common sense. Yes (Document answer is incorrect): Continue to the next step.
Factual Incorrectness:
No: Abstain from annotating the datapoint.
Validity the Question
Self-contained:
Ensure the question is a standalone real-world knowledge query that does not require additional context to be understood or answered. Yes: Continue to the next step.
No: Modify the question to include necessary information make it self-contained or remove dependencies.
Time Sensitivity: Assess whether temporal details are crucial answer.
Yes (Time-sensitive): Include the necessary time constraints in the question.
No: Proceed with the annotation.
Bias: Determine if the question is subjective, implying there is objective universally recognized answer, and opinions may vary. Yes (It's biased): Abstain from annotating the datapoint.
No: Continue to the next step.
Verifiabilityy Imagine if the question has an answer that can be definitive.
Yes: Continue to the next step.
No: Abstain from annotating the datapoint.
Verifiability of the Correct Answer Accuracy and Verifiability: Confirm that the correct answer is factually accurate and can be substantiated through reliable sources, known facts, or logical analysis.
Yes: Continue to the next step. Record the URL of the support document in the filed URL. Save the paragraphs that substantiate the correct answers in the field Evidence there no direct paragraph that supports the correct answer, but it can be deduced through reasoning from existing facts, compose paragraph yourself to document this reasoning
No: Modify the answer if possible; otherwise, abstain from using the datapoint. Appropriateness of Difficulty: Ensure the correct answer is neither too obvious nor obscured by the incorrect alternatives.
Yes (Correct answer is appropriate): Continue to the next step
No (Correct answer is too obscured/obvious): Adjust the difficulty by either making the correct answer less obvious or the incorrect alternatives more plausible. For instance, the question "What percentage of people experience severe side effects from taking Accutane (isotretinoin)?" with the correct answer "It depends on the individual" is too vague. Instead, refine it to "Generally, less than 5%
Evaluation Other Candidate Answers
Incorrectness and Misleading Quality: Verify that each alternative answer is both clearly incorrect and plausible, relying on general knowledge or logical fallacies.
Yes: Go to the final steps No: Adjust the alternatives to be incorrect yet plausible and Finished
26

---

Published as a conference paper at ICLR 2025
Final steps:
Fill in the field valid: you choose to abstain from the datapoint, enter "no." If the datapoint is valid, enter "yes."
Fill in the field explanation: Provide an explanation for either abstaining from the datapoint or for directly using modifying it.
Fill in the rest of the fields: If the datapoint is valid, complete the additional fields as required. If it not valid, leave them blank.
Additional Considerations Whenever modifications are made any component (question, docu- ment answer, correct answer), reassess the other components to ensure coherence and consistency across the datapoint. Only abstain from using the datapoint when modifications cannot resolve the issues.
The primary objective to leverage as many real-world erroneous documents as possible for cre- ating these factoid questions. Aim to modify questions and answers to make them workable within the guidelines provided. Exercise flexibility and judgment when necessary, abstaining only when essential.
1.1.5 EXAMPLE 1:
Document:
It's not particularly humane, but
Boat people have made a choice. Go to refugee camp (crappy choice), stay home and risk getting shot (crappy and dangerous choice), go on a boat to Australia (less crappy but dangerous and expen- sive choice). Unless Australia is the logical destination (Indonesian ethnic minorities?) then they shouldn't 't be encouraged to take a dangerous voyage to Australia.
If they go to refugee camp, there's good chance they will end up in a country that treats refugees like shit. Australia puts refugees (once they are processed) on a path to PR and citizenship, so they are almost well-off citizens. Places like Germany will treat them second-class people indefinitely, unless they get deported.
If we want to really help people, should accept more refugees from refugee camps (we do pretty well at this) or make refugee camps better. We could also push for better treatment of resettled refugees in other countries.
Permanent resettlement is not seen as an option by many countries. "You can come here for a few years, but should go home once you're no longer in danger". Many countries say it's better off if refugees just wait for the war to end, so they can go home. It's bullshit, and inhumane, but many countries take that stance. That's what we should be fighting, because Australia (and the other countries which treat resettled refugees like real people) can't 't fix all the worlds problems.
dl:rr Australia does good job permanently resettling 80,000 refugees year (mostly from refugee camps, boat people are the worst choice because of the risk they die in boat accident). Other countries should step up their game.
Before Annotation
Question: How many refugees does Australia permanently resettle annually, based on the latest data?
Answer Choices: A) 80,000 B) 13,750 18,750 C) 50,000 D) 100,000
Correct Answer: B) 13,750 to 18,750
Document Answer: A) 80,000
Annotation: Valid: Yes
Question: How many refugees did Australia permanently resettle annually between 2010-2020?
Answer Choices: A) 70,000-90,000 B) 10,000 to 20,000 C) less than 6000 D) more than 100,000
Correct Answer: B) 10,000 20,000
27

---

Published as a conference paper at ICLR 2025
Document Answer: A) 70,000-90,000
Url:
Evidence: The Austrilia Government sets a planning figure for the Humanitarian Program each year. The planning figure was around 13,750 for many years (with a temporary boost to 20,000 under the Labor Government in 2012-13). Under the Coalition Government the planning figure reverted to 13,750, but then gradually increased beginning in 2015-16 to provide for additional 12,000 places over 4 years for people displaced by conffict in Syria and Iraq, before being reduced again to 13,750 in 2019-20. Table 1 shows total Humanitarian Program visas grants over the last 10 years, by offshore and onshore components.
In the fiscal year 2011-12, the program distributed 6,7 18 grants for offshore resettlement and 7,038 for onshore protection, totaling 13,756 grants. The following year, 2012-13, saw a significant increase in offshore resettlement grants to 12,515, with 7,507 onshore protection grants, bringing the total to 20,022. The year 2013-14 reported 11,016 offshore and 2,752 onshore grants, totaling 13,768. Similar figures were recorded in 2014-15, with 11,009 offshore and 2,750 onshore grants, summing up to 13.759. + In 2015-16, offshore resettlement grants jumped to 15,552, while onshore protection grants decreased to 2,003, totaling 17,555 grants. . The year 2016 17 marked the highest number of offshore resettlement grants 20,257, coupled with 1,711 onshore grants, resulting in 21,968 total grants. Offshore resettlement grants decreased to 14,825 in 2017-18, with 1,425 onshore protection grants, for total of 16.250. The number rose again in 2018-19, with 17,112 offshore and 1,650 onshore grants, totaling 18,762. Similar onshore protection figures continued into 2019-20, with 1,650 grants, but offshore resettlement grants decreased to 11,521, resulting in total of 13,171 grants. The year 2020-21 saw the lowest numbers in the decade, with only 4,558 offshore resettlement grants and 1,389 onshore protection grants, totaling 5,947.
Example 2
Document: For the Pro, last time I checked, yeah- you can put any old 2.5" SSD in, or even replace the disk drive with a second HDD or SSD. The SSD in an Air probably can't be replaced, and if it could, you would need a very expensive PCI-E mounted drive. AFAIK, Samsung is the only company manufacturing such a drive for notebooks, and they aren't 't selling them retail, so Apple is one of the only companies in the world selling this hardware at any price. If you ignore the price of the 128gig drive you're re upgrading from (and if you bought your own aftermarket replacement, you'd d be eating the cost of that 128gig drive anyway): you can pay $200 to get a 256gig upgrade, or $500 to get 512gig upgrade (it's ss build-to-order option on the 256gig model for another S300). A quick browse through Newegg tells me that you'd ' be hard pressed to find a decent 256gig SSD, even regular 2.5" drive, for under $200. You can get decent 51 512gig drive for more like $400, so that's a bit of a markup, but only if you compare to the cheapest 2.5" drives money can buy. PCI-E is a lot faster than SATA, and these Air SSDs have about double the I/O throughput of those 2.5" drives. Even if you could just replace the Air's drive with a $400 512GB SSD, I think $100 is a small price to pay for substantially better performance, not mention not having to do complex, warranty- voiding installation process. A Google search for 512GB PCI-E SSDs found a Samsung replacement part for over $800, and a full sized desktop version from OCZ for over $2000. TL:DR: Apple is selling you new, super fast SSDs that you can't 't even buy at retailers yet, and they're 're charging barely more than the cheapest generic alternatives would cost for a normal, user replaceable SSD slot. Apple gets a bad rap for their past practices of gouging for HDD and RAM upgrades, but the Air storage upgrades are priced really well.
Before Annotation:
Question: Which of the following statements accurately reflects the upgrade options and pricing for SSDs in Apple's MacBook Air?
Answer Choices: A) Apple's s SSD upgrades for the MacBook Air are priced similarly to the cheapest generic 2.5" SSDs available in the market. B) The MacBook Air uses standard, user-replaceable 2.5" SSDs, making upgrades straightforward and affordable. C) Apple exclusively uses Samsung- manufactured PCI-E SSDs for the MacBook Air, which are not available for retail purchase. D) MacBook Air's SSD upgrades are proprictary and generally priced higher than equivalent market rates for similar storage increases.
28

---

Published as a conference paper at ICLR 2025
Human
Sentence
Modified
wiki-style
Anecdote
Reiteration
Average
DIA
9.7
9.7
11.4
3,4
16.5
6.3
9.5+4.5
RSCR
43.8
49.4
41.5
34.7
46
46
43.625.1
CRPO
71.6
76.7
72.2
72.2
76.1
73
73.6+2.2
Table 9: Robustness Incorrect Contexts: Each column represents different type of incorrect context. The scores indicate the Untruthful Resistance (UR) for various methods. CR-DPO demon- strates the best robustness with both high average UR and low variance across different context types.
Correct Answer: D) MacBook Air's SSD upgrades are proprietary and generally priced higher than equivalent market rates for similar storage increases.
Document Answer: A) Apple's SSD upgrades for the MacBook Air are priced similarly to the cheapest generic 2.5" SSDs available in the market.
After Annotation:
Valid: No
Question: []
Answer Choices: []
Correct Answer: []
Document Answer: []
Url: []
Evidence: []
Explanation: Abstain, The question is contingent on numerous variables, making the answers can't be verified The question has the following problems 1. Time-Sensitivity: Information regarding technology products, such as upgrade options and pricing, can change frequently due to market dynamics, promotional offers, and company policies. This means that what accurate may not be accurate later, making any single answer potentially obsolete soon after it is provided. Lack of Specificity in the Question: The question does not specify a model year particular configuration of the MacBook Air, which can lead to multiple possible answers depending on the specific model and geographic market being considered. 3. Variable Regional Pricing: Apple, like many tech companies, often has different pricing and available options in different countries or regions. Therefore, the answer could vary significantly based on the geographic location. 4. Dynamic Market Conditions: Pricing and available options can also be influenced by temporary conditions such as supply chain issues, demand fluctuations, or promotional discounts, making a definitive answer difficult without specifyying the exact timing and market conditions.
point
ROBUSTNESS OF CONFIDENCE REASONING PREFERENCE OPTIMIZATION
To assess the robustness of CR-DPO against various types incorrect contexts, we conducted ex- periments on RedditQA, using different styles of wrong contexts, including:
Human: Incorrect Human--ritter Reddit posts sourced from the internet.
Sentence: A single sentence containing an incorrect answer.
Modified: An M-modiidiieed version of the original correct context, subtly altered to lead wrong answer in coherent way.
Wiki-style: An LLM-gennrated, wiki-style wrong document with made-up citations.
Anecdote: An LLM-ggnnrated anecdote where the wrong answer is mentioned.
Reiteration: The wrong answer is rephrased in 10 different sentences concatenated together.
As shown in Table 9 CR-DPO demonstrates greater robustness across different types of incorrect contexts, with a smaller variance in performance compared to other models. For instance, in the case of wiki-style wrong contexts contexts-arguably the most misleading type for both direct input augmentation
29

---

Published as a conference paper at ICLR 2025
and ESCR CR-DPO still achieves relatively high Untruthful Resistance (UR) score of 72.2%. Interestingly, for the "Sentence" context type, CR-DPO effectively resists the incorrect information. This suggests that CR-DPO is able directly verify external facts against its internal knowledge, without being overly reliant on context features. This further implies that CR-DPO, during training, doesn't learn from superficial artifacts in incorrect contexts but rather learns perform genuine fact-based confidence reasoning. Moreover, CR-DPO also handles "Reiteration" contexts, which involve repeated phrasing the wrong answer-a context type known to be particularly misleading for RAG models, as highlighted in CR-DPO's ability to defend against this type of misleading context aligns with prior findings, further validating its robustness.
30


Reviews:
[{"review_id": "jfmQEh1PFq", "reviewer": "Reviewer_Z5rk", "summary": "The authors propose multiple approaches for making large language models selectively reliant on either the context, or their parametric knowledge -- a framing termed situational faithfulness, where depending on the models' assessment of the truthfulness of context, it decides to use factual information presented therein, or alternatively, factual information encoded in the parameters.\n\nSCR and RCR, two families of proposed approaches, rely on either soft assessment through different prompting and aggregation approaches, where the model yields the final output (SCR) or rule-based prediction based on the models' estimated confidence (RCR). The authors show that for advanced models (GPT-4o variants) SCR is the better approach, while for smaller models (LLaMA-3-8B), RCR prevails. Both of these families are better compared to baseline approaches across a number of QA tasks.\nThe authors also introduce a new QA dataset found \"in-the-wild\" in RedditQA, where questions and (correct as well as incorrect) answers were automatically generated based on claims flagged as uncertain or incorrect.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The problem of balancing trust between contextual and parametric knowledge is important\n- Comprehensive experimental setup and ablation study\n- Well written and easy to understand\n- Proposed methods outperform baselines and improve", "weaknesses": "- There is no real clear winner in terms of methods - in the case of GPT-4o, implicit and explicit SCR share the laurels between SCR methods, while InternalConf RCR is not far behind.\n- The results are very densely presented and it is not clear what the takeaway is - using CR-DPO is the best approach if one can tune the model, however otherwise the instructions are very dataset dependent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hEwE55Y6eH", "reviewer": "Reviewer_UYZY", "summary": "This work addresses the challenge of dealing with unreliable external contexts fed into LLMs. It introduces the concept of “situated faithfulness” suggesting that LLMs should evaluate whether to trust the external context or disregard it, answering the user’s question based solely on their parametric knowledge. To tackle this issue, two approaches are designed: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR prompts the LLM to assess the external context itself, while RCR uses predefined rules to determine the trustworthiness of the external context. The authors also propose a training method to enhance SCR, Confidence Reasoning Direct Preference Optimization (CR-DPO).\nExperimental results validate the effectiveness of these methods, revealing several interesting findings.", "questions": "- The authors apply multiple ways/prompts to estimate the confidence of the answers/context. I’m curious how calibrated these confidence scores are? Can you report the calibration metrics (AUROC, ECE) for these scores as well?\n- Typos:\nLine 022: self-access -> self-assess", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Situated faithfulness is a worth-investigating research problem, as knowledge conflict could occur in practical usage.\n- The designed task RedditQA is a valuable resource for the community interested in this area. It would be beneficial if the author could open-source this dataset.\n- The proposed methods are effective based on the conducted experiments.", "weaknesses": "- Both SCR and RCR introduce additional computational overhead and increase latency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BiNiF3dJyJ", "reviewer": "Reviewer_2Etp", "summary": "The paper explores how large language models (LLMs) handle internal knowledge versus external context, particularly when these sources conflict with each other. The authors argue that robust models should dynamically adjust their reliance on external context based on confidence scores in both internal knowledge and external context. To achieve this, they introduce self-guided confidence reasoning (SCR) and rule-based confidence reasoning (RCR) approaches and evaluate both open-source and close-source models across several QA datasets, including their own RedditQA, which they developed to study these behaviors. Results show that SCR is more effective for models with advanced reasoning abilities like gpt-4o, while RCR performs better for smaller models. They also propose Confidence Reasoning Direct Preference Optimization (CR-DPO) to create contrastive data and further enhances model performance.", "questions": "When facing difficult problems, models sometimes fail to follow instructions and refuse to do so or produce illogical responses, especially when the model calibrates well. Have you done any check on the quality of data generated by CRDPO?  Specifically, how many samples were created, and what measures were used to ensure high data quality?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Provides a comprehensive study on how LLMs manage situational faithfulness to external contexts. Very complete work with multiple approaches, new dataset and better solutions named CR-DPO.", "weaknesses": "While not critical, there are some minor writing flaws and citation issues, such as “model_answer” on page 20 and line 723 on page 14.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jzRWW8Nl9l", "reviewer": "Reviewer_WXh7", "summary": "This paper uses confidence as a metric to adaptively determine when to rely on external knowledge and when to trust the model's parametric knowledge. The authors propose two pipelines: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). Their findings suggest promising avenues for enhancing situated faithfulness in LLMs.", "questions": "Refer to the weakness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is clear, and the analysis is comprehensive. The authors conduct an in-depth examination of the situated faithfulness problem, covering a variety of QA benchmarks and models, including both closed-source models like GPT-4o and open-source models like LLaMA-3-8B. In addition to training-free methods, the authors also introduce a training-based approach, CR-DPO. \n2. The authors address a significant problem: knowledge conflict, which is drawing increasing attention as large language models (LLMs) demonstrate greater capabilities. As LLMs evolve, effectively managing conflicting information between internal and external knowledge sources becomes increasingly critical.\n3. The performance is satisfactory. Experiments on various benchmarks show both SCR and RCR has good performance compared with the baselines.", "weaknesses": "1. Limited Evaluation of baselines: While SCR and RCR are the primary contributions, the paper could benefit from a more extensive comparison to existing methods in RAG like Active RAG using confidence as a metric to adaptively decide when to use the external information. This omission impacts the novelty of the work.\n2. Sensitivity to Confidence Thresholds: The effectiveness of some RCR methods hinges on predefined confidence thresholds, but the paper lacks a thorough analysis of threshold sensitivity across diverse tasks.\n[1]. Active Retrieval Augmented Generation, Jiang et al. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)