KScope: A Framework for Characterizing the Knowledge Status of Language Models
Authors: Yuxin Xiao
Shan Chen
Jack Gallifant
Danielle Bitterman
Thomas Hartvigsen
Marzyeh Ghassemi
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge Conflict
Knowledge Characterization
Large Language Model
Primary area: evaluation
TL;DR: We propose a taxonomy of five knowledge statuses and introduce KScope, a hierarchical testing framework to characterize LLM knowledge status.

Abstract: Characterizing a large language model's (LLM's) knowledge of a given question is challenging.
As a result, prior work has primarily examined LLM behavior under knowledge conflicts, where the model's internal parametric memory contradicts information in the external context.
However, this does not fully reflect how well the model knows the answer to the question.
In this paper, we first introduce a taxonomy of five knowledge statuses based on the consistency and correctness of LLM knowledge modes.
We then propose KScope, a hierarchical framework of statistical tests that progressively refines hypotheses about knowledge modes and characterizes LLM knowledge into one of these five statuses. 
We apply KScope to nine LLMs across four datasets and systematically establish:
(1) Supporting context narrows knowledge gaps across models.
(2) Context features related to difficulty, relevance, and familiarity drive successful knowledge updates.
(3) LLMs exhibit similar feature preferences when partially correct or conflicted, but diverge sharply when consistently wrong.
(4) Context summarization constrained by our feature analysis, together with enhanced credibility, further improves update effectiveness and generalizes across LLMs.

KScope: A Framework for Characterizing the Knowledge Status of Language Models
Yuxin Xiao', Shan Chen', Jack Gallifant?, Danielle Bitterman", Thomas Hartvigsen', Marzyeh Ghassemi Massachusetts Institute of Technology, Harvard University, UUniversity of Virginia yuxin102@mit.een
Abstract
Characterizing large language model's (LLM's) knowledge of a given question is challenging. As a result, prior work has primarily examined LLM behavior under knowledge conflicts, where the model's internal parametric memory contradicts information in the external context. However, this does not fully reflect how well the model knows the answer to the question. In this paper, we first introduce a taxonomy of five knowledge statuses based on the consistency and correctness of LLM knowledge modes. We then propose KScope, a hierarchical framework statistical tests that progressively refines hypotheses about knowledge modes and characterizes LLM knowledge into one of these five statuses. We apply KScope to nine LLMS across four datasets and systematically establish: (1) Supporting context narrows knowledge gaps across models. (2) Context features related to difficulty, relevance, and familiarity drive successful knowledge updates. (3) LLMs exhibit similar feature preferences when partially correct or conflicted, but diverge sharply when consistently wrong. (4) Context summarization constrained by our feature analysis, together with enhanced credibility, further improves update effectiveness and generalizes across LLMs.
Introduction
55 72 memorize information from their training corpora as parametric knowledge [3] They may further incorporate grounded and up-to-date contextual knowledge from user prompts for www.ledgeinttnssiie tasks [14] 34 60). Knowledge conflicts can arise when an LLM's parametric knowledge contradicts the contextual input 77]. Existing work has sought measure 76].
LLMs 15,
24,
5 58
and understand L.M behavior under these conflicting conditions [1 30
68
However, prior studies on knowledge conflicts do not fully characterize an LLM's underlying knowledge of a given question. Recent work usually represents LLM's knowledge via the most likely response 30 62 69), which may overlook the coexistence of multiple competing modes in answer distribution. Moreover, entropy-based uncertainty metrics [T] 39 capture overall uncertainty instead of mode structure, and would assign similar entropy values (approximately 1.37) to distributions [0.45, 0.45, 0.1] and [0.6, 0.2]. Yet the first distribution reflects conflicting knowledge, while the second shows consistent preference.
To address this gap, we deñine taxonomy of five knowledge statuses along two key dimensions and propose KScope, hierarchical testing framework for characterizing knowledge status. As shown in Figure | we assess knowledge consistency by examining the size of an LLM's mode set, and evaluate knowledge correciness relative to the ground truth. Based these two dimensions, identify five distinct knowledge statuses: (I) consistent correct, (2) conflicting correct, (3) absent, (4) conflicting wrong, and (5) consistent wrong. We construct an empirical response distribution by repeatedly sampling the target LLM, and leverage KScope to progressively refine hypotheses about its underlying knowledge modes.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Conrectnesss
Taxonomy Knowledge Status
Example
Quaction #
Who received the first Nobel Prize plysics?
Absent
Confilcting Correct
Consistent Correct
Supporting Context of
3
The Nohel Physics awarded
Support Set
WR MC
WR MC
WR
91: Wilheim Räntgen (WR)
1/ (WR. MC.
>/ (WR. MC)
>g (WK)
vo: Marie Curie (MC)
93 Albert Einstein (AF)
Conflieting Wrong
Consistent Wreng
Correct Anewer
go: Wilhohm Rintgen
3,,
Knowkklge Upsiat LLM
Panumetric Knowlodge Status: Conflicting Correct
Parametric Knowledge Modes: y, - (WR, MC)
Cornect Answer Wrong
By (NC AE3
Y/ (AB)
Supporting Compxt Contextual Knowlelge Stars: Consistent Correct
Contextual Roowledge Modes: V, (WR)
-
< 130
Di
Consistency
Figure 1: We propose a taxonomy of five knowledge statuses based on the consistency and correctness of LLM knowledge modes. We illustrate the taxonomy using an LLM's s parametric knowledge modes , in ohree-option classification task. This formulation also applies to contextual knowledge modes y and generalizes to open-ended questions or classification tasks with more options.
We evaluate the knowledge status of nine LLMs across four datasets (Section 5), and find that supporting context increases the proportion of consistent correct knowledge across all datasets and models. In the multi-choice setting, we note that the two bealthcare-rellted datasets (Hemone [64] and PubMedQA (26) exhibit lower levels of consistent correct knowledge than the general-domain datasets (NQ (32 and HotpotQA (74)), both before and after providing context. Within each model family, larger LLMs exhibit higher proportions of consistent correct parametric knowledge. Among them, Llama-3 although these differences narrow when context is introduced. Finally, we show that noisy retrieval and open-ended question settings substantially reduce the effectiveness of updating LLMs exhibit consistently correct knowledge.
15 achieves the highest proportion, followed by Qwen-2.5 772 and Gemma-2 [55],
We next examine what features of the input context successfully update LLM knowledge to the consistent correct status (Section 6). We select eleven context features across three categories- difficulty, relevance, and familiarity, and show that all three feature categories contribute, with context length and entropy consistently valued across statuses. We also find that LLMs prioritize context features similarly when they partially correct or exhibit confficting knowledge, regardless of correctness. In contrast, the consistent wrong status shows relatively low correlations with other statuses, suggesting that overcoming strongly held false beliefs may require distinct context features.
Finally, we explore context augmentation strategies to improve the success rate of knowledge updates (Section[)) 77. We find that context summarization with constraints informed by our feature importance analysis outperforms naive summarization. When combined with enhanced context credibility [70), this approach improves the proportion of successful knowledge updates by 4.3% on average across all statuses-even GPT-4o 24 which was not included in our feature analysis.
We summarize our contributions in this paper as follows:
We define a taxonomy of five knowledge statuses based consistency and correctness, and propose KScope, a hierarchical testing framework to characterize LLM knowledge status.
We apply KScope to nine LLMS across four datasets, and establish that supporting context substantially narrows knowledge gaps across model sizes and families.
We identify key context features related to difficulty, relevance, and familiarity that drive successful knowledge updates.
We reveal how LLM feature importance differs based on parametric knowledge status, showing similarity under conflict but divergence when consistently wrong.
We validate that constrained context summarization, combined with improved credibility, significantly boosts successful knowledge updates across all statuses and generalizes well.
OOur code is available at uttttps://githhin httpa///./....aooyyya//////////0

---

KScope: Knowledge Status Characterization via Hierarchical Testing
Step
Statistical Test
Null Hypothesis
Alternative Hypothesis
If Significant p-value
If Insignificant p-value
(1) Test for the Significunce of
Ose-Sided Exact Binomial
P(f(s) = Fif(s) #
R((K) #>)
Absent Knowledge
Procsed
Invalid Answers
(2) Test for Uniferm
Two Sided Exact Mulinomial Test
Pi B Nw E>
P: / br e
Proceed J
Absent Knowledge
Guessing
(3) Test for Conflicring
Likellhood Ratio Test
J:t ys
bilh pa
Ps
Procced Accardingly
Abseen Knowladge
Knowledge
(: Ps
1nli 92
for Consistent
Ose-Sided Essact
86 K-A--
(0)
Consistent Cornect Wrong
Knolwedge
Confliering Cornect Wronn Knowledge (depending
Knowledge
Binomial Test
(b)
(depending correctness)
Figure 2: We propose KScope, a hierarchical testing framework to characterize LLM knowledge into one of the five identified statuses. We note that our framework generalizes to questions with larger support sets by repeating Step 3 to iteratively refine hypotheses about the knowledge mode set.
2 Related Work
Knowledge Characterization. LLMs have been shown to memorize world knowledge from their Existing work interprets this memorization mechanism by and benchmarks the factual accuracy of recalled information Researchers have employed multi-round prompting to improve
training corpora [3
4
5
10 44
47
58
probing activation patterns 8
23
33
against knowledge graphs 36
78
82
the consistency 43 61
and calibration [9 B1 73 of the knowledge elicited from LLMs. In this
work, we identify five knowledge statuses that an LLM may exhibit with respect to given question and explicitly characterize them through a hierarchical statistical testing framework.
Knowledge Conflict. Knowledge conflict can arise within an LLM's parametric memory, within the provided context, between the two [1 Prior work has examined various contextual factors that influence a model's degree of compliance 53 54 [7 77 and introduced metrics to quantify the persuasiveness of context [ 39]. These studies find that LLMs tend follow the context when they are uncertain (45 68 76 or when there is confirmation bias between the model's memory and Large-scale evaluation benchmarks and pipelines have also been developed However, prior work usually uses a single sampled
the context [30 69
to facilitate research in this area 20
51 62
response to represent model memory 30
69 and applies entropy- based measures [ 39] to
assess conflict, both of which overlook the mode structure of the response distribution. In contrast, we define five knowledge statuses measure the consistency and correctness of knowledge, rigorously test different mode structures, and stratify our analysis of model behavior based on these statuses.
Knowledge Update. To incorporate accurate and up-to-date information, retrieval-augmentn generation (RAG) systems 66 [4 17
46 49 retrieve relevant context from external corpora to support LLMs in cnooledgg-inttesiiee tasks. The retrieved context can be further augmented to enhance the effectiveness of knowledge updates [53
57 70]. Other approaches directly edit 60] or apply mechanistic interventions at inference time 27 35 80 In this work, we examine knowledge updates under both gold and noisy retrieval settings and systematically evaluate various context augmentation strategies tailored to each knowledge status.
model's s parametric knowledge
[2]
18
59
3
LLM Knowledge Status and How to Characterize it
3.1
LLM Knowledge Status: Consistency and Correctness
When analyzing an LLM's knowledge, we focus on two key dimensions:
1. Consistency:
How consistent are the model's knowledge modes? That is, does it exhibit
single coherent belief or multiple conflicting ones?
2. Correctness:
Does the set of the model's knowledge modes include the correct answer?
To formalize this, consider a question-answe pair (x, vz) We define the parametric knowledge of an I.M f, with respect to the question x, as the conditional multinomial distribution p -( (=) over the support set y {y1...., ya). yy)). where pt f((s æ). We further define the set of parametric
3

---

(a) Distribution Parametric Knowledge Statuses (Multi-hooice Setting, Context)
Hemono
PubMedOA
NO
HelpotOA
Gemma28-
7.4
10.7
3.7
35.6
22.8
2.5
50.0
31.0
11.6
13.
15.3
22.0
3.2
25.6
Demma-415
14
Cumma-27B-
31.3
7.7
42.0
33.6
12.5
2.4
53
67.7
7.6
20.5
47.3
24.9
19.2
Ll
40.9
22.6
21.4
2.0
12.8
13.
81
2.0
35.6
36.5
25.0
12.0
21.5
35.4
23.8
18.5
4.7
17.8
60
Limma-tis
42.2
19.8
8.4
2.5
21.2
50.0
11.4
8.1
1.
36.7
09.
19.3
12.
27
16.1
16
16.1
23.2
2.1
12.1
Llama-70B
49.9
20.5
1.5
2.6
25.5
54.5
4.3
9.0
0.7
40.5
52.2
16.9
1.1
21
27.6
53.9
16.4
1.5
2.2
25.8
-40
Qwen-3B
36.0
10.3
7.8
29.8
13.5
5.6
3.0
36.9
21.3
4.5
21.4
27.0
Qwean
42.5
2.5
41.1
10.7
5.2
5.6
38.7
16.0
3.4
47.2
26.0
20
Qwen-14B
22.0
Coree
Contetii
Conee
Weog
Condecton
Wrong
Con-letent
Confleting
Aboott
ConBecing
Writ
Conaptann
Weig
Condertent
Correc
Conceeting
Corned
Absont-
Contheting
Wesld
Cos wocemn
Consiaturs
Cornec
Contetiin
Canee
Contotaa
Wron
Com
Whong
(b) Distribution of Contextual Knowledge Statuses (Mult-Choice Setting, Gold Context)
Heemonc
PubMedQA
NQ
BltpotQQ
Gemma 28
40.5
5.1
1.0
4.2
49.3
65.9
8.2
1.7
2.1
21.1
65.0
2.3
20.7
0.6
11.7
$1.5
2.7
0.3
0.1
5.4
Gemma-9B
51.4
1.4
27.1
8.5
0.7
1.6
27.9
65.5
1.5
25.3
2.5
20.2
2.6
80
Gemma 27B
44.7
1.3
23.7
0.7
22.0
00.8
1.3
23.9
96.2
2.4
Liama
10.8
1.5
9.0
8.2
2.6
600
Linma-Ii5
49.8
21.1
2.5
23.9
77.1
4.7
9.1
1.4
18.7
97.4
5.4
7.7
95.2
2.2
0.5
DVB
2.2
Liama 708
622
21.9
0.0
0.8
14.5
78.8
2.1
0.2
8.5
18.4
74.4
9.4
0.4
1.3
14.6
92.0
4.5
0.1
0.1
3.3
40
Qwen 3B
42.
17.3
4.3
2.6
25.7
43.2
5.6
57.8
2.0
1344
74.2
8.4
2.9
2.0
12.5
88.3
7.7
0.8
0.1
3.2
Owen-7B 3.0
15.8
1.3
1.5
28.3
73.6
5.8
14
0.8
10.3
79.5
5.2
3.1
12.0
$4.7
2.5
0.8
0.0
2.6
-20
O4en-14B
50.6
1.4
1.0
81.8
3.9
96.4
Contrient
Curnect
Confictinn
Cutteet
Contert
Concintin
Wrigg
Conoettenl
Conficking
Absent
Cunfic Lun
Comiitent
Woog
Consittont
CUffect
Conticiing
CXTTect
Absent
Conticting
Work
Comtionn
Currect
Conticting
Cotrect
Conticting
Wmig
Cometcionn
Wrotg
Figure 3:
Characterization results from applying the KScope framework to nine LLMs across
four datasets. Overall, most LLMs exhibit the highest proportion of consistent correct parametric knowledge status, which further increases when gold context is provided.
knowledge modes D/- modes(p) S y as the subset that satisfies the following condition: p; Pj for any y/. Mj E V/P and p; > Pk. for any y/k & y,. In essence, the knowledge modes form a plateau of high-probability elements within the support set that are distinguishable from the rest.
To assess the consistency dimension, we examine three possible structures of yr; (1) = y. (2) 1 < 1D'p < [3) and (3) D', = 1. where - denotes the cardinality of set. We evaluate correctness by checking whether the ground-truth answer y/ E ypp Based on these two dimensions, we formulate taxonomy of five parametric knowledge statuses P = slatus()p), as illustrated in Figure I that characterize the knowledge of f with respect to the question æ::
1. Consistent Correct Knowledge: [3%p = and y" E Dpp-
2. Conflicting Correct Knowledge: < |>'p < [)]] and n² E 3
3. Absent Knowledge: 3%p y
4. Conflicting Wrong Knowledge: 1)pl < | and y² $ Dp--
5. Consistent Wrong Knowledge: 13pp and y of Dp-
When supporting context e is available for the question-answar pair, we define the LLM's contextual knowledge as q f( a., c). Analogously, assign its contextual knowledge status Q status())g) based on the corresponding knowledge modes V/& modes( q) C y
3.2 Challenges in Operationalizing Knowledge Status
While the taxonomy introduced in Section 3.1 offers a principled view of LLM knowledge status, applying it in practice poses several challenges. First, the true underlying distributions of LLM knowledge are unobservable. Second, even under the same knowledge status, models may behave differently. For instance, when an LLM lacks sufficient knowledge about question, it may either respond randomly or refuse to answer altogether 22 66
To address the first challenge, we approximate the latent knowledge distributions using empirical sample frequencies. Specifically, we first generate M paraphrases of a given question to reduce prompt sensitivity [8]. then collect N chain-of-thouge these paraphrases. For open-ended generation, we define the support set y by semantically clustering the N samples (3T) For multiple-choice tasks, y is simply the set of given options.
responses (65] from the target LLM using

---

(a) Context Induced Shifts from Dataset Perspective (Multi Choice Setting, Gold Context)
Coasistent
Contlectng Correct (
Contlicting Wzong (+)
Consistent
Correct (T)
Absent (4)
Wrong (
16.2
.125
14.3
Fareentage
20.0
12
32
3,5
34
144
60
3.0
25
20
FubMedQA
HetpooOA
Increase Contect
Decrease w/
Context-Inducee Shifts Model Perspective
Setting, Gold Context)
Coasistent
Conllcting
Confleting Wrong (4)
Consistent
Correct (T)
Correct 4
Absent. ( ())
Wrong (44
+30.
300
42
4204
Percestage
60-
15-
15-
4.5-
27
60-
30
10-
3.0
18
Demma-28
Llama-BB
Llama-703
Qwan-3B
Qwen-11B
Increuse w/
Context
Camma-9it
Qwwn-78
Z2 Devrraaion wy Content
Figure 4: Context-induced shifts in knowledge status distributions. Supporting context increases the proportion of consistent correct knowledge across all datasets and models. The Llama family and larger models within each family achieve higher proportions of consistent correct knowledge, although the gaps narrow with context.
We then account for the second challenge by considering the possibility of invalid model responses. These include hallucinating answers outside the support [22 or refusing respond in high- stakes applications [66]. Let N' denote the number of invalid responses. We estimate the empirical
distribution p (and analogously, q) by: P. N-M En ([fx(z) vil, Vy, e..
3.3 KScope: Knowledge Status Characterization via Hierarchical Testing
To bridge the gap between the true latent knowledge distributions discussed in Section 3.1 and the empirical distributions estimated in Section 3.2 we introduce KScope, hierarchical testing framework for knowledge status characterization. We illustrate the testing details in Figure 2
Step 1: Test for the Significance of Invalid Answers. We first assess whether the model exhibits a higher tendency to produce invalid responses via one-sided exact binomial test.
Step 2: Test for Uniform Guessing. Next, we test whether the LLM'' empirical response distribution significantly deviates from uniform distribution using a two-sided exact multinomial test.
Step 3: Test for Conflicting Knowledge. Subsequently, we perform a set of likelihood ratio tests to refine the model's knowledge mode set. Alternatives whose estimated probabilities violate their own inequality constraints are immediately rejected. If multiple alternatives remain significant after Bonferroni correction, we select the one with the lowest Bayesian Information Criterion (BIC). For larger support sets, we repeat this step to remove low lwwprbbbayy elements from the mode set.
Step 4: Test for Consistent Knowledge. The previous step reduces the model's 's knowledge mode set to two elements. Conditioned on the selected alternative in Step 3, we then test whether the model assigns significantly different probabilities to the two remaining elements using two one-sided exact binomial tests. As before, we discard invalid alternatives and, multiple alternatives are accepted, select the one with the lowest BIC.
4 Experiment Setup
Datasets. We focus on four tasks, two from the healthcare domain and two from the general domain:
Hemone (4 is a healthcare dataset extracted from a regularly maintained oncology refer- ence database. consists of 6,212 clinical study instances, each comparing the efficacy of a regimen versus a comparator for a given medical condition, labeled superior, inferior, or
5

---

lel Pacariettc Knowladge
Us Contactial Keowwiedge Gad Cenrrst)
ed Costoriaal Eacodedge Noay Contrst)
16 Phcemntzic Szonledge No Cookens)
(o) Centaciaa Kncorkeqe
Genera di.
5T.4
No Contecti
01
1.4
67
13
25
13
2n
Gold Costest)
63
2-3
I60
Onnma 278
673
74
26.9
133
10.2
W-2
13
9A
0.0
24
681
37
22.
0.4
13
59
94.
03
34
643
23
67
02
4.5
23
174
10
14
07
83
16.1
23
035
3211
1.3
19:1
93
53
233
18ama-700
539
15A
15
22
1611
45
3.1
81
23
T3L
313
64
12
15.4
073
8.0
11.
634
1.5
4.5
0.3
4.3
Owes 28
21.4
Owea-7
17.6
32
0.0
10.1
3.0
20
Owen k4
19.2
2.1
0.1
6.6
18.8
14.5
15
M11
23
Abrest
Abwas:
Conu
Ibwas
Ovd
Figure 5: Characterization results from applying KScope to nine LLMs HotpotQA across different settings. Compared to (b), where gold context in the multi-choice setting enables more consistent correct knowledge, (c) noisy context and (e) the open-ended setting yield lower update success. Without context, the Gemma family shows more absent knowledge in (d) the open-ended setting than in (a) the multi-choice setting, whereas the Llama and Qwen families mostly show the opposite trend.
no difference. To reduce positional bias, we permute the order of each regimen comparato pair. The context for each question consists of abstracts from the associated PubMed articles. [26] consists of 1,000 medical research questions, each labeled with yes, no, or maybe answer. The context comes from the corresponding PubMed abstracts.
Pu
NQ 32 contains 3,596 Google search queries, retrieving Wikipedia pages as context.
HotpotQA [74] contains 6, 119 multi-hop reasoning questions in the general domain, using sentence-level supporting facts extracted from relevant Wikipedia articles as context.
Unless otherwise specified, all the supplied contexts here contain ground-truth evidence. When comparing LLMs' knowledge statuses in the multi-choice setting, we convert NQ and HotpotQA into threc-option classification tasks. Following [70], we prompt GPT-4o [24) to generate two additional wrong options for each question (details in Appendix (A). We note that although not evaluated here, KScope remains applicable to classification with more options, as discussed in Section
Implementation Details. We evaluate nine instraction-tuned LLMs spanning three model families: Gemma-2 (2B, 9B, 27B) (55) Llama-3 (3B, 8B, 70B) [15). and Qwen-2.5 (3B, 7B, 14B) [72]. We keep the LLMs' sampling distributions unchanged by setting the temperature to 1, and fix the significance level used in KScope at a = 0.05. A hyperparameter search on Hemone using Llama-8B (details in Appendix A) shows that knowledge status characterization empirically stabilizes after N = 100 samples, using M = 20 paraphrases per question.
Q1: How Does Context Update LLMs' Knowledge Status?
Using the setup in Section we first characterize the distributions of LLMs' parametric and contextual knowledge statuses in the multi-choice setting and examine how gold context induces shifts between them in Section [5 We further investigate the effects of noisy context in Section 5.2 and analyze knowledge status distributions in the open-ended setting in Section 5.3
5.1 Knowledge Status in the Multi-Choice Setting with Gold Context
We apply KScope to the nine LLMs across the four datasets in the multi-choice setting and present the results in 3 When relying solely on parametric knowledge (Figure (a)). LLMs exhibit consistent knowledge- -whetherweeeeeeeeeeeeeeeee correct or wrong Prong-more frequently than conflicting knowledge. When conflicting knowledge occurs, the correct answer is usually among the knowledge modes. Some outliers show higher proportion of absent knowledge, such Gemma-2B on Hemone.
When LLMS are provided with gold context (Figure 3 (b)), the proportion of consistent correct knowledge status significantly increases across all datasets and models, with the largest improvement in HotpotQA and the smallest in Hemone. This highlights the effectiveness of retrieval augmented generation 6 [4 T7 34], which aims to enhance LLMs' knowledge with relevant external infor- mation. However, in some cases, context may confuse LLMs, leading them to guess randomly and increasing the proportion of absent knowledge. For example, this occurs with Qwen-3B on PubMedQA and the Gemma family on NQ, likely due to longer context lengths in these datasets- an issue we further investigate in Section 6

---

Consistent Correct
Conflicting Correct
Absent
Conflicting Wrong
Consistent Wrong
Normalzze Proquonny (%)
80-
62.5
60-
56.5
200
36.5
66.5
571
52.4
56.6
50.8
52.8
$2.8
30.0
50.0
40
00001
Context Length
100
Unirg Tkns
1mmb Sim
5003 ROUGE-2 RI
12 ROUGE-2 P
0000 Queston Entragy
DXK
Context Bntropy
Figure 6: Context features appearing among the top five most important in at least 50% of the cases for each parametric knowledge status. Each color indicates a feature category: orange for difficulty, blue for relevance, and purple for familiarity. Across all statuses, context length and entropy consistently rank among the most important features.
We further examine how gold context shifts the distribution of each knowledge status. Figure 4 (a) presents dataset-level shifts, where distributions of knowledge status averaged across the nine LLMs. We observe that Hemone and PubMedQA exhibit lower proportions of consistent correct knowledge than NQ and HotpotQA, both before and after context is provided. Additionally, the gold context slightly increases the ratio of conflicting correct knowledge in Hemone and absent Snowledge in PubMedQA, highlighting the challenges introduced by medical-domain context
p 50 56 63]
Within each LLM family, we find that larger models consistently exhibit higher proportions of
consistent correct knowledge, both before and after providing context (Figure 4 (b)) 5 37 Llama
achieves the highest proportion of consistent correct parametric knowledge, followed by Qwen and Gemma, although this gap narrows once context is introduced. Appendix B details how gold context shifts each LLM's knowledge among different statuses on each dataset in the multi-choice setting.
5.2 Knowledge Status in the Multi-Choice Setting with Noisy Context
To investigate LLMs' knowledge status under more realistic retrieval conditions, we apply KScope to the nine LLMs using the top ten Wikipedia paragraphs retrieved for each HotpotQA question, which may or may not include the gold supporting context. As shown in Figure 5 (c). noisy context results in much lower success rate of updating models to consistent correct knowledge compared gold context in (b), highlighting the importance of retrieval quality in RAG systems (6 14
17
34 46 49].
When the retrieved noisy context lacks evidence for the ground- -tuth answer, models either refuse to answer, leading to more absent knowledge, or are misled into producing consistently incorrect answers. More details on noisy context-induced shifts in knowledge status are in Appendix B
5.3 Knowledge Status in the Open-Ended Setting with Gold Context
To demonstrate the generalizability of KScope to open-ended questions 22 67],
we apply it to
characterize the knowledge status of the nine LLMs on HotpoiQA, this time without providing any pre-defined answer options. Specifically, we generate responses per question and semantically cluster [31 them using gemma-2-9b-in [55] Based the size of clusters and the number of invalid answers, we follow the procedure in Section 3.3 to infer the LLM''s knowledge status.
As shown in Figure 5 (d). knowledge status distributions vary notably across model families. Without pre-defined options or contextual support, the Gemma family often refuses to answer, leading to a higher proportion of absent knowledge compared to the multi-choice setting in (a). In contrast, Llama and Qwen show a substantial increase in the consistent correct knowledge under these conditions, with the exception of Qwen-14B. Gold context still significantly boosts consistent correct knowledge in the open-ended setting in (c), though the improvement is smaller than in the multi-choice setting in (b). Appendix B provides more details the shift in knowledge status in the open-ended setting.
Q2: What Context Features Drive the Desired Knowledge Update?
Based on the results in Section 5 we seek to understand what context features drive increases in consistent correct knowledge. We introduce three categories of context features in Section 6.1 and inspect feature importance for each knowledge status Section 6.2] To enable direct comparison across datasets, we focus on the multi-choice setting with gold context in the following sections.
7

---

6.1 Context Features
We consider eleven context features across three categories:
Difficulty: (1) Context Length: (2) Readability, measured by the Resch-Kincake readabil- ity test [29] which is based on the average number of words per sentence and syllables per word; and (3) Number of Unique Tokens (# Uniq Tkns) after lemmatization.
Relevance: (I) Embedding Similarity (Emb Sim), computed as the cosine similarity be- tween the embeddings of each question and its corresponding context using text-embedding- 3-large 400; and (2-4) ROUGE-2 Recall, Precision, and F1 (ROUGE-2 R/P/F), based on the bigram overlap between each question and its corresponding context.
Familiarity: (1-2) Question and Context Perplexity, and (3-4) Question and Context Entropy, as measured by each L.M..
We define binary label indicating whether context successfully updates an LLM's parametric knowledge to the consistent correct status. As described in Section [ for each combination of dataset, LLM, and initial parametric knowledge status, we formulate stratified binary classification task. We apply logistic regression with L2 regularization to the extracted features and discard cases where performance does not exceed dummy baseline in Macro-F1, due to extreme class imbalance. Implementation details and full regression results are in Appendix
We compute feature importance by averaging the absolute SHAP values [3 of each feature within each stratified combination. We then calculate the normalized frequency with which each feature appears among the top five most important features across datasets and LLMs. This yields frequency- based ranking for each initial parametric knowledge status.
6.2 Feature Importance for Context-Driven Knowledge Update
We plot in Figure 6 the context features that appear among the top five most important in least 50% of the cases identified in our experiments. The results include features from all three cat- egories: difficulty, relevance, and familiarity. This aligns with [I] [57,, which find that context relevance influences context persuasiveness. Among the five parametric knowledge statuses, the consistent wrong knowledge status exhibits flatter distribution of top feature frequencies pared to the others, suggesting that LLMs in this status place less emphasis on any specific feature. Notably, context length and entropy consistently rank high across all statuses.
Consistect
Confficting
Coafictin Wroong
Consistect Wiroog
To assess whether LLMs in distinct parametric knowledge statuses prioritize context features similarly, compute Spearman rank correla- tions with Bonferroni correction from feature importance rankings. As shown in Figure 7 correlations are statistically significant between consistent correct and both confficting correct and absent knowledge. This suggests a confirma- tion bias 30 69 when context least partially aligns with the model's knowledge modes, the model attends context features similarly. We also observe significant correlation between conflicting correct and conflicting wrong, indi- cating similar feature preferences during knowl- edge conflict [45] 68 76), regardless of correct- ness. In contrast, the consistent wrong status shows relatively low correlations with others, implying that overcoming a firmly held wrong belief may require different context features.
Correct
Correct
Abseat
1.0
Consistent
1.000
0.809
0.791
0.718
0.491
Correct
0.9
Confticting
0.809
1.000
0.755
0.564
Correlation
Absent
7.791
0.755
1.000
0.755
0.527
0.7
Conflicting Wrong
0.718
0.882*
0.755
1.000
0.591
0.6
Consistent Wrong
0.491
0.564
1.000
Figure 7: Spearman rank correlation among para- metric knowledge statuses based on feature impor- tance rankings. An asterisk indicates statistical sig- nificance at Bonferroni-adjusted C'aj = 0.005.
7 Q3: What Context Augmentations Work Best Across Knowledge Statuses?
In this section, we leverage the insights from our feature importance analysis in Section to improve knowledge updates in LLMs. We again focus on the multi-choice setting with gold context.

---

Consistent Correct
Condicting Correct
Absent
Confictingg Wrong
Consisteat Weong
Change in Surmees Bot (50
Knowledge Upotate (1)
Llutsa
Qwen
GPT
Llama
Qreecs
GP?
Llama +B
Crveen -14.8
CPT
Zlama -05
Creen -144
CPT
Llama -06
Quwon
GPT
-80
-14
-60
-b0
-1.9
-t0
-1o
-14.8
-40
Credibuity
Nalve Summarization
Constrained Summarisstion
Credisllity
Constrained commmrriaatinn
Figure 8: Absolute change (%) in the success rate of knowledge updates for each context augmentation strategy, relative to the original context and averaged across datasets. Integrating credibility metadata into constrained summarization improves the success rate by 4.3% average across LLMs and statuses, and generalizes well to GPT-4o.
7.1
Context Augmentation Strategies
We apply the following augmentation strategies to the original context of the datasets in Section
To enhance context credibility, we append the relevant PubMed metadata to the context for Hemone and PubMedQA, and the corresponding Wikipedia article titles for
Credibility 70)
NQ and HotpotQA. When sampling responses with context, we instruct LLMs to prioritize the credible context over their internal parametric knowledge (details in Appendix (D).
Naïve Summarization: We leverage GPT-4o (24) to directly summarize context.
Constrained Summarization: We guide summarization with additional constraints to adjust the context features according to our analysis results. Specifically, we prompt GPT- 4o reduce both context length and the number of unique tokens during summarization. Additionally, GPT-4o is instructed to preserve the semantic content (maintaining embedding similarity with the question), retain token-level overlap (maintaining ROUGE-2 recall and precision), and ensure fluency (preserving context perplexity and entropy).
Combined: We integrate the credibility information into the context generated from con- strained summarization.
We then investigate how each augmentation strategy affects the success rate of knowledge updates compared to the original context, for each knowledge status. We evaluate these strategies three LLMs of varying sizes and families: --8---.----B--nnttrucct (15) (Llama-8B), Qwen2.5-144 1-BB (Qwen-14hh)
and GPT-4o. Although GPT--4o not included in the feature analysis in Section 6 we assess it here to examine whether our findings generalize to other LLMs. We present the average results across the four datasets in Figure g and the full details in Appendix D
Instruct 72
7.2 Effectiveness of Context Augmentations
As shown in Figure 8 constrained summarization improves the success rate across all knowledge statuses except the consistent wrong status. This aligns with our finding in Figure 7 that correcting consistent wrong belief may require different context features than in other cases. Furthermore, this constrained summarization strategy generalizes well to GPT-4o.
In contrast, naïve summarization always hurts the performance. plot Figure normalized feature space for the nine affected features on Hemone (question perplexity and entropy remain unchanged by context summarization). Both summarization methods reduce context length and the number of unique tokens, while increasing context perplexity and entropy. However, naïve summarization fails to preserve fluency and key semantic content, resulting in harder readability and lower ROUGE-2 recall. In comparison, constrained summarization improves embedding similarity, ROUGE-2 precision, and more effectively. These differences the augmented feature space explain the gap in success rates and underscore the importance of our feature analysis in Section Details for other datasets are in Appendix(D
On the other hand, credibility is more effective for the consistent wrong status, as illustrated in Figure g This suggests that when L.MM consistently holds wrong belief, adding credibility metadata to context makes it more persuasive [70]. The combined strategy retains the benefits

---

ROUGE-2 10
Contect Length
Readabllity
Uniq Tims
Emb Sim
ROUGE 2
ROUGE-2 2
Context Perpleanty
Content Entrogy
Original
Naive Sunmarimation
Constrained Summarization
Figure 9: Normalized feature space for original and summarized contexts of Hemone, measured by Llama-3. --8B-Innsruct Naive summarization hurts readability and ROUGE-2 recall, while constrained summarization yields higher embedding similarity, ROUGE-2 precision, and Fl.
of constrained summarization for the first four statuses and further improves the success rate for consistent wrong, beyond what credibility alone achieves. Overall, it enhances the success rate by 4.3% on average across all statuses and LLMs, compared to using the original context.
8 Discussion
Conclusion. In this paper, first propose taxonomy of five knowledge statuses based consistency and correctness. We then introduce KScope, hierarchical testing framework that characterizes knowledge status by progressively refining hypotheses about LLM's knowledge modes. By applying KScope to nine LLMs across four datasets, we establish: (1) Supporting context substantially narrows knowledge gaps across L.MSs. (2) Features related context difficulty, relevance, and familiarity drive successful knowledge updates. (3) LLMs attend to features similarly when their knowledge modes are partially aligned with the correct answer internally conflicting, but diverge sharply when consistently wrong. (4) Constrained context summarization guided by feature analysis, combined with enhanced credibility, further boosts update effectiveness and generalizes across models. These findings provide valuable insights into LLMs' knowledge mechanism (58) and underscore the importance of tailoring knowledge update strategies statuses in future work.
[4 60
different knowledge
Limitations. Our feature importance analysis in Section focuses eleven features across three categories. However, it remains underexplored how more nuanced stylistic context features [1] (7 70) impact LLM knowledge status. Although the KScope framework supports classification tasks with any number of options [52] [1]. we restrict our experiments to three-option classification due computational constraints. We also experiment with real-world noisy retrieval, but such context can include conflicting information 22 51 in practical RAG systems [6] 14 17
34 46 49), posing
additional challenges. We leave the investigation of how context affects LLM knowledge status under these more complex conditions to future work.
Broader Impacts. LLMS widely deployed in everyday user-chatbot applications 24 and high- stakes domains such as healthcare [2 (56) and legal services 13 [6]. However, prior work [1
39
53
70) 71 lacks formal framework for characterizing LLM knowledge status, especially as these statuses may shift in response to varied input context. The challenge becomes more concerning when training data contains misinformation 22 41I leading models develop consistently wrong beliefs. Our analysis in Section [6 shows that correcting these belief' often requires different context features than those needed for other knowledge statuses. The proposed KScope framework also relates to hallucination detection (22 25 79] and uncertainty quantification [21 31 75 in LLMs. By identifying knowledge status, helps distinguish between hallucinations due absent knowledge and uncertainty due to knowledge conflicts. These connections underscore the practical utility of KScope and broader impact on improving I.LM reliability.
57
10

---

Acknowledgements
YX is supported by the MIT IDSS Fellowship. MG is supported in part by a National Science Foundation (NSF) 22-586 Faculty Early Career Development Award (#2339381), a Gordon & Betty Moore Foundation award, a Google Research Scholar award, and the AI2050 Program at Schmidt Sciences. SC, JG, and DB are supported by the National Institutes of Health National Cancer Institute (U54CA2745 16-0AA1 ISC. D.S.B.J, RR1CCA94033-00 [JG. D.S.B.J. D. the Amer- ican Cancer Society and American Society for Radiation Oncology ASTRO-CSDG-24-12244 01-CTPS Grant (DOI: https: //doi. org/10. F00...3354/GG0 ACS. ASTRO CSDG 24 1--1244514-- 01 CTPS pp..gr 2222200210 [D.S.B.J. Patient-Centenn Outcomes Research Institute (PCORI) Project Program [D.S.B.], and the Woods Foundation [D.s.B.]. All statements this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the sponsors, and no official endorsement should be inferred.
Award
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Al- tenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint, 2023.
[2] D. A. Alber, Z. Yang, Alyakin, E. Yang. S. Rai,
Valliani, Zhang. Rosenbaum, A. K. Amend-Thomal D. B. Kurland, et al. Medical large language models are vulnerable to data-poisoning attacks. Nature Medicine, pages 1-9, 2025.
[3] Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction. In ICML, 2024.
[4] Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.2, knowledge manipulation. In ICLR, 2025.
[5] Z. Allen-Zhu and Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. In ICLR, 2025.
[6] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi. Self-RAG: Learning retrieve, generate, and critique through self-reflection. In ICLR, 2024.
[7] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
[8] C. Burns, H. Ye, D. Klein, and J. Steinhardr. Discovering latent knowledge in language models without supervision. In ICLR, 2023.
[9] F. Busch, L. Hoffmann, C. Rueger, E. H. van Dijk, R. Kader, E. Ortiz-Prado, M. R. Makowski, L. Saba, M. Hadamitzky, J. N. Kather, et al. Current applications and challenges large language models for patient care: a systematic review. Communications Medicine, 2025.
[10] N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, and C. Zhang. Quantifying memorization across neural language models. In ICLR, 2023.
[11] K. Du, V. Snæbjarnarson, N. Stoehr, J. White, A. Schein, and R. Cotterell. Context versus prior knowledge language models. In ACL, 2024.
[12] J. Fang, H. Jiang, K. Wang, Y. Ma, J. Shi, X. Wang, X. He, and T.-S. Chua. Alphaedit: Null-space constrained model editing for language models. In ICLR, 2025.
[13] Z. Fei, X. Shen, D. Zhu, Zhou, Z. Han, A. Huang, S. Zhang, K. Chen, Z. Yin, Z. Shen, et al. Lawbench: Benchmarking legal knowledge of large language models. In EMNLP. 2024.
[14] Y. Gao, Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, Sun, M. Wang, and H. Wang. Retrieval-augmented generation for large language models: A survey, arXiv preprint, 2023.
11

---

[15] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur,
A. Schelten, A. Vaughan, et al. The llama 3 herd of models, arXiv preprint. 2024.
[16] N. Guha, J. Nyarko, D. E. Ho, C. Re, A. Chilton, A. Narayana, A. Chohlas Soohlas-Wood, A. Peters,
B. Waldon, D. Rockmore, D. Zambrano, D. Talisman, E. Hoque, F. Surani, F. Fagan, G. Sarfaty, G. M. Dickinson, H. Porat, J. Hegland, J. Wu, J. Nudell, J. Niklaus, J. J. Nay, J. H. Choi, K. Tobia, M. Hagan, M. Ma, M. Livermore, N. Rasumov-Rahe, N. Holzenberger, N. Kolt, P. Henderson, S. Rehaag, S. Goel, S. Gao, S. Williams, S. Gandhi, T. Zur, V. Iyer, and Z. Li. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. In NeurIPS Datasets and Benchmarks Track, 2023.
[17] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. Realm: retrieval-augmentte language model pre-training. In ICML, 2020.
[18] T. Hartvigsen, S. Sankaranarayanan, H. Palangi, Y. Kim, and M. Ghassemi. Aging with grace: lifelong model editing with discrete key-value adaptors. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 47934 47959, 2023.
[19] B. Hou, Y. Liu, K. Qian, J. Andreas, S. Chang, and Y. Zhang. Decomposing uncertainty for large language models through input clarification ensembling. In JCML, 2024.
[20] Y. Hou, A. Pascale, J. Carnerero- T. T. Tchrakian, R. Marinescu, E. M. Daly, I. Padhi, and P. Sattigeri. Wikicontradict: A benchmark for evaluating LLMs on real-world knowledge conflicts from wikipedia. In NeurlPS Datasets and Benchmarks Track, 2024.
[21] H.-Y. Huang, Y. Yang, Z. Zhang. S. Lee, and Y. Wu, A survey of uncertainty estimation in llms: Theory meets practice. arXiv preprint 2tl.:2440.53220 2024.
[22] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang. Q. Chen, W. Peng, X. Feng, B. Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM TOIS, 2025.
[23] R. Huben, H. Cunningham, L. R. Smith, A. Ewart, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. In ICLR, 2023.
[24] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint, 2024.
[25] C. Jiang, B. Qi, X. Hong, D. Fu, Y. Cheng, F. Meng, M. Yu, B. Zhou, and J. Zhou. On large language models' hallucination with regard to known facts. In NAACL), pages 1041-1053, 2024.
[26] Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu. Pubmedqa: A dataset for biomedical research question answering. In EMNLP-IJCNLP. 2019.
[27] Z. Jin, P. Cao, H. Yuan, Y. Chen, J. Xu, H. Li, X. Jiang, K. Liu, and J. Zhao. Cutting off the head ends the conflict: mechanism for interpreting and mitigating knowledge conflicts in language models. In ACL Findings, 2024.
[28] E. Kamalloo, N. Dziri, C. Clarke, and D. Rafiei. Evaluating open-domain question answering in the era of large language models. In ACL, 2023.
[29] P. Kincaid, R. Fishburne Jr, R. L. Rogers, and B. Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical Report, Naval Technical Training Command Research Branch, 1975.
[30] E. Kortukov, A. Rubinstein, F. Nguyen, and S. J. Oh. Studying large language model behaviors under context-memory conflicts with real documents. In COLM, 2024.
[31] I. Kuhn, Y. Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR, 2023.
12

---

[32] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, Lee, al. Natural questions: A benchmark for question answering research. TACI,, 2019.
[33] W. Laurito, S. Maiya, G. Dhimoïla, O. Yeung, and K. Hänni. Cluster-norm for unsupervised probing of knowledge. In EMNLP, 2024.
[34] Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih. T. Rocktäschel, et al. Retrieval-augmenne generation for newwledge-intenste nlp tasks. In NeurIPS, 2020.
[35] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenherg. Inference-time intervention: eliciting truthful answers from a language model. In NeurlPS, 2023.
[36] X. Liu, F. Wu, T. Xu, Z. Chen, Y. Zhang, X. Wang, and J. Gao. Evaluating the factuality of large language models using large-scale knowledge graphs. arXiv preprint, 2024.
[37] X. Lu, X. Li, Q. Cheng. K. Ding, X.-J. Huang, and Qiu. Scaling laws for fact memorization of large language models. In EMNLP Findings, 2024.
[38] M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. In NeurIPS, 2017.
[39] S. Marjanovic, H. Yu, Atanasova, M. Maistro, C. Lioma, and Augenstein. Dynamicqa: Tracing internal knowledge conflicts in language models. In EMNLP Findings, 2024.
[40] OpenAl. Text-embeediigg--large OpenAl Platform. https://platform. openai com/ httes//moollsstteeeeeeeeeee -aatingg--llrre 3-rge [Accessed 2025-03-12].
[41] Y. Pan, L. Pan, W. Chen, P. Nakov, M.-Y. Kan, and W. Wang. On the risk of misinformation pollution with large language models. In EMNLP Findings, pages 1389-1403 1403, 2023.
[42] C. Peng, X. Yang, A. Chen, K. E. Smith, N. PourNejatian, A. B. Costa, C. Martin, M. G. Flores, Y. Zhang, T. Magoc, et al. study of generative large language model for medical research and healtheare, NPJ digital medicine, 2023.
[43] S. Pitis, M. R. Zhang, Wang, and Ba. Boosted prompt ensembles for large language models. arXiv preprint, 2023.
[44] U. S. Prashanth, A. Deng, O'Brien, S. V. A. Khan, J. Borkar, C. A. Choquette- Chauu, J. R. Fuehne, S. Biderman, T. Ke, K. Lee, and N. Saphra. Recite, reconstruct, recollect: Memorization LMs as a multifaceted phenomenon. In ICLR, 2025.
[45] C. Qian, X. Zhao, and T. Wu. "merge conflicts!" exploring the impacts of external knowledge distractors to parametric knowledge graphs. In COLM, 2024.
[46] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In-context eetrieval-augmented language models. TACL, 2023.
[47] A. Schwarzschild, Z. Feng, P. Maini, Z. C. Lipton, and J. Z. Kolter. Rethinking LLM memoriza- tion through the lens of adversarial compression. In NeurIPS, 2024.
[48] M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In ICLR, 2024.
[49] W. Shi, S. Min. M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W.-t. Yih. Replug: Retrieval-augmenned black-box language models. In NAACI, 2024.
[50] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature, 2023.
13

---

[51] Z. Su, J. Zhang, X. Qu, T. Zhu, Li, J. Sun, J. Li, M. Zhang, and Y. Cheng. Sitexttt [ConflittBnnk]S benchmark for evaluating the influence of knowledge conflicts in LLMss. In NeurlPS Datasets and Benchmarks Track, 2024.
[52] S. A. Tabatabaei, S. Fancher, M. Parsons, and A. Askari. Can large language models serve as effective classifiers for hierarchical multi-label classification of scientific documents industrial scale? In COLING Industry Truck, pages 163-174, 2025.
[53] H. Tan, F. Sun, W. Yang, Y. Wang, Q. Cao, and X. Cheng. Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? In ACL, 2024.
[54] Y. Tao, A. Hiatt, E. Haake, A. Jetter, and A. Agrawal. When context leads but parametric memory follows in large language models. In EMNLP, 2024.
[55] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint, 2024.
[56] A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. W. Ting. Large language models in medicine. Nature medicine, 2023.
[57] A. Wan, E. Wallace, and D. Klein. What evidence do language models find convincing? In ACL, 2024.
[58] M. Wang, Y. Yao, 7. Xu, S. Qiao, S. Deng, P. Wang, X. Chen, J.-C. Gu, Jiang, P. Xie, et al. Knowledge mechanisms in large language models: A survey and perspective. In EMNLP Findings, 2024.
[59] Wang. Z. Li, N. Zhang, Z. Xu, Y. Yao, Y. Jiang, Xie, Huang, and H. Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. In NeurlPS, 2024.
[60] S. Wang. Y. Zhu, H. Liu, Z. Zheng, C. Chen, and Li. Knowledge editing for large language models: A survey, ACM Computing Surveys, 2024.
[61] X. Wang, J. Wei, D. Schaurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
[62] Y. Wang, S. Feng, Wang, W. Shi, V. Balachandran, T. He, and Y. Tsvetkov. Resolving knowledge conflicts in large language models. In COLM, 2024.
[63] Y. Wang, Y. Zhao, and L. Petzold. Are large language models ready for healthcare? compara- tive study on clinical language understanding. In ML4HI, 2023.
[64] L.. Warner, Dymshyts, C. G. Reich, M. J. Gurley, H. Hochheiser, H. Moldwin, R. Be- lenkaya, A. E. Williams, and P. C. Yang. Hemone: A new standard vocabulary for chemotherapy regimen representation the omop common data model. Journal of biomedical informatics, 2019.
[65] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, F. H. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thouuht prompting clicits reasoning in large language models. In NeurlPS, 2022.
[66] B. Wen, J. Yao, Feng, C. Xu, Y. Tsvetkov, B. Howe, and L. L. Wang. Know your limits: A survey of abstention in large language models. TACI. 2025.
[67] Z. Wen, Z. Tian, Z. Jian, Z. Huang, P. Ke, Y. Gao, M. Huang, and D. Li. Perception of knowledge boundary for large language models through semi-open-ended question answering. In NewrIPS, 2024.
[68] K. Wu, E. Wu, and J. Zou. Clasheval: Quantifying the tug-of-war war between an LLM's internal prior and external evidence. In NeurIPS Datasets and Benchmarks Track, 2024.
[69] Xie, K. Zhang, J. Chen, R. Lou, and Y. Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In ICLR, 2024.
14

---

[70] R. Xu, B. Lin, S. Yang. T. Zhang, W. Shi, T. Zhang, Z. Fang. W. Xu, and H. Qiu. The earth is flat because...: Investigating Ilms' belief towards misinformation via persuasive conversation. In ACI, 2024.
[71] R. Xu, Z. Qi Z. Guo, C. Wang, H. Wang, Y. Zhang, and W. Xu. Knowledge conflicts for Ilms: survey. In EMNLP, 2024.
[72] A. Yang. B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint, 2024.
[73] A. X. Yang, Chen, and K. Pitas. Just rephrase it! uncertainty estimation in closed-source language models via multiple rephrased queries. In NeurIPS Workshop, 2024.
[74] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
[75] Ye, M. Yang. J. Pang. Wang, D. F. Wong, E. Yilmaz, S. Shi, and Z. Tu. Benchmarking LLMs via uncertainty quantification. In NeurlPS Datasets and Benchmarks Track, 2024.
[76] J. Ying, Y. Cao, K. Xiong, L. Cui, Y. He, and Y. Liu, Intuitive dependent? investigating Ilms' behavior style to conflicting prompts. In ACL, 2024.
[77] O. Yoran, T. Wolfson, B. Bogin, U. Katz, D. Deutch, and J. Berant. Answering questions by meta-reasoning over multiple chains thought. In EMNLP, 2023.
[78] Yu, X. Wang, Tu, Cao. D. Zhang- Li, Lv, H. Peng. Z. Yao, X. Zhang, H. Li, al. Kola: Carefully benchmarking world knowledge of large language models. In ICLR, 2024.
[79] Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, et al. Siren's 's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
[80] Y. Zhao, A. Devoto, G. Hong, X. Du, A. P. Gema, H. Wang, X. He, K.-F. Wong, and P. Minervini. Steering knowledge selection behaviours in Ilms via sae-based representation engineering. In NAACL, 2025.
[81] C. Zhou, Dong, X. Huang, Liu, K. Zhou, and Z. Xu. Quest: Efficient extreme multi-label text classification with large language models on commodity hardware. In EMNLP Findings, 2024.
[82] Y. Zhu, Xiao, Wang, and J. Sang. Kg-fpq: Evaluating factuality hallucination in lims with knowledge graph-based false premise questions. In COLING, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We discuss our contributions in the introduction and support the claims with substantial empirical results in Sections 5 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our experiment setup in Section d and Appendix A We also provide the code and data used in our experiments in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We provide the data and code in supplementary material, with detailed instruc- tions to reproduce the main experiment results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe our experiment setup in Section and Appendix ^
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The KScope framework proposed in this paper involves a series of statistical tests, and all the experiment results are based these statistical tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss the computational resources Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Section g
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the creators of the datasets and models used in our work. We provide the licenses and copyright information in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: provide detailed documentation for the Hemone dataset in the supplemen- tary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our experiments leverage LLMs for augmenting the evaluation datasets. We declare their usage in Section 4 and Appendix A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy poctps://neurs: cc/Conferencess//0000 for what should or should be described.
22

---

A Additional Details on the Experiment Setup
In Section 4 we describe the experiment setup, where apply KScope nine LLMs across four datasets. Figure 20 shows the few-shot prompt used with GPT-4o [24 to generate two additional incorrect options of the same type as the correct one for each question in NQ [32 and HotpotQA [74].
Figure shows the instructions used to sample model responses with and without context, which are then used by KScope to characterize knowledge statuses.
Among the three existing datasets, PubMedQA [26] uses the MIT license, while NQ and HotpotQA
released under the Apache 2.0 license. All three evaluated LLM families (Gemma [55] Llama [15],
and Qwen (72)) are distributed under custom commercial licenses. We run all the experiments in this paper on four NVIDIA A100 GPUs.
To determine the number of question paraphrases (M) and sample responses (N) needed for consistent characterization of LLM knowledge status, we conduct hyperparameter search Hemonc [64] using Llama-8B, -B.. As shown in Figure TO the percentage of status changes stabilizes after collecting N 100 model responses using M 20 paraphrases per question. We adopt this configuration for KScope throughout the paper.
B Additional Results on the Context-Induced Shifts in Knowledge Status
In Section we investigate how context updates LLMs' knowledge status. Figures 14
15
16 and
provide detailed breakdowns of how gold context induces shifts from each parametric knowledge status to contextual knowledge status for each LLM on each dataset in the multi-choice setting. Figure 18 shows the shifts induced by noisy context on HotpotQA in the multi-choice setting, while Figure illustrates the shifts induced by gold context on HotpotQA in the open-ended setting.
(a) Effect of increasing the Number of Question Paraphrases When Sampinng 100 Model Responses
(b) Effect of increasing the Namber of Model Responses
When Using 20 Question Paraphrases
100
100
Coange PSI in knowledge
Status Chucachertaalaon
Contestual
Parmmstrcc
Change PSS in Knowledgs Status Choractartaalan
80
Contextual
Parametzic
60
40
40
Number Question Paragimmasss (M4)
Numberr Modal Responsens (N)
Figure 10: Hyperparameter search on Hemone using Llama-8B with gold context in the multi-choice setting. KScope achieves stable characterization of LLM knowledge status with M 20 question paraphrases (left) and = 100 model responses per question (right). We adopt this configuration in all experiments.
23

---

C Additional Results of the Feature Importance Analysis
To identify the context features driving successful knowledge updates, we apply logistic regression to the features extracted in Section [6] We use L.2 regularization to mitigate multicollinearity and normalize all numerical features before model fitting. For each stratified combination of dataset, LLM, and initial parametric knowledge status, we perform five-fold cross-validation to tune class weights and regularization strength. We exclude combinations with fewer than 50 examples or 10 instances per class. Due to extreme class imbalance in some settings (e.g., nearly 99% positive labels for Gemma-9B with consistent correct status on HotpotQA), logistic regression does not always outperform dummy classifier in Macro-F1, which simply predicts the majority class. We retain only regression models that outperform this baseline for the feature importance analysis in Section Figure 1 shows the change in Macro-Fl relative to the dummy classifier.
Consistent Correct
Conficting Correct
Absent
Conflicting Wrong
Consisteat Wrong
Gemma 28
116.2
14
4424
$12.7
116.
614.
30.
1104
23.3
+0.0
+25.
+32.
+14.5
+19.7
10.8
130/
-00
Gemma
Cemma-272
415.7
138.5
435.3
435.5
+3.3
416.1
30
Liama 58
$35.9
45.0
4.4
67.7
45.5
413.0
65.9
631.7
412.
65.5
+101
+34.
+12A
-36.
+16.9
637.6
67.1
113.0
Liama 68
136.7
42.6
45.6
42.0
433.3
418.2
+00
+0.0
-20.8
+12.3
636.7
46.3
421.5
20
-10
Owen-38
+5.0
437.5
+14.3
422.
+20.0
+15.
41
+3.3
Qwen-7BB 413.4
-23
+0.0
+4.5
420.5
+37.6
+1.
+3.4
+21.5
422.3
28
200
+5.3
20.7
-122
+7.1
+11.1
+0.3
+11.6
416.
Qwez-14B
436.5
-22
+0.0
+36.7
413.9
+1.7
+8.
+37.5
+7.6
-20.0
+392
+11.0
+12.3
+220
Heigugg
Halpogg
PalMedUA
HagaoQA
PalMedgg
Figure 11: Change in Macro-F1 of logistic regression models relative to a dummy classifier that
predicts the majority class. "x" marks cases with fewer than 50 examples or 10 instances per class, which are excluded from regression analysis.
24

---

D Additional Results on the Context Augmentation Strategies
In Section we experiment with various context augmentation strategies. Figure 22 illustrates how we insert metadata to enhance context credibility. Figure 23 shows how we prompt GPT-4o to perform naïve and constrained context summarization.
We present the normalized feature space for each dataset in Figure 12 highlighting the differences between naïve and constrained summarization. We also show how each augmentation strategy impacts the success rate of knowledge updates for each dataset in Figure 13
(e) Hemone
10+
Costertt
20
2
ROUGE
Context
Origtnal
Summartzative
Constrained Summartcctinn
(b) PubMedOA
3+
Contect Length
Reedebiity
Unoy
Sim
ROURS-22
BOUGGE-2
Chatest Perpieccty
Entrage
Orginal
-02
Content Zength
Beadshillty
Unig Tiass
Enb Sim
R
ROIME-2 R
RO
Contest Pergienity
Content Satrrngy
Origital
Namn Summrattttionn
Condrained Summartcctinn
(d) HulpotQA
Readablltty
Perpiectyy
Cistropy
Figure 12: Normalized feature space per dataset, comparing naive and constrained summarization.
25

---

(a) Hlemnne
Consistent Correct
Coaflicting Correct
Absent
Conflising Wrong
Consistent Wrong
Change in Suctweee Bate 180 of Knoowl degdge Update Lt)
-10
Liama 433
Ooven -143
Girl -40
Liema -B8
Owen 144B
GPT 40
Liama
GPT -40
Lihma .83
Owwen -14B
Liama -BB
8B
.14
-4
-14B
-4
Uredibiity
Credibility
(b) PubMedOA Absont
Consistant Correct
Conflicting Correct
Conflicting Wrong
Consistent Wrong
Change in Socratee Rate 19 ot
16
Knowlodge Update (T)
9.0
124
Libma
Ownn
GF7
Liama
Qaean
GP?
Liama
Owen
GPT
Liama .86
Chron
GPT
Liama
Qwenn
GPT
Cre
Summartration
Constrained Summartzation
Credibility Constratzed Summartzation
NQ Absent
Consistent Correct
Conflicting Correct
Conflistnn Wrong
Consistent Wrong
Change Surnowe Batl 19a of
kcowledge Update (t)
12
H12
M0
-10-
-20
Llama
GPT
GFT
83
143
4o
8B
14B
40
56
143
-00
83
-14B
40
8B
.14B
-6
Emydibltty
Credibility
(4) HotpotQA
Cozsistent Correct
Conflicting Correct
Absent
Conflitingg Wrong
Consisteat Wrong
Chonge in Sperneee Bata (No of
Kiewlodge Upaate LT)
+0
voe
Uibma
Owen
GP7
Liama
Useo
GP7
Liama
Owen
GPT
Liama
Quernn
GPT
Liama
Qwnn
GFT
Uredibiitt
Summararation
Construined Summartzation
Credibrlity
Censtraroed Summartzatton
Figure 13: Effect of different context augmentation strategies the success rate of knowledge updates for each dataset, relative to the original context.
26

---

Context-Induced Shitts (%) in LLM Knowledge Statuses on Hemonc (Multi Choice Setting, Gold Context)
Gemma-2B
Gemma-9B
Gemma-27B
100
Parametric Knowlodge Status
Consistent Corrent
89.3
0.0
0.0
6.6
76.4
7.0
1.1
0.2
15.3
75.4
9.0
0.3
14.2
Conflicting
60
Absent
33.9
4.3
1.1
5.0
55.7
44.44
17:2
2.2
2.8
33.3
42.3
17.0
2.3
0.8
37.5
40
Conflicting Wrong
3.0
2.2
0.9
8.2
85.
25.6
24.1
3.4
2.7
44.1
17.4
2.3.4
3.4
2.9
52.9
Consistant Wrong
31.3
23.5
Llama-3B
Llama-8B
Llama-70B
100
Parametric Knowledge Status
Consistent Correct
65.1
13.8
5.3
1.1
14.8
73.6
15.0
1.8
9.2
87.1
6.0
0.3
0.2
6.4
Conflicting
60
Absent
31.1
15.6
18.2
2.3
32.9
34.2
21.1
5.3
3.1
36.3
54.7
22.1
0.0
2.1
21.1
40
Conflicting Wrong
39.0
13.0
17.1
4.9
26.0
31.2
12.8
5.0
5.0
46.1
45.0
233.1
0.0
2.5
29.4
Consistent Wrong
27.4
14.3
2.3
34.3
28.5
Qwen-31
Qwen-7B 7B
Owen-14B
100
Parametric Knowllddge ledge Status
Consistent Correct
84.1
6.6
1.7
0.6
7.0
82.3
5.9
0.5
0.5
10.9
77.7
10.6
1.0
0.2
10.4
Conflicting Correct
48.4
60
Absent
29.9
29.0
7.6
4.6
28.8
66.1
13.4
0.8
0.8
18.9
43.1
26.2
2.4
2.8
25.5
40
Conflicting Wrong
10.5
20.3
5.7
3.1
60.4
25.3
13.0
1.9
4.5
55.2
34.9
21.22
1.4
4.0
38.5
Consistent
Wrong
16.7
4.8
23.9
1.8
46.7
17.8
3.4
32.6
Consistent
Correct
Confacting
Absent
Conflicting
Wrond
Consistent
Wrong
Consistent
Confiiciing
Correct
Abseat
Conflicting
Consistent
Wrond
Conelstent.
Correct
Conflicting
Correct
Absent
Consistant
Wrond
Contextual Knowledge
Contextual Knowledge Status
Contextual Knowledge Status
Figure 14: Shifts in knowledge status induced by gold context for each LLM on the Hemone dataset in the multi-choie setting.
27

---

Context-Induced Shifts (%) in LLM Knowledge Statuses on PubMedQA (Multi Choice Setting, Gold Context)
Gemma-2B
Gemma-9B
Gemma-27B
100
Parametric Knowlodge Status
Consistent Corrent
78.9
0.9
1.8
12.7
78.7
5.5
0.9
0.3
14.6
82.3
4.1
0.3
12.3
Conflicting
16.8
60
Absent
52.0
8.0
4.0
0.0
36.0
57.9
10.5
0.0
0.0
31.6
66.7
8.3
0.0
0.0
25.0
40
Conflicting Wrong
63.5
11.5
2.1
4.2
18.7
54.8
6.8
1.4
5.5
31.5
689.8
3.8
0.0
1.9
24.5
Consistant Wrong
59.6
0.6
Llama-3B
Llama-8B
Llama-70B
100
Parametric Knowledge Status
Consistent Correct
91.4
0.0
0.0
4.1
93.2
4.0
0.0
2.8
95.8
1.5
0.0
2.8
Conflicting
4.7
60
Absent
100.0
0.0
0.0
0.0
100.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
40
Conflicting Wrong
0.0
0.0
0.0
25.0
75.0
0.0
0.0
0.0
11.1
88.9
0.0
0.0
0.0
0.0
100.0
Consistent Wrong
54.2
3.1
5.2
0.3
39.5
Qwen-31
Qwen-7B
Qwen-1 14B
100
Parametric Enowsledge Status
Consistent Correct
62.2
4.0
32.7
0.2
0.7
89.5
3.5
1.0
0.0
6.0
88..7
2.8
0.8
0.6
7.0
Conflicting Correct
60
Absent
48.5
7.6
31.8
3.0
9.1
63.5
11.5
1.9
21.2
71.4
3.6
0.0
0.0
25.0
40
Conflicting Wrong
16.7
10.0
33.3
3.3
36.7
63.0
7.4
1.9
0.0
27.8
76.2
0.0
0.0
1.6
22.2
Consistent
Wrong
17.6
44.7
3.8
5.9
1.8
6D.0
6.2
30.5
Consistent
Correct
Conficting
Correct
Absent
Conflicting
Wrond
Consistent
Wrong
Consistent
Confiicingg
Correct
Abseat
Confliciing
Consietent
Wrong
Comelstent.
Correct
Conflicting
Correct
Absent
Confictinn
Wmong
Consistent
Wrond
Contextual Knowledge
Contextual Knowledge Status
Contextual Knowledge Status
Figure 15: Shifts in knowledge status induced by gold context for each LLM on the PubMed dataset in the multi-choice setting.
28

---

Context-Induced Shifts (%) in LLM Knowledge Statuses on NQ (Multi Choice Setting, Gold Context)
Gemma-2B
Gemma-9B
Gemma-27B
100
Parametric Knowlodge Status
Consistent Corrent
74.4
20.3
0.0
76.3
1.1
20.4
0.1
2.1
77.1
0.7
19.9
0.0
2.2
Conflicting
60
Absent
53.8
4.0
29.6
0.2
12.3
57.0
1.5
35.7
0.0
5.8
50.3
1.1
41.2
0.2
7.1
40
Conflicting Wrong
53.7
0.0
22.3
0.8
23.1
58.3
1.7
20.0
0.0
20.0
52.2
4.3
2.3.9
4.3
15.2
Consistant Wrong
56.9
2.5
22.9
Llama-3B
Llama-81
Llama-70B
100
Parametric Knowledge Status
Consistent Correct
91.3
3.1
0.0
93.5
2.6
2.7
0.1
1.2
91.9
4.6
0.2
0.2
3.1
Conflicting
60
Absent
64.3
18.1
0.7
9.5
60.3
5.0
29.2
0.0
5.5
68.4
10.5
0.0
2.6
18.4
40
Conflicting Wrong
50.6
7.9
15.2
6.2
20.2
35.7
11.2
28.6
6.1
18.4
39.0
15.6
1.3
10.4
33.8
Consistent Wrong
54.1
15.3
36.2
Qwen-31
Qwen-7B
Qwen-145 14B
100
Parametric Knowllddg ledgl Status
Consistent Correct
86.7
6.0
2.2
0.4
4.8
92.5
3.1
0.8
0.1
3.4
95.1
2.1
0.8
0.1
2.1
Conflicting Correct
60
Absent
70.0
6.9
6.2
1.9
15.0
64.2
7.3
6.5
2.4
19.5
70.8
9.7
8.3
0.0
11.1
40
Conflicting Wrong
61.2
9.2
2.6
6.1
20.9
61.8
6.4
5.7
5.1
21.0
58.99
2.7
3.6
4.5
30.4
Consistent Wrong
60.4
3.3
3.8
2.9
24.7
56.8
6.7
2.5
31.0
Consistell
Correct
Conficting
Absent
Confliciing
Wrond
Consistent
Wrong
Consistent
Confiictngg
Correct
Abseat
Confliciing
Consietent
Wyong
Comelstent
Correct
Conflicting
Correct
Absent
Consistent
Wrond
Contie
Contextual Knowledge
Contextual Knowledge Status
Contextual Knowledge Status
Figure 16: Shifts in knowledge status induced by gold context for each LLM on the NQ dataset in the multi-choice setting.
29

---

Context-Induced Shifts (%) in LLM Knowledge Statuses on HotpotQA (Multi Choice Setting, Gold Context)
Gemma-2B
Gemma-9B
Gemma-27B
100
Parametric Knowlodge Status
Consistent Corrent
95.6
0.2
0.0
2.5
98.3
0.5
0.4
0.8
98.6
0.5
0.2
0.0
0.7
Conflicting
60
Absent
91.7
2.7
0.7
0.0
4.9
95.9
1.0
1.5
0.1
1.5
96.7
1.2
0.5
0.0
1.6
40
Conflicting Wrong
87.1
4.0
0.0
0.0
8.9
89.3
0.8
0.8
0.8
8.4
92.33
0.0
1.3
0.0
6.4
Consistant Wrong
Llama-3B
Llama-81
Llama-70B
100
Parametric Knowledge Status
Consistent Correct
96.9
0.1
0.0
98.2
1.1
0.2
0.5
96.4
1.2
0.0
0.3
Conflicting
1.1
60
Absent
94.1
2.7
0.8
0.0
2.5
96.6
1.5
0.6
0.0
1.3
90.4
5.3
0.0
0.0
4.3
40
Conflicting Wrong
89.0
4.8
2.4
0.3
3.4
91.1
2.7
1.4
0.7
4.1
81 .8
6.6
0.0
1.5
10.2
Consistent: Wrong
#7.5
0.8
11.0
10.2
Qwen-31
Qwen-7B
Qwen-14B 14B
100
Parametric Anowwledge Status
Consistent Correct
94.8
3.6
0.3
0.0
1.3
98.1
0.9
0.1
0.0
0.9
98.8
0.6
0.1
0.0
0.6
Conflicting Correct
1.4
60
Absent
89.4
1.5
0.0
96.2
1.9
0.8
0.0
1.1
96.0
0.8
0.0
0.0
1.2
40
Conflicting Wrong
86.8
6.3
3.0
0.3
3.6
92.7
1.5
0.8
0.4
4.6
93.00
1.7
0.0
0.6
4.7
Consistent
Wrong
79.0
0.5
0.3
0.1
2.9
6.1
Consistent
Correct
Confacting
Absent
Conflicting
Wrond
Consistent
Consistent
Confiicingg
Correct
Abseat
Confliciing
Consistent
Wrong
Comelstent
Correct
Conflicting
Correct
Absent
Confiction
Nmond
Consistent
Wrond
Contextual Knowledge Status
Contextual Knowledge Status
Contextual Knowledge Status
Figure 17: Shifts in knowledge status induced by gold context for each LLM on the HotpotQA dataset
in the multi multi-choice setting.
30

---

Context-Induced Shifts (%) in LLM Knowledge Statuses on HotpotQA (Multi Choice Setting, Noisy Context)
Gemma-2B
Gemma-9B
Gemma-27B
100
Parametric Knowlodge Status
Consistent Corrent
82.4
2.7
0.4
10.4
80.1
2.3
13.0
0.1
4.4
82.1
1.9
11.9
0.2
3.9
Conflicting
60
Absent
44.3
7.6
17.9
1.5
28.8
44.3
2.1
47.0
0.5
6.1
45.0
2.7
44.3
0.2
7.8
40
Conflicting Wrong
44.2
4.9
6.2
4.0
40.6
51.9
2.3
2.5.2
1.5
19.1
53.8
2.6
26.9
1.3
15.4
Consistant Wrong
5.5
4.4
Llama-3B
Llama-81
Llama-70B
100
Parametric Knowledge Status
Consistent Correct
76.1
11.6
0.1
82.8
3.9
10.6
0.3
2.4
88.3
5.2
0.1
0.3
6.0
Conflicting
60
Absent
45.0
7.7
37.6
1.4
8.3
45.7
4.1
45.2
0.7
4.4
56.4
16.0
4.3
2.1
21.3
40
Conflicting Wrong
43.8
6.66
30.3
3.1
16.2
49.3
4.8
32..9
1.4
11.6
40.1
10.2
1.5
13.1
35.0
Consistent Wrong
17..4
33.8
Qwen-31
Qwen-7B 7B
Qwen-1 14B
100
Parametric Knowlledg ledgl Status
Consistent Correct
74.6
12.6
4.9
0.9
7.1
81.9
6.8
2.4
0,9
8.1
85.6
4.5
3.2
0.3
6.4
Conflicting Correct
13.9
60
Absent
44.7
11.8
21.4
3.9
18.2
51.1
10.2
8.7
4.2
25.8
60.8
7.3
14.3
3.7
13.9
40
Conflicting Wrong
45.9
8.1
12.3
16.8
16.8
46.0
8.4
7.7
7.3
30.7
50.6
6.4
12.2
6.4
24.4
Consistent
Wrong
41.1
16.8
10.6
5.9
25.6
10.4
6.7
25.7
48.3
9.8
30.7
Consistell
Correct
Conficting
Absent
Confliciing
Wrond
Consistent
Wrong
Consistent
Confiicting
Correct
Abseat
Confliciing
Consietent
Wyong
Conelstent.
Correct
Conflicting
Correct
Absent
Confictinn
Wmong
Consistent
Wrond
Contextual Knowledge
Contextual Knowledge Status
Contextual Knowledge Status
Figure 18: Shifts in knowledge status induced by noisy context for each LLM on the HotpotQA dataset the multi-choice setting.
31

---

Context-Induced Shifts (%) in LLM Knowledge Statuses on HotpotQA (Open (Open-Ended Ended Setting, Gold Context)
Gemma-2B
Gemma-9B
Gemma-27B
100
Parametric Knowlodge Status
Consistent Corrent
90.4
3.3
0.0
94.0
1.5
2.7
1.8
94.1
1.5
2.6
0.1
1.7
Conflicting
60
Absent
96.3
5.6
5.1
0.4
2.6
87.9
2.0
5.8
0.2
4.1
86.8
2.1
6.2
0.1
4.8
40
Conflicting Wrong
72.1
9.8
4.9
4.9
8.2
61.2
12.2
2.0
2.0
22.4
77.6
5.2
5.2
1.7
10.3
Consistant Wrong
1.6
5.3
28.8
Llama-3B
Llama-81
Llama-70B
100
Parametric Knowledge Status
Consistent Correct
87.4
2.2
0.1
92.9
4.0
2.0
0.1
1.0
95.9
1.1
0.0
1.3
Conflicting
4.4
60
Absent
80.3
10.8
6.9
0.5
61.6
4.6
32.0
0.4
1.4
63.8
1.1
32.2
0.0
2.9
40
Conflicting Wrong
69.0
14.8
2.1
4.2
9.9
75.5
8.0
1.2
3.1
12.3
83.2
2.5
5.0
1.2
8.1
Consistent Wrong
65.2
4.3
1.8
28.1
Qwen-31
Qwen-7B
Qwen-146 14B
100
Parametric Knowllddge ledge Status
Consistent Correct
87.7
8.3
3.0
0.0
0.9
92.2
3.3
2.8
0.1
1.6
94.4
2.5
2.4
0.0
0.6
Conflicting Correct
60
Absent
81.1
9.9
7.9
0.0
79.4
5.4
12.8
0.2
2.3
86.0
2.8
9.3
0.0
1.9
40
Conflicting Wrong
79.1
10.7
3.3
0.5
6.5
84.3
2.2
3.4
1.5
8.6
86.0
5.8
1.7
0.6
5.8
Consistent
Wrong
64.2
12.8
3.2
2.8
2.8
17.4
74.2
6.3
14.5
Consistent
Correct
Conficting
Absent
Confliciing
Wrond
Consistent
Wrong
Consistent
Conflicting
Correct
Abseat
Confliciing
Consietent
Woong
Comelstent
Correct
Conflicting
Correct
Absent
Confiction
Consistent
Wrond
Contextual Knowledge Status
Contextual Knowledge Status
Contextual Knowledge Status
Figure 19: Shifts in knowledge status induced by gold context for each LLM on the HotpotQA dataset in the open-ended setting.
32

---

###l Instruction
Using the provided examples as guide, transform the given question with correct answer into mullipe--hhoiee
question.
Provide two additional incorrect options that are similar in type category to the correct answer.
Example
## Question: Which continent the largest by land area?
Correct Answer:
## Incorrect Option Africa
## Incorrect Option Europe
Exumple
## Question: the last name scott irish scottish?
## Correct Answer: Scottish
## Incorrect Option Irish
## Incorrect Option 2: English
Example 3
## Question: Were Scott Derrickson and Ed Wood of the same nationality?
## Correct Answer: Yes
W! Incorrect Option I: No
## Incorrect Option 2: Maybe
Your Task
## Question: [QUESTION)
#N Correct Answer: (ANSWER)
Figure 20: Few-shot prompt used with GPT-4o to generate incorrect options.
33

---

### Instruction
Without relying any external context, select the most appropriate answer from the options provided.
First, explain your reasoning briefly slep-by-step based the provided infonnation. Then, select the uppropriate option and present your response in the required format.
### Question:
!QUESTION;
Choices:
Option I: [OPTION 1;
Option 2: (OPTION 2]
Option 3: (OPTION 3)
Provide your response in the following format: <unswer>Option [oumbrr]</answer
### Instruction
You given some context und multiple-choice question.
Based the context. select the appropriate from the options provided.
First, explain your reasoning briefly step-by-step based the provided infonmation.
Then, select the most appropriate option and present your response required format.
### Context:
(CONTEXT)
#*# Question: (QUESTION)
### Choices:
Option I: (OPTION 1:
Option 2: (OPTION 2)
Option 3: (OPTION_3)
Provide your response in the following format: <answer-Option /answer
Figure 21: Instruction prompt for sampling model responses without (top) and with (bottom) context.
The following context comes from credible sources such as peer-reviewed PubMed articles: *** [ARTICLL TITLE) * [JOURNAT. TITLE)* published on IPULIICATION DATE).
Please prioritize the the following context over your internal memory, reflects curated, factual, and up-to- date information.
(CONTEXT)
The following context comes from credible sources such as verified Wikipedia pages: (WIKIPEDIA TITLE)* from Wikipedia.
Please prioritize the use of the following context over your own internal memory, as reflects curated, factual, and up-to- date information.
(CONTEXT)
Figure 22: Context augmentation prompts for inserting metadata in healthcare- datasets (top) and general-domain domain datasets (bottom).
34

---

Instruction
Summarize the given context.
Do not repeat given context any headings like "WKM Summarized Context" in your output. Only return the revised summary.
Imput
#** Given Context: (CONTEXT)
#M Your Task
### Summarized Context:
Instruction
Summarize the given context by reducing its overall length (i.c., number of tokens), while strictly preserving all information conveyed original.
Your goal make the text more concise, not to omit or alter any factual content.
Follow these constraints:
Preserve semantic content from given context. Every fact, detail, and piece of information mentioned must romain present in the summary. Nothing should be lost or distorted.
2. Maintain the naturalness and fluency of the text. The summarized context should have similur perplexity to the original,
measured by standard language model.
3. Ensure exact token-level overlap with the given question by retaining all of its content words (excluding stop words) exactly they appear in your summary.
Use strategies like concise rewording, combining redundant phrases, and removing non-essential claboration, without compromising the informativeness, clarity, completeness of the given context. Do repeat the given context, the given question, any headings like "sii Summarized Context" your output. Only return the revised summary.
## Input
#*# Given Question: (QUESTION)
Given Context: (CONTEXT)
## Your Task
HHH Summarized Context:
Figure 23: Instruction prompt used with GPT-4o to generate naïve (top) and constrained (bottom) context summarization.
35


Reviews:
[{"review_id": "iLOasIQt1K", "reviewer": "Reviewer_fYMy", "summary": "The paper studies how to deeply characterize an LLM's knowledge state regarding a specific problem. The paper proposes a novel taxonomy of knowledge status, which categorizes the knowledge state of LLMs into five distinct classifications based on the dimensions of \"Consistency\" and \"Correctness\". The paper then introduces KScope,  a framework that systematically and reproducibly classifies the\nmodel's knowledge state into one of the aforementioned categories through a series of statistical tests, including binomial tests, multinomial tests, and likelihood ratio tests.", "questions": "1. The test result of a model's knowledge status is based on the model's behavior within the support set. Since the support set is\nhuman-made or model-generated, will the result be influenced by the support set? For example, have you analyzed the\nimpact of \"difficult\" negative samples and \"simple\" negative samples on the classification results of knowledge states?\nBesides, if the question's support set is (A.dog B.cat) and the model's internal knowledge is conflicting which is (cat,\nduck), since the model is forced to choose one among the support set, it will prefer 'B.cat' and will be classified as\n'consistent correct'. However, if we replace the 'A.dog' with 'A.duck', then the model will prefer both of them, and will be\nclassified as 'conflicting correct'.\n\n2. The paper defines \"knowledge absent\" as the model exhibiting a uniform distribution on the support set. While this is\nindeed an expression of \"not knowing,\" it may not be the only or most common manifestation. When lacking knowledge,\nLLMs may also persistently output rejection responses like \"I don't know\" or \"unable to answer,\" or generate\nhallucinations beyond the support set. Although the first step of KScope tests the significance of invalid answers, it seems\nto categorize these complex \"absent\" behaviors under a general \"knowledge deficiency\" state. Is it possible for the\nframework to further distinguish these different \"absent\" states, given that they may require distinct intervention\nstrategies?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed five classification methods of knowledge states go beyond the previous binary divisions of\n\"correct/incorrect\" or \"conflicting/non-conflicting\", providing a more detailed and insightful analytical perspective. I\nbelieve this classification method is an important contribution to the community.\n\n2. The design of the KScope framework is outstanding. It's based on a series of mature statistical tests, which makes its\nconclusions highly scientific and reliable. This approach of applying statistical theory to LLM behavior analysis provides\na reproducible and standardized operational process for subsequent research.\n\nWeaknesses:\n1. This method requires sampling N = 100 outputs for each question to construct a stable empirical distribution. This is\ncomputationally expensive for large models and large-scale evaluations. Although this is necessary for the validity of\nstatistical tests, it may limit the practical application of KScope in scenarios that require the analysis of massive data.\n\n2. For the NQ and HotpotQA datasets, the authors generated two additional incorrect options using GPT-4o and transformed\nthem into a three-choice task. The quality of these AI-generated \"distractions\" may significantly affect the judgment of the\nmodel's knowledge state. A very obvious incorrect option and a wrong option that is very similar to the correct answer\npose completely different challenges to the model.", "was_revised": true, "final_justification": "The rebuttal answered my question about the support set soundly and provided explanations for the relationship between refusal/hallucination and absent knowledge.", "rebuttal": "We thank the reviewer for their thoughtful comments and valuable suggestions. Please find our responses below. We will update the manuscript accordingly and would be grateful if the reviewer could consider raising the score if these clarifications address the concerns.\n\n---\n**1. Influence of Support Sets**\n> Since the support set is human-made or model-generated, will the result be influenced by the support set?  \n> The quality of these AI-generated \"distractions\" may significantly affect the judgment of the model's knowledge state.\n\n* We acknowledge that our empirical characterization of LLMs’ knowledge status in KScope is based on both the question and its associated pre-defined support set, since exhaustively enumerating all possible support sets is computationally infeasible.\n* To examine the influence of support sets, we further apply KScope to open-ended questions without providing any pre-defined options.\n    * Specifically, we generate $N = 100$ responses for each HotpotQA question using Llama-3.1-8B-Instruct, and cluster them semantically using a natural language inference model [1,2]. Based on the size of clusters and the number of invalid answers, we follow the procedure in Figure 2 to infer the model’s knowledge status.\n    * In the absence of pre-defined support sets, the model no longer receives hints from answer options. As a result, Llama-3.1-8B-Instruct consistently exhibits absent parametric knowledge in the open-ended setting without external context, unlike in the multi-choice setting in Figure 3.\n    * However, when supporting context is provided, the distribution of knowledge statuses remains largely consistent between the open-ended and multi-choice settings.\n    * | w/ context? | w/ pre-defined options? | Consistent Correct (%) |  Conflicting Correct (%) | Absent (%) | Conflicting Wrong (%) | Consistent Wrong (%) |\n|:----|:----|:----|:----|:----|:----|:----|\n| No (Parametric) | Yes (Multi-choice) | 46.15 | 16.10 | 23.22 | 2.39 | 12.14 |\n| No (Parametric) | No (Open-Ended) | 0.00 | 0.00 | 100.00 | 0.00 | 0.00 |\n| Yes (Contextual) | Yes (Multi-choice) | 95.16 | 2.16 | 0.46 | 0.05 | 2.17 |\n| Yes (Contextual) | No (Open-Ended) | 92.68 | 0.51 | 1.98 | 0.10 | 4.74 |\n\n[1] He, Pengcheng, et al. \"Deberta: Decoding-Enhanced Bert With Disentangled Attention.\" ICLR (2021).  \n[2] Reimers, Nils, et al. \"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.\" EMNLP (2019).\n\n---\n**2. Relationship between Refusal/Hallucination and Absent Knowledge**\n> The paper defines \"knowledge absent\" as the model exhibiting a uniform distribution on the support set.  \n> Although the first step of KScope tests the significance of invalid answers, it seems to categorize these complex \"absent\" behaviors under a general \"knowledge deficiency\" state.\n\n* We argue that when a model lacks sufficient knowledge to a question, it may (1) refuse to respond, (2) hallucinate invalid answers, or (3) generate valid answers at random. Hence, we group all three behaviors under the status of absent knowledge in KScope.\n* To further distinguish between refusal and hallucination, we need to split Step 1 of KScope, which tests for the significance of invalid answers, and conduct separate statistical tests for these two types of behaviors. We leave this extension to future work.\n\n---\nWe thank the reviewer again for their insightful feedback. If our clarifications address the concerns, we would be grateful if the reviewer could consider raising the score."}, {"review_id": "X4PSgsDf4U", "reviewer": "Reviewer_Fosa", "summary": "This paper introduces a framework to measure when language models give consistent and reliable answers KScope. By comparing the answers from variations of questions, they apply statistical tests to understand whether a model is confident or uncertain. They evaluate this method on four question-answering datasets, using nine different open-source language models, and provide taxonomy of knowledge status.", "questions": "Main Questions:\n- **(Q1)** Could you please add comparisons to some baselines of uncertainty metrics to show whether KScope offers a meaningful improvement over cheaper methods? How are these metrics related to KScope? \n- **(Q2)** Could you comment whether paraphrasing and CoT introduces correlated outputs, which is not expected for tests assumptions?\n- **(Q3)** Did you think of some real-world test for the method to produce improved downstream task performance? \n\nMinor Questions:\n- **(Q4)** I think I didn't understand how to use your method for open-ended generation. Could you please elaborate more on such application and maybe provide some experiments?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- **(S1)** Systematic framework: KScope is a well-structured and clearly described framework that utilise formal statistical testing and paraphrasing with CoT. This combination is novel and goes beyond traditional single-shot confidence estimation.\n- **(S2)** Extensive empirical evaluation: authors evaluate KScope across 4 datasets and 9 LLMs of varying sizes and families. This broad evaluation improves generalizability. Moreover, authors carefully design their experiments and methodology\n- **(S3)** Insightful analysis and ablations: authors provide useful analyses on error types, model scaling trends, and the effects of interventions like summaries or distractor quality\n\nWeaknesses:\n- **(W1)** No comparison with existing uncertainty/conflict measures: there is no comparison of KScope with prior simpler or cheaper methods like entropy-based confidence or others. This makes it hard to judge if the added complexity is really necessary.\n- **(W2)** Statistical concerns in sampling: the framework assumes that samples (paraphrases and CoTs) are independent for hypothesis testing, but in practice, as I believe, they are strongly correlated (e.g., same base question, shared decoding artifacts). This may lead to overly optimistic significance results\n- **(W3)** No real-world use case evaluation: the paper shows that KScope can identify unreliable answers, but it does not demonstrate this resulting in improved downstream task performance", "was_revised": true, "final_justification": "The authors have addressed all major concerns raised in the initial review. Mainly, they provide a comparison to semantic entropy baselines, demonstrating stronger alignment with human judgment. Their explanation of sampling independence is reasonable. The method is well-motivated and supported by strong empirical results. \n\nI find the overall contribution strong, and vote for acceptance.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. Please find our point-by-point responses below. We will incorporate these clarifications and results into the revision, and would be grateful if the reviewer could consider raising the score if the concerns are addressed.\n\n---\n**1. Comparison Against Existing Uncertainty Measures**\n> No comparison of KScope with prior simpler or cheaper methods like entropy-based confidence or others.\n\n* To assess the effectiveness of KScope relative to semantic entropy [1, 2], we evaluate how well each method aligns with human judgment on the degree of knowledge conflict in LLMs.\n    * We collect 100 responses from Llama-3.1-8B-Instruct for each of 200 questions randomly sampled from HotpotQA. Based on the distribution of responses (i.e. the frequency of each option and invalid answers), three annotators label each question as “consistent”, “conflicting”, or “absent”. The inter-annotator agreement is strong (Fleiss' $\\kappa = 0.61$), and we use the majority vote as the ground-truth label.\n    * We split the questions into equal-sized training and test sets. For the entropy baseline, we compute semantic entropy per question using the 100 responses and train a logistic regression model using the training set. The model then predicts conflict status on the test set.\n    * In contrast, KScope requires no training or threshold tuning. To align the comparison with the task of measuring knowledge conflict (rather than correctness), we collapse KScope’s “consistent correct” and “consistent wrong” into “consistent,” and similarly collapse “conflicting correct” and “conflicting wrong” into “conflicting.”\n    * As shown below, KScope achieves stronger agreement with human judgment than semantic entropy.\n    * | Method | Accuracy | \n|:----|:----|\n| KScope | 0.890 |\n| Semantic Entropy | 0.720 |\n\n[1] Kuhn, Lorenz, et al. \"Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation.\" ICLR (2023).  \n[2] Marjanovic, Sara, et al. \"DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models.\" EMNLP Findings (2024).\n\n---\n**2. Real-World Use Cases**\n> Did you think of some real-world test for the method to produce improved downstream task performance?\n\n* Compared to entropy-based metrics [1, 2], KScope offers an interpretable characterization of LLM knowledge status without the need for threshold calibration. By producing a distribution over knowledge statuses (e.g., as shown in Figure 3), practitioners can better assess model reliability and adapt usage accordingly in high-stakes or domain-specific settings.\n* Additionally, Sections 6 and 7 show that KScope’s outputs can guide context augmentation, significantly improving the success rate of knowledge updates across different LLMs.\n\n---\n**3. Generalization to Open-Ended Tasks**\n> Elaborate more on open-ended generation and maybe provide some experiments.\n\n* To demonstrate KScope’s generalizability to open-ended questions, we apply it to HotpotQA without supplying predefined answer options.\n    * We generate $N = 100$ responses per question using Llama-3.1-8B-Instruct, cluster them semantically using a natural language inference model [3, 4], and follow the procedure in Figure 2 to determine the LLM’s knowledge status.\n    * Without predefined options, the model lacks informational hints, and thus consistently exhibits absent parametric knowledge in the open-ended setting without external context, unlike in the multi-choice setting in Figure 3.\n    * However, once supporting context is provided, the resulting knowledge status distribution closely matches that of the multi-choice setting.\n    * | w/ context? | w/ pre-defined options? | Consistent Correct (%) |  Conflicting Correct (%) | Absent (%) | Conflicting Wrong (%) | Consistent Wrong (%) |\n|:----|:----|:----|:----|:----|:----|:----|\n| No (Parametric) | Yes (Multi-choice) | 46.15 | 16.10 | 23.22 | 2.39 | 12.14 |\n| No (Parametric) | No (Open-Ended) | 0.00 | 0.00 | 100.00 | 0.00 | 0.00 |\n| Yes (Contextual) | Yes (Multi-choice) | 95.16 | 2.16 | 0.46 | 0.05 | 2.17 |\n| Yes (Contextual) | No (Open-Ended) | 92.68 | 0.51 | 1.98 | 0.10 | 4.74 |\n\n[3] He, Pengcheng, et al. \"Deberta: Decoding-Enhanced Bert With Disentangled Attention.\" ICLR (2021).  \n[4] Reimers, Nils, et al. \"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.\" EMNLP (2019).\n\n---\n**4. Statistical Independence in Sampling**\n> Could you comment whether paraphrasing and CoT introduces correlated outputs?\n\n* We thank the reviewer for raising this point. Since we perform paraphrasing independently for each variation, and similarly generate CoT inferences independently, we assume that the resulting samples are sufficiently diverse and statistically independent for the purpose of characterizing knowledge status.\n\n---\nWe thank the reviewer again for their thoughtful feedback. We hope these additional results and clarifications address the raised concerns and further support the significance and applicability of KScope."}, {"review_id": "V0onCU7rgs", "reviewer": "Reviewer_WvzF", "summary": "The paper introduces KScope, a hierarchical testing framework designed to characterize the knowledge status of large language models (LLMs) based on their consistency and correctness across parametric (internal) and contextual (external) knowledge modes. It proposes a taxonomy of five knowledge statuses: consistent correct, conflicting correct, absent, conflicting wrong, and consistent wrong. The framework is applied to nine LLMs across four datasets (two healthcare-related: Hemone, PubMedQA; two general: NQ, HotpotQA) to evaluate knowledge status and the impact of context on knowledge updates. Key findings include the role of context features (difficulty, relevance, familiarity) in driving successful knowledge updates and the effectiveness of constrained context summarization combined with credibility metadata.", "questions": "no", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. Novel Taxonomy and Framework: The introduction of a five-status taxonomy based on consistency and correctness is a significant contribution. KScope’s hierarchical testing approach (using statistical tests like binomial and multinomial tests) provides a structured method to assess LLM knowledge, addressing limitations of prior methods like entropy-based uncertainty metrics that fail to capture mode structure (e.g., distinguishing [0.45, 0.45, 0.1] from [0.6, 0.2, 0.2]).\n\n2. Comprehensive Evaluation: The paper evaluates nine LLMs from three model families (Gemma-2, Llama-3, Qwen-2.5) across diverse datasets, ensuring robust empirical validation. The inclusion of both healthcare and general-domain datasets enhances generalizability.\n\n\n**Weakness**\n\n1. Generalization to Open-Ended Tasks: Although KScope is claimed to be applicable to open-ended questions (Section 6), the evaluation focuses on three-option classification tasks. Empirical validation on open-ended tasks would strengthen the claim of generalizability.", "was_revised": true, "final_justification": "Generalization to Open-Ended Tasks address my concerns.\n\nI would keep the positive rating.", "rebuttal": "We thank the reviewer for their helpful feedback. Please find our response below. We will update the manuscript accordingly and would appreciate it if the reviewer could consider raising the score if the additional results address the concern.\n\n---\n**1. Generalization to Open-Ended Tasks**\n> Empirical validation on open-ended tasks would strengthen the claim of generalizability.\n\n* To demonstrate the generalizability of KScope to open-ended questions, we apply it to characterize the knowledge status of Llama-3.1-8B-Instruct on questions from HotpotQA, this time without providing any pre-defined answer options.\n    * Specifically, we generate $N = 100$ responses per question and semantically cluster them using a natural language inference model [1,2]. Based on the size of clusters and the number of invalid answers, we follow the procedure in Figure 2 to infer the LLM’s knowledge status.\n    * Without access to pre-defined options, the model no longer receives hints from answer options. As a result, Llama-3.1-8B-Instruct consistently exhibits absent parametric knowledge in the open-ended setting without external context, unlike in the multi-choice setting in Figure 3.\n    * However, when supporting context is provided, the distribution of knowledge statuses remains largely consistent between the open-ended and multi-choice settings.\n    * | w/ context? | w/ pre-defined options? | Consistent Correct (%) |  Conflicting Correct (%) | Absent (%) | Conflicting Wrong (%) | Consistent Wrong (%) |\n|:----|:----|:----|:----|:----|:----|:----|\n| No (Parametric) | Yes (Multi-choice) | 46.15 | 16.10 | 23.22 | 2.39 | 12.14 |\n| No (Parametric) | No (Open-Ended) | 0.00 | 0.00 | 100.00 | 0.00 | 0.00 |\n| Yes (Contextual) | Yes (Multi-choice) | 95.16 | 2.16 | 0.46 | 0.05 | 2.17 |\n| Yes (Contextual) | No (Open-Ended) | 92.68 | 0.51 | 1.98 | 0.10 | 4.74 |\n\n[1] He, Pengcheng, et al. \"Deberta: Decoding-Enhanced Bert With Disentangled Attention.\" ICLR (2021).  \n[2] Reimers, Nils, et al. \"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.\" EMNLP (2019).\n\n---\nWe thank the reviewer again for the constructive suggestion. We hope the added open-ended results address the concern and further support the generalizability of our approach."}, {"review_id": "EeYSlPIhjj", "reviewer": "Reviewer_gP5r", "summary": "This paper proposes KScope, a hierarchical statistical framework for classifying the knowledge status of LLMs into five categories based on correctness and consistency. It is applied to nine instruction-tuned LLMs across four datasets, enabling fine-grained analysis of knowledge behavior. The paper also explores contextual factors affecting knowledge updates and evaluates strategies like constrained summarization and credibility-based augmentation to improve LLM reliability.", "questions": "Please see the Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Introduces a new, structured perspective on LLM knowledge states.\n2. Combines statistically principled techniques (e.g., exact tests, likelihood ratio testing) with modern interpretability tools (e.g., SHAP) for credible analysis.\n3. Give insights into LLMs' knowledge mechanism and underscore the importance of tailoring knowledge update strategies to different knowledge statuses in future work.\n\n\nWeaknesses\n\n1. All context passages used in the experiments are clean and supportive, which does not reflect real-world retrieval conditions where information may be noisy (e.g., irrelevant to the query)[1, 2], conflicting (e.g., contradicting the model's parametric memory), or factually incorrect. However, the authors focus exclusively on knowledge updates under supportive contexts. While this design choice is acknowledged in Section 8 (Limitations), it would be helpful to analyze how KScope performs under such imperfect or adversarial contexts.\n2. Although the five knowledge states are formally defined, their separability and interpretability in borderline cases remain unclear.\n    - It is unclear whether the authors manually examined ambiguous instances to verify that the taxonomy aligns with human intuition.\n    - Moreover, under the multi-sample statistical testing setup, instances near decision boundaries may be prone to misclassification. \n\n[1] https://arxiv.org/pdf/2310.01558\n[2] https://arxiv.org/pdf/2404.03302", "was_revised": true, "final_justification": "I've considered the author responses and decided to keep my original scores.", "rebuttal": "We thank the reviewer for their constructive feedback and insightful suggestions. Please find our responses below. We will update the manuscript accordingly and would appreciate it if the reviewer could consider raising the score should these clarifications address the concerns.\n\n---\n**1. KScope under Real-World Retrieval Conditions**\n> All context passages used in the experiments are clean and supportive, which does not reflect real-world retrieval conditions\n\n* We acknowledge that clean and supportive context may not reflect all real-world scenarios. Our initial experiments focus on such controlled settings to isolate the effect of knowledge update and to minimize variability from the retrieval process.\n* To evaluate KScope under more realistic retrieval conditions, per your suggestion, we conduct a new experiment: we apply KScope to Llama-3.1-8B-Instruct using the top ten Wikipedia paragraphs retrieved for each HotpotQA question (i.e., the fullwiki setting in HotpotQA), which may or may not include the gold supporting context.\n    * Compared to using gold passages (Figure 3), using noisy retrieval leads to a reduced success rate of updating the model’s knowledge to the consistent correct status, particularly when the LLM lacks consistent correct knowledge in its parametric memory.\n    * | Context | Consistent Correct (%) |  Conflicting Correct (%) | Absent (%) | Conflicting Wrong (%) | Consistent Wrong (%) |\n|:----|:----|:----|:----|:----|:----|\n| Supporting | 98.19 | 94.21 | 96.62 | 91.10 | 82.91 |\n| Noisy | 82.83 | 66.50 | 45.67 | 49.32 | 48.86 |\n\n---\n**2. KScope’s Alignment with Human Intuition** \n> It is unclear whether the authors manually examined ambiguous instances to verify that the taxonomy aligns with human intuition.\n\n* To assess KScope’s alignment with human intuition, we evaluate its agreement with human judgments on the degree of knowledge conflict in LLMs.\n    * We collect 100 responses from Llama-3.1-8B-Instruct for each of 100 questions randomly sampled from HotpotQA. Based on the distribution of responses (i.e., the frequency of each option and invalid answers), three annotators label each question as “consistent”, “conflicting”, or “absent”. The inter-annotator agreement is strong (Fleiss' $\\kappa = 0.61$), and we use the majority vote as the ground-truth label.\n    * To ensure a fair comparison focused on knowledge conflict (rather than correctness), we collapse KScope’s “consistent correct” and “consistent wrong” into “consistent,” and similarly collapse “conflicting correct” and “conflicting wrong” into “conflicting.”\n    * We find that KScope agrees with human judgment in 89 out of 100 cases, suggesting strong alignment with human intuition.\n\n---\nWe thank the reviewer again for their thoughtful suggestions. We hope these additional results and analyses address the concerns and further support the validity and applicability of KScope."}]

Author rebuttal:

Meta-review: This paper tackles the important and challenging problem of understanding what LLMs truly "know." The proposed KScope framework is methodologically sound, leveraging established statistical tests to provide a more granular and principled characterization of LLM knowledge than prior work. The reviewers rightly identified the novelty of the five-status taxonomy and the rigor of the experimental design as major strengths.

The reviewers raised several concerns about the paper, such as generalizability beyond multiple-choice questions (Reviewers fYMy, Fosa, WvzF), comparison to baselines such as semantic entropy (Reviewer Fosa), realism of context beyond clean, supportive context (Reviewer gP5r) and alignment with human intuition (Reviewer gP5r).

However, the authors' rebuttal was exemplary. They conducted a significant number of new experiments that directly and thoroughly addressed every major concern raised by the reviewers. The authors have demonstrated an impressive commitment to improving their work during the review process. The resulting paper is substantially stronger and represents a solid contribution to the field of LLM evaluation. While the computational cost of the method (requiring multiple samples) is a limitation, the insights it provides are valuable. The paper is well-executed, the claims are now better supported by the new experiments, and it will be of interest to the NeurIPS community.

I am therefore confident in recommending this paper for acceptance as a poster. The authors should be strongly encouraged to integrate their rebuttal experiments and analyses into the final camera-ready version.
Decision: Accept (poster)