Inside-Out: Hidden Factual Knowledge in LLMs
Authors: Zorik Gekhman
Eyal Ben-David
Hadas Orgad
Eran Ofek
Yonatan Belinkov
Idan Szpektor
Jonathan Herzig
Roi Reichart
Venue: COLM 2025 (2025)
Keywords: LLMs
Knowledge
TL;DR: We introduce a framework for evaluating the gap between the knowledge LLMs encode internally and what they express in their outputs, and provide strong evidence of this gap across popular LLMs.

Abstract: This work presents a framework for assessing whether large language models (LLMs) encode more factual knowledge in their parameters than what they express in their outputs. While a few studies hint at this possibility, none has clearly defined or demonstrated this phenomenon. We first propose a formal definition of knowledge, quantifying it for a given question as the fraction of correct-incorrect answer pairs where the correct one is ranked higher. This gives rise to external and internal knowledge, depending on the information used to score individual answer candidates: either the model’s observable token-level probabilities or its intermediate computations. Hidden knowledge arises when internal knowledge exceeds external knowledge. We then present a case study, applying this framework to three popular open-weights LLMs in a closed-book QA setup. Our results indicate that: (1) LLMs consistently encode more factual knowledge internally than what they express externally, with an average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a model can internally know an answer perfectly, yet fail to generate it even once, despite large-scale repeated sampling of 1,000 answers. This reveals fundamental limitations in the generation capabilities of LLMs, which (3) puts a practical constraint on scaling test-time compute via repeated answer sampling in closed-book QA: significant performance improvements remain inaccessible because some answers are practically never sampled, yet if they were, we would be guaranteed to rank them first.

Published as a conference paper at COLM 2025
Inside-Out: Hidden Factual Knowledge in LLMs
Zorik Gekhman
Eyal Ben David
Hadas Orgad"
Eran Ofek
Yonatan Belinkov
Idan Szpector
Jonathan Herzig
Roi Reichart
TTechnion - Israel Institute of Technology GGoogl Research
zorlgehhman&gmaill.oom
Abstract
This work presents framework for assessing whether large language models (LLMs) encode more factual knowledge in their parameters than what they express in their outputs. While few studies hint this possi- bility, none have clearly defined or demonstrated this phenomenon. We first propose a formal definition of knowledge, quantifying it for a given question as the fraction of correct-incorrect answer pairs where the correct one is ranked higher. This gives rise to external and internal knowledge, depending on the information used to score individual answer candidates: either the model's observable token-level probabilities or its intermediate computations. Hidden knowledge arises when internal knowledge exceeds external knowledge. We then present a case study, applying this framework three popular open-weight LLMs in a closed-book QA setup. Our results indicate that: (1) LLMs consistently encode more factual knowledge inter- nally than what they express externally, with an average relative gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that model can internally know an answer perfectly, yet fail to generate even once, despite large-scale repeated sampling of 1,000 answers. This reveals fundamental limitations in the generation capabilities of LLMs, which (3) put a practical constraint on scaling test-time compute via repeated answer sampling in closed-book QA: significant performance improvements remain inaccessible because some answers are practically never sampled, yet if they were, we would be guaranteed rank them first.'
Introduction
Large language models (LLMs) excel at knowledge-intenee tasks, yet fundamental ques- tions remain open about the nature of their factual knowledge. What does it mean for LLMs to know a fact? And equally important, do LLMs store more factual information in their parameters than they express in their outputs? Answering the latter question rigorously can be highly valuable: if such knowledge is stored but not used, we might be able to develop methods to surface improving performance and reliability. From a safety perspective, undisclosed knowledge poses risks if it unexpectedly surfaces, revealing sensitive informa- tion or producing outputs never meant to be shared. Lastly, beyond practical considerations, this question is key to advancing interpretability: if models encode more knowledge than they express in their outputs, it highlights the need to understand how this knowledge is accessed or suppressed during inference.
In this work, our goal is to study whether LLMs encode more factual knowledge in their parameters than they express in their outputs, phenomenon we refer to as hidden knowledge. So far, prior work has only hinted its existence (Burns al., 2023; Orgad et al., 2025), but none have clearly defined it or the conditions required to demonstrate its existence. Consequently, our core contribution is proposing a concrete definition that provides framework for systematically studying hidden knowledge.
httpps: //gi thub. homm/oorikginnside-onn

---

Published as a conference paper at COLM 2025
To define hidden knowledge we first need to define knowledge, which is challenging since there is no well-defined notion of it for LLMs in the literature (Fierro et al., 2024). We couple knowledge with the ability to rank correct answers above incorrect ones using scoring method (e.g., token-level likelihood) and quantify it per-question as the fraction of (correct, incorrect) answer pairs ranked correctly ($2.1). While other possible definitions could be proposed, ours has three key advantages. First, unlike existing perspectives on LLM's knowledge (Fierro et al., 2024), it is specified with clear computational procedure, making it useful for empirical studies. Second, it addresses limitations in the common practice of measuring knowledge by evaluating performance of a single predicted answer on closed-book question-answeringg Lastly, and most importantly, it is designed with the hidden knowledge research question in mind, ensuring that both the knowledge a model expresses externally and the knowledge it encodes internally are measured under a unified definition. Since we quantify knowledge relative to function that assigns scores to candidate answers using information from the model, testing for hidden knowledge comes down to measuring knowledge through different scoring functions: external o which are restricted to use only observable signals based on the model's token-level probabilities, and internal ones, which can use intermediate computations. Specifically, we define hidden knowledge as the existence of an internal function that ranks answers more accurately than any external function ($2.2).
(QA) benchmarks (Petroni et al., 2019; Wei et al., 2024).
Using this framework, we design a study using 1,700 closed-book QA questions. We estimate the set of (correct, incorrect) answer pairs per-question using 1,000 gedee-generated answers, labeled for correctness by an LLM judge that compares against the ground truth (Wei et al., 2024). For internal scoring, we train a linear classifier to predict correctness from the model's ss hidden representations of question-answer pairs (Hupkes et al., 2018, 8; Belinkov & Glass, 2019). We then compare the internal knowledge measured by this classifier to external knowledge measured by popular scoring methods that use the model's token-level probabilities. Our results strongly indicate the presence of hidden knowledge. Across three LLMs, the internal scoring method consistently measures more knowledge than any external one, with all differences statistically significant and an average relative gap of
But how deeply can knowledge be hidden? In 56% of the questions, all the model-generatee answers are wrong. So we manually add the ground-truth answer to the set of candidate answers, and analyze its impact on the model's internal knowledge scores. Surprisingly, in 9% of the questions, the internal scoring method scores the ground-trute answer higher than any incorrect candidate, despite the model failing to generate if even once across 1,000 attempts. This illustrates an extreme case of hidden knowledge: while models may occasionally generate an incorrect answer despite knowing the correct one (Simhi et al., 2024), it highly surprising that a known correct answer is practically never generated, even with large scale repeated sampling. This highlights fundamental limitation in the generation process of modern LLMs, which we hope future research on decoding mechanisms will explore.
Lastly, we leverage our findings to enhance closed-book QA performance. We obtain 12% average relative improvement over greedy decoding by increasing test-time compute through sampling a large set of answers and selecting the top one based on our internal scoring function. This extends existing evidence on the potential of using verifiers trained on LLMs hidden states to improve performance (Orgad et al., 2025). However, the more interesting insight is that there is potential for substantial gains of additional 40% relative improvement that remain inaccsssible due to the constraints we identified in the LLM generation process: these constraints prevent some correct answers from even being sampled, yet if they were, we would be guaranteed to choose them since they would always be ranked first.
To conclude, we introduce a framework for evaluating the gap between the knowledge LLMs encode internally and what they express externally, and provide evidence of this gap in popular LLMs. Notably, its magnitude varies significantly: Gemma (Team et al., 2024) shows a relative gap of 57% but Llama (Dubey et al., 2024) 14%, highlighting crucial direction for future work: understanding these differences and developing methods to surface internal knowledge more effectively, leading to more transparent and reliable LLMs.
The term external may suggest elements outside the model, which is not our intention. An alternative name could be obseroable. We use external since it reads more smoothly in our context.
2

---

Published as a conference paper at COLM 2025
2 Hidden Knowledge
In this section, we tackle the challenge of evaluating hidden factual knowledge in LLMs, go- ing beyond prior discussions ($5) by proposing a formal definition. We focus on knowledge represented as (subject, relation, object) triplets, whose structured nature simplifies evaluation and helps control for confounding factors such as ambiguous or easily guessable answers and overlaps between training and test examples. We first define knowledge ($2.1), focusing on subjects and relations with a unique object for clarity, as extension to multiple objects is straightforward. We then define the conditions under which model is said to possess hidden factual knowledge ($2.2) and outline how to estimate it for a given LLM ($2.3).
2.1 Defining Knowledge Relative to Answer Scoring Method
We consider an auto-regressive language model M, where the next-token distribution given a context x is PM x). Given a question 4 and an answer a, we denote M's token-level probability of generating as (M q) = =II= -T=
PM q, In practice, M is prompted to
answer 9 (see $A.2), so denotes Prompt(q), but we omit the Prompt notation for simplicity.
A common way to test if M knows a fact, e.g., ("Empire State Building", location, "NYC"), to prompt it with a question like "Where is the Empire State Building located?" (Wei et al., 2024). However, M's outputs may vary with decoding strategy or question phrasing. Furthermore, decoding algorithms can lead to an incorrect answer despite M assigning higher likelihood to a correct one. Lastly, one correct answer does not guarantee full knowledge, e.g,, M should know that both "NYC" and "New York City" are correct. This calls into question the reliability of inferring M's knowledge from one answer. We propose to examine scores, such as token-level probabilities, that M assigns to all plausible answer candidates. Specifically, we quantify knowledge per-question relative to the ability to score any correct answer higher than any plausible incorrect one, regardless of question phrasing. Since there are many possible ways to use M to score an answer, we define knowledge relative to a scoring method, which is also useful for our objective of comparing between internal parametric knowledge, and the knowledge that is expressed externally ($2.2). We focus on a question-answer structure since it simplifies the definition and discuss alternative settings in SA.15.
Definition 1 (Knowledge of Model w.r. a Scoring Method).
For a model M and a fact represented as a (subject, relation, object) triplet (s, egg., ("France", capital, "Paris"), we define the following sels:
Q(s, r): All paraphrases of questions based on and r. E.g., (s,r) = ("France", capital), it may include "What is the capital of France?", "Which city is the capital of France?", elc.
(o): All plausible answers to r),, defined as all paraphrases of entities that have the same type as o. E.g., if o = "Paris", may include paraphrases of city names such as "Paris", "The city of New York", etc.
A(o) C ((): All paraphrases of o. E.g,, if o "Paris", , may include "Paris", "The city of Paris", elc.
((s, == A()) x [Ā(o) \ A(o)]: : All ordered pairs of correct answer and plausible incorrect answer to (s,,r). E.g., if (s,r) = ("France", capital), it may include ("Paris",
"London") ) ("Paris city", "NYC"), etc.
Given a scoring function Su Q(s,r) x A(o) + R, which, given a question answer pair (q a), uses M to predict whether a a correct answer to q, we define a per-question score Kg (s, r, o; Sw)) to quantify the ability to rank correct answers above plausible incorrect ones. Formally:
Ks = [N(s,,,))
Σ I(SM(q,a) > Sx)(q.a) (2ā))E((5TT.))
(1)
The overall knowledge degree of M for the fact faetfact r, 0) according to Su is then defined as:
SSM accounts for the different ways to score based on information from M. In case it has parameters
(e-g, a probing classifier), we assume that they were not optimized using information about (s, r, o)
3

---

Published as a conference paper at COLM 2025
S(, [s,,r)
Σ Kg
(2)
qeQ(s.r
Finally, we define K* to reflect cases of perfect knowledge, where Su correctly ranks all pairs:
K (s, r, Su) =
(0ss,,,:S)=<(((sss..)))) I(K(s,,,,;S,))) =1)
(3)
Since we quantify Kg based on answer from (s,,,, o),, Sy could in theory score an implausible answer (e.g,, "#% ") higher than a correct one. While it is extremely unlikely, we can also enforce it formally. We present an extended definition supporting this in $A.9. We also note that Kg is conceptually related to the area under the ROC curve (AUC-ROC), interpretation of AUC-ROC is the probability that a random positive answer is ranked higher than random negative one (Fawcett, 2006). We discuss the differences in $A.11.
one
2.2 Evidence of Hidden Knowledge
We define hidden knowledge as cases where M embeds more knowledge in its parameters than expresses externally. To formalize this, we differentiate between internal and external scoring functions. External functions are restricted to using only M's observable signals based on its predicted token-level probabilities, while internal functions can leverage inter- mediate computations, such hidden states. Hidden knowledge is measured by comparing the knowledge captured by internal vs. external scoring functions.
Definition 2 (Evidence of Hidden Knowledge).
Given model M, let TM be an internal scoring function, St. the set of all external scoring ccoringfunctions functions,
and D = { ss ri, o;) | a dataset where each element is a unique fact. We say that Th demonstrates that M has hidden knowledge of DD if M has more knowledge of D according to Taa than according to any SM E St. with a margin A that ensures the difference is sufficiently large to rule out insignificant variations. Formally it is defined as follows:
K(s,, I, 0j; Tm) >
max SM
1 -
Ki n, } +
}
(4)
2.3 Estimation
We now discuss best practices for effective estimation hidden knowledge, independently of the specific choices for our study ($3.2). To evaluate hidden knowledge, we must (1) select an internal scoring function Tu, (2) estimate the set of external functions S (3) compute K for each scoring function, and (4) compare K(:; Tx) with K(-;SM) for any external function.
Scoring Functions. For S natural choice is (M q). On top of that, can include normalizing (M 9) by a's length (Wang et al., 2023) and prompting M to verify a's correctness (Kadavath et al., 2022). Tm can exploit internal model structures, such as hidden states. A common choice is a probing classifier (Hupkes et al., 2018; Belinkov & Glass, 2019).
K. To estimate K(s, r, Om)) we need to (1) estimate r),, (o), and (o)), and (2) score each a) E r), x (o) with Su. Qs,, can be obtained by paraphrasing one question. For Ao)), it is typically computationally infeasible to enumerate all entities from the relevant type. E.g., if is "married to", there are approximately 8.2 billion people worldwide. Uni- formly sampling a subset of entities may lead to trivial negatives, answers that are clearly incorrect and easily ranked below correct ones. To increase the likelihood of negative answers that are hard for M, we can sample answers from M. However, this introduces risk of failing to sample correct answers that are deemed unlikely by M, or even not sampling correct answers at all. To mitigate this, we can manually include and its paraphrases. Lastly, to estimate A(o) C Á()), we must determine the correctness of each à E A(o) by comparing it to o. Since string matching is sensitive to paraphrasing or differences in granularity (Yona et al., 2024), we recommend using an LLM-as-a (Wei et al., 2024).
3 Study Design
We now present the design of our study that approximates hidden knowledge according to our definition (52). In our main setup throughout the paper we use three popular open-
4

---

Published as a conference paper at COLM 2025
weights LLMs: Llama-3-8 3-8B-Instruct (Dubey et al., 2024), Mistral-7B-Instruce (Jiang et al., 2023), and Gemma-2-99 2-9-sn--ttt (Team et al., 2024). To provide evidence of hidden knowledge in larger models, in $A.10 we also run the main experiment with Qwen3-32B (Yang et al., 2025), but we smaller set of answers to approximate (o) (see $3.2) due to compute budget limitations. More details on the models are in $A.12.
3.1 Collecting the Set of Factual Triplets D {(s,, r; 0,)
We build on EntityQuestions (Sciavolino et al., 2021), as it contains triplets from Wikidata (Vrandečić & Krötzsch, 2014) that were already converted into QA pairs. We categorize the relations (Table in the appendix) focus on ones that are (1) hard to guess and (2) have single, unambiguous answer, making them easy to grade. Specifically, we use P26 (spouse), P176 (manufacturer), P264 (record label), and P50 (author). To form the development and test sets, we use 500 questions per-relation from the Enti EntityQuestions test split, allocating 10% for development. The training set is based on 500 questions per-relation from the Entityquestions train split. The training and development sets are used only to train the internal scoring function. Full details on the data creation process are in $A.1 and data statistics are in Table 3 in the appendix.
3.2 Approximating the Quantities Interest
k There are two main paradigms for external scoring functions, which operate on the M's observable token-level probabilities, we test both:
Production-oriented. Measure how well M assigns probability mass to when prompted to answer 9, under its autoregressive decoding process. We compute this likelihood using M's token-level probability, P(alq)= I[ P(a: q, a.aci), as well as its length-normalized variant (Brown et al., 2020; Wang et al., 2023): P Poorm (a q)
(IT P(a: 9. 9) = exp ( Σ" log P( q, q.a<i)).
Verification Estimate the likelihood M to classify as a correct answer to q. This classification can be implemented either by prompting M to generate verbal scores (Lin et al., 2022; Tian et al., 2023), or by prompting it to assess a's correctness and inspecting the likelihood of generating the "True" token (Kadavath et al., 2022). We focus on the latter as worked substantially better in early experiments. Formally, we use (TTru)), defined as P("True" q.a).
Á(o) To approximate the set of plausible answers, we (1) generate one answer with greedy decoding and (2) sample additional 1, 000 answers with a temperature of Figure demonstrates the diminishing returns of sampling more answers. E.g., for P50, the last 200 answers (beyond the first 800) contributed 0.003 to the probability mass, indicating that answers obtained at this stage are either duplicates or have very low token-level probabilities. Next, (3) we add the gold answer from intityquustions (i.e., o), when was not sampled, which occurs in 64% of questions on average.
40241
+0007 10.008 +0.005 -0003
0.8
0.6
0.4
0.2
P264
P176
P30
200
600
800
1k
Ao) The set of correct answers is estimated using an LLM judge that compares each ă E A(o) to o. We discard ~8% of the questions where all sampled answers lack comparative value; alternatively, we could manually set K to 1. More details and human evaluation of the judge are in SA.3.
Figure 1: Accumulated prob- ability mass as a function of sample size for Mistral-7B- Instruct. For P50, the differ- ence between every 200 sam- ples is annotated.
correct, as they
Q(s, r). We the original question from Enti EntityQuestions (i.e., (Q(sr)) = 1), avoiding paraphrasing because our experiments are already com- putationally expensive: they require sampling a large number of answers per question, annotating each with an LLM judge, and scoring them using three different methods.
WW tested generating verbal scores different scales (0-1, 1-10, 1-100), but the performance was poor. We note that verbal scores have mainly shown effectiveness in reasoning tasks.
5

---

Published as a conference paper at COLM 2025
Llama 3 8B
Mistral
Gemma 9B
+23%
+28%
166%
06
02
1756
P134
11766
F50
P35
1264
F176
1701
17h
1264
P116
150
0.3
+8.5%
-12%
+15P%
K
+38%
+200%
+12I%
+355%
0.1
+1.5k%
P25
P154
P176
F50
P35
P26
F176
150
F2b
7264
P176
PS0
P(alq)
External
P Pnorm (a/q)
Exiernal
P(True)
External
Probe
Internal
Figure Average K (top) and K* (bottom) scores, as defined in equations 2 and 3, for each scoring function, relation and model. The bars are sorted according to the order in the legend (for color-blind readers), and the percentage difference between the best best-performing external scoring function and our internal scoring function (Probe) is annotated. All those differences are statistically significant with < 0.05.
A. We set the threshold A (Equation 4) dynamically per setup based on statistical significance by performing paired t-test with p-value of 0.05. Technical details are in SA.7.
T TM- The space of possible internal functions is vast, and finding the optimal one is beyond this work's scope. Since our goal is to explore the existence of hidden knowledge, demon- strating it with a single internal function is sufficient. Consequently, our results should be interpreted as lower bound. We pick Ty to be probing classifier (Ettinger et al., 2016; Belinkov et al., 2017; Hupkes et al., 2018; Belinkov & Glass, 2019). Specifically, our probe linear classifier, trained with a logistic regression objective, that receives M's hidden state hM q, a), obtained by encoding 9 and a, and classifies as correct or incorrect. We then use its output probability, representing the likelihood that a is correct, as Tи- The probe is trained on (g,a) a) pairs labeled for correctness, with full technical details provided in $A.5. We create the training data based on the training split using the process described in SA.1. We train probes for all layers and choose the best layer based development set.
The fact that the probe is trained introduces a risk that may rely on knowledge acquired during its own training, rather than reflecting M's internal representation of truthfulness. We mitigate this through careful data curation, ensuring that the factual information present in the training set is not useful for classifying test examples. Since map to (subject, relation, object) triplets, we can ensure that there are no subject and object overlaps between the training and test splits. In SA.6 we also empirically verify that the factual information from the training set is not useful for classifying test examples.
QA pairs that
Lastly, we ensure that train mostly on questions for which have high certainty that M knows the answer. The intuition behind this is that we want to train the probe to distinguish between multiple answers the same question, so we want to ensure that when we train on (q. a) pair, M encodes the information about a's correctness. We refer to this approach as knowledge-aware probing and discuss detail in $A.5.
4 Results
4.1 Evidence of Hidden Knowledge in LLMs
Figure presents the average K and K* scores across all three models and four relations. Notably, across all 12 setups, the average K based on the internal scoring function is larger than based on any external one, with all the differences being statistically significant (see SA.7). This is also the case for the stricter definition of full knowledge K*. This provides strong evidence of the existence of hidden knowledge in LLMss. The magnitude of the

---

Published as a conference paper at COLM 2025
Llama 3 8B
Mistral 7B
Gemma 2 9B
0.3
P(a
+5%
0.1
-16%
-20%
0%
-21%%
0%
0%
P26
1264
P176
150
P26
P264
P176
P50
P26
P264
P176
P50
+88%
+127%
+93%
Probe
0.2
+580%
+62%
+163%
+675%
+108%%
+86%
+00%%
P264
P176
P264
P176
F26
P264
P50
Sampled Only
Sampled
Gold
Figure 3: Comparison of average K* values (see Equation (3)), under two conditions: without manually adding the gold (Sampled Only, left bars) and with manually adding the gold (Sampled Gold, right bars).
gap between internal and external knowledge varies substantially across models, with an average difference in K of 57% for Gemma but only 14% for Llama. This indicates that models differ in their ability to fully expose their knowledge, likely due to factors such as differences in training data, methodologies or architecture, calling for future work explore how to better facilitate knowledge exposure.
When examining the external scoring functions, we see clear advantage for P(True), as it outperforms other external functions in every setting. P(True) also accounts for the relatively low magnitude of hidden knowledge in Llama, as the gap between Probe and the other two external functions is considerably larger. This shows that LLMs can exhibit gap between the answers they can generate" 5 with reasonable likelihood and those they can verify as correct when prompted to do so (Huang et al., 2025; Rodriguez et al., 2025).
4.2 LLMs Can Fail to Generate Facts They Fully Know, Even After 1,000 Attempts
In human cognition, the tip of the tongue state describes inability to recall a known word (Brown & McNeill, 1966). Psycholinguistics distinguishes comprehension from production, noting that people may understand words yet fail to retrieve them (Treiman et al., 2003). Linguistic theory emphasizes that performance (language use) may not reflect competence (language knowledge) (Chomsky, 1965). Could LLMs exhibit a similar cognitively plausible gap between retrieval mechanisms and their internal knowledge?
We have shown that LLMs encode more knowledge than they express externally. We now demonstrate that some knowledge so deeply hidden that M struggles to even consider known correct answer as a candidate during generation. To this end, we take a closer look at the impact of manually adding the gold answer to A(o) when it was not sampled ($3.2). this setup, As is a correct answer that M is highly unlikely to generate, as was not sampled after 1 attempts, allowing us to test if M can know ag despite failing to generate it. Figure 3 compares K* scores between using only the sampled answers as A(o) and our main setup (64.1) where ag is manually added. We focus on our notion of full knowledge K' since aim to show cases of perfect knowledge but failure to generate (we report and discuss K scores in SA.13).
As would expect, adding ag never improves K scores for P(alq) (top row). Any improvement would require K* to transition from 0 to for certain questions, which is highly unlikely since will require ag receive a higher score than any incorrect candidate.
Since autoregressive generation involves sampling from M's token probability distribution, the likelihood assigned to a given (measured by P( q]) directly reflects how likely M is to generate it. Therefore, if M assigns higher P( (a)q) scores to to generate correct answer.
correct answers than to incorrect ones, it is more likely

---

Published as a conference paper at COLM 2025
P(a q)
Paorm (a q)
P(True)
Probe
Answer
Score
Answer
Score
Answer
Score
Answer
Score
BMW
0.761
BMW
0.873
BMW Group
0.980
Volvo Buses
0.463
Volvo
0.012
BMW Group
0.114
Volvo Buses
0,980
Volvo
0,080
BMW Group
0.001
Volvo
0.110
BMW
0.941
BMW Group
0.065
Stellantis
0
Stellantis
0.041
Valvo
0,926
BMW engines
0.028
BMW engines
engines
BMW engines
Volvo Buses
0
Volvo Buses
0.001
Stellantis
Stellantis
0.002
K = 0.375 (3/8)
K 0.25 (2/8)
K = 0.625 (5/8)
== (8/8)
Figure A real example of scores assigned to each answer E A(r) to the question "Which company is Volvo B58 produced by?", according to each scoring function using Gemma 2 9B. Answers are sorted by score, with correct ones colored in green and marked +, and incorrect ones colored in red and marked
Notably, for Probe (bottom row), we observe considerable improvements across all setups, indicating that K" often transitions from 0 to This happens when there were no correct answers sampled, and thus K* was manually set to O, yet when ag was added A(o), Probe ranked it higher than all the incorrect candidates. This demonstrates an extreme case of hidden knowledge: M fails generate ag after 1K attempts, yet still perfectly knows that ag is the correct answer, it is able to rank it higher than any incorrect candidate.
To further substantiate this finding, we directly quantify cases where M has perfect knowl- edge of a fact but is extremely unlikely to consider correct answer during generation. We define such cases by the following conditions: (1) No correct answer was sampled after
1k attempts, (2) P(ac q) < 0.01, and (3) K 1. On average, these cases occur in 7.2% of the questions, demonstrating that they are not just rare cases with negligible impact. This finding highlights a fundamental limitation in the generation process of LLMs. While it is expected that M may occasionally generate an incorrect answer despite knowing the correct one (Simhi et al., 2024), it is highly surprising that a known correct answer is practically never generated, not even once, even with large-scale repeated sampling. This limitation holds to many popular decoding methods that use next-token probabilities, since they are guided by the P(alq) distribution, meaning that if P(alq) is low, the chance of generating remains low. Understanding the cause of this limitation and how to mitigate it is an important direction for future research on decoding mechanisms. The solution may lie in decoding paradigms that consider internal signals, e.g,, Rimsky et al. (2024).
4.3 A Case Study
Figure 4 presents a case study, comparing scores assigned to each answer E A to the question "Which company is Volvo B58 produced by?". Since "Volvo" appears in the text, the question may seem easy. However, while "Volvo B58" refers to a bus made by "Volvo Buses", the term "B58" is also the name of an engine produced by "BMW". So the model must recognize that "B58" refers to a Volvo bus, not the BMW engine. It is particularly evident that the likelihood of generating" a correct answer, reflected by P(alq) and Pnorm al a))), is extremely low. Interestingly, even though P P(True) is also exlernal score, ranks "Volvo Buses" significantly higher, demonstrating the gap discussed in $4.1 between the ability to generate and the ability to verify correctness. However, P(rue) assigns the same score to "Volvo Buses" and the wrong answer "BMW Group", indicating that M struggles to distinguish between them. Despite that, the internal scoring function perfectly ranks the answers, providing a real example of a case where M encodes the knowledge in its parameters but fails to express it externally. Lastly, the gold answer in this case is "Volvo Buses", but it was not generated by M in all 1,000 samples; only "Volvo" was sampled, possibly because "Volvo" appears in the question. This illustrates the limitations LLMs' generation capabilities we discussed in $4.2.
This example also highlights the importance of our two key design choices. First, assessing knowledge w.r.t. to large set of answers ensures a meaningful comparison. E.g., if we considered only "Volvo" and "BMW engines", all methods would yield a similar ranking. Second, approximating A(o) by sampling from M is very useful as allows to automatically generate challenging candidates like "BMW".
8

---

Published as a conference paper at COLM 2025
Llama-3-8B
Mistral-7B
Gemma-2-99
Average
Greedy
22.1
18.8
22.7
21.2
Random
16.4" (-25.8%)
12.3" (-34.6%)
11.3" (-50.2%)
13.3" (-36.9%)
Majority
23.7" (+7.2%)
19.7" (+4.8%)
22.6 (-0.4%)
22.0" (+3.9%
P(alq)
23.6" (+6.8%)
20.0" (+6.4%)
23.2" (+2.2%)
22.3" (+5.1%)
Probe
25.4" (+14.9%)
22.0" (+17.0%%
23.7" (+4.4%)
23.7" (+12.1%)
Oracle
44.2" (+100.0%%
38.9" (-106.9%)
49.8" (+119.4%)
44.3" (+108.8%)
Probe w. gold
34.5" (+56.1%)
33.9" (+80,3%)
27.6" (+21.6%)
32.0" (+52.7%)
Table 1: Closed-book QA accuracy when selecting the top-scoring answer from 1k samples using different scoring methods. Oracle score each correct answer as 1 and each wrong one as O. Probe w. gold includes the gold answer if it was not sampled. marks statistically significant differences (p 0.05) from greedy.
4.4 Increasing Test- Time Compute via Repeated Answer Sampling and Ranking in Closed-Book QA
A practical implication of hidden knowledge is the potential to develop methods that better expose it, improving downstream task performance. A simple approach is to sample multiple answer candidates and select the correct one (Brown et al., 2024; Hassid et al., 2024; Zhao et al., 2025). We test its feasibility in our setup when sampling 1K answer candidates and selecting the highest-scoring one, aiming to surpass greedy decoding. Table 1 presents the results. For brevity, we aggregate all relations per model.
Notably, greedy decoding performs poorly, indicating a challenging setup, likely due to long- tail knowledge requirements. Interestingly, even with short entity-focused answers, greedy decoding does not always select the globally optimal path, as evidenced by the improved performance when selecting answers using q( This highlights the importance of the p()lq) baseline in controlling for a major confounder: ensuring that the probe does not simply resort to selecting the answer with the highest global token-level probability. Probe demonstrates notable relative improvements of 12% on average across LLMs, providing further evidence of the potential of self-verification based on the model's hidden states to enhance downstream performance (Orgad et al., 2025). The Oracle baseline, where a correct sampled answer is always ranked first, provides an upper bound for perfect scoring method. The fact that it remains the highest, shows that for considerable amount of questions, we successfully sample correct answer but fail to identify it, which could be attributed to guessing rather than knowing (Yona et al., 2025). However, the most interesting results are for Probe w. gold, where the gold answer ag is manually included if not sampled. Consistent with $4.2, ag would often be selected as the top answer if only it was sampled, which could lead to 52% average improvement over greedy (i.e. additional 40% compared to Probe). This result highlights a substantial potential for improving performance that remains inaccessible due to the constraints we discovered in the LLMs' generation capabilities. A natural step for future work is to develop better sampling methods that produce high- quality and diverse candidates to boost test-time performance.
Related Work
Knowledge of LLMs. The factual knowledge of LLMs has been widely studied. Early work considered a model to know fact if it correctly completed a cloze sentence (Petroni et al., 2019; Jiang et al., 2020; Kassner et al., 2020, inter alia), or directly answered a question, either in a zero-shot setting (Radford et al., 2019) after fine-tuning (Roberts al., 2020). Modern LLMs, capable of instruction following, are typically directly prompted to answer questions (Wei et al., 2024; Singhal et al., 2023; Anil et al., 2023; Dubey al., 2024; Cohen et al., 2023, inter alia). These efforts have largely been guided by what appears intuitively reasonable, without a clear definition of knowledge (Fierro al., 2024). Beyond the lack of formal definition, key limitation in previous work is that even though studies emphasized the importance of evaluating predictions across semantically equivalent questions (Elazar
'It is important to stress that while higher K increases the chances of success under inference scaling, it does not guarantee it. In SA.14, we include detailed discussion on the relationship between our K measure and inference scaling.

---

Published as a conference paper at COLM 2025
et al., 2021; De Cao et al., 2021; Zheng et al., 2023), most of them focused on evaluating single model response. As discussed in 12.1, we posit that the relative ranking of all plausible answers is important and design our definition to reflect this. One aspect we leave out of scope is verifying related facts when measuring knowledge (Kassner et al., 2021; Zhong et al., 2023; Cohen et al., 2024). E.g., to conclude that a model knows that Paris is France's capital, we may also check that it knows that Paris is a city in France. We hope future research will explore corresponding extensions our definition.
Hidden Knowledge in LLMs. Findings from prior work suggest that LLMs may encode what we define as hidden knowledge. There is growing evidence that LLMs encode truthfulness information, enabling assessment of individual candidate answers' correctness either via probing the model's s internal states (Burns et al., 2023; Azaria & Mitchell, 2023; Marks & Tegmark, 2024, inter alia) or by prompting it directly (Lin et al., 2022, Kadavath et al., 2022; Tian et al., 2023), with these approaches not always agreeing (Liu et al., 2023). Other studies showed that a model can be "steered" to answer correctly where it previously failed et al., 2023b; Zhang et al., 2024; Tulchinskii et al., 2024; Rimsky et al., 2024). Turpin et al. (2023) showed a model can be biased by its input structure towards incorrect answers about known facts, and even generate a plausible justification. Gekhman et al. (2024) showed that fine-tuning on new knowledge can cause hallucinations on facts that were known to the pre-trained model, raising questions about whether those facts are fully forgotten. Indeed, some results suggest that even when fine-tuned model fails to recall fact, may still encode information about it in its representations (Gottesman & Geva, 2024; Patil et al., 2024). These findings only hint at the existence of hidden knowledge but do not define or systematically demonstrate it. For example, studies that show that LLMs encode truthfulness information usually test the ability to classify individual statements as correct or incorrect. However, success in such classification could stem from uncertainty representation rather than factual knowledge. E.g., knowing that an answer is wrong does not guarantee knowledge of the right answer. Although our probe is trained in related manner, we differ since rather than evaluating performance across a large set of individual QA pairs, we quantify the ability to correctly rank all possible answer candidates for a specific question, which is much more likely to reflect knowledge of the relevant fact.
Scaling Test-Time Compute. Considerable progress has been made in improving per- formance by increasing inference compute (Snell et al., 2025; OpenAI, 2024; Guo et al., 2025). A popular approach is to sample diverse responses and use verifiers to identify the correct one (Brown et al., 2024; Hassid et al., 2024; Zhao et al., 2025). Most studies focus on reasoning tasks, raising the question of whether such approaches can be effective for knowledge-intensive QA with short answers, where reasoning diversity matters less. Orgad et al. (2025) provided initial evidence by using probing classifier to select the best answer among 30 model-generater candidates. We extend this evidence in a highly controlled setup facilitated by our framework and, more importantly, show that further performance gains are possible but constrained by the model's generation capabilities, specifically its ability to recognize the correct answer while failing to generate it as a candidate.
6 Conclusion and Future Work
We present framework for assessing the extent to which model encodes more factual knowledge in its parameters than it expresses in its outputs. Our definition of knowledge addresses limitations of measuring it based on single generation's performance, and enables a unified measure of both internal and external knowledge, facilitating our definition of hidden knowledge. We then conduct a controlled study which indicate that LLMs consis- tently exhibit hidden knowledge to varying degrees, stressing the need to understand these differences and build models that better use their knowledge, for which our framework can serve as a foundation. We provide an extended discussion about potential reasons for hidden knowledge and possible mitigations in $A.16. We also demonstrate an extreme case of hidden knowledge, where the model perfectly knows an answer but is highly unlikely to generate it even once via repeated sampling. This highlights a limitation in the LLMs' generation capabilities which puts practical constraint scaling test-time compute via repeated answer sampling in closed-book QA, and opens an interesting direction for future research on decoding mechanisms.
10

---

Published as a conference paper at COLM 2025
References
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arxiv preprint ar ari::2305510000 2023.
Amos Azaria and Tom M. Mitchell. The internal state of an LLM knows when it's lying. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 967-976. Association for Computational Linguistics, 2023. doi: (11655/VV//022/ 2023.FINDINGGS-E https://doo..org/..... 18653/v1/22222
MNLP.68. URL wttpps//dde org/10.
pendings--enlp.... 68.
Yonatan Belinkov and James R. Glass. Analysis methods in neural language processing: A survey. Trans. Assoc. Comput. Linguistics, 7:49-72, 2019. doi: 10.1162/TACII, A 00254.
URL https://doi....... org/10. 1162/tacl_a_02221
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural machine translation models learn about morphology? In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861-872 Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ 11/033/11P17110//// URL https: /aclaa/thol/laaooogoooo .org/P17-1080/.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint ar Xio:2407.211785 2024.
Roger Brown and David McNeill. The "tip of the tongue" phenomenon. Journal of Verbal Learning and Verbal Behavior, 5(4):325-337, 1966. ISSN 0022-5371. doi: https://doi.org, (0066/50022-5333116))00044 URL https: www.sciieneeiree...... wttp:ieeeett.tccceeeeeeeeeeeeeee com/science/articlle
pi1/500225371668804333
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virlual, 2020. URL https::/t///... /ttpp:::////ceeeeeeeeeee http://prooeeein........ neuriis cc/ cc/paper///p/p/pppccccpceeeeeeet htap/hash/1/55 457cc 0d6bfcb4967418bfff 4af./66a71 42f644 html.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openrrvviee net net/pdf? http://w.eee//d/PPdddddCCCuuooo id=ETKGubyehcs.
Noam Chomsky. Aspects of the Theory of Syntax. The MIT Press, 50 edition, 1965, ISBN 97802625274088. URL htttp://wwe jstor www.:jjstoroor/stteeeoo htt.://abbl/..cct1kkk1222 ctt17kk81z.
Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. Crawling the internal knowledge-base of language models. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACL 2023, pp. 1856-1869, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653 URL https://aclanthol org/2823. findings-eacl. 139.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Trans. Assoc. Comput. Linguistics, 12:283- 298, 2024. doi: 10. /112/TTACL A) 00644. URL https:////doo org/10 1162/tacl 00644.
11

---

Published as a conference paper at COLM 2025
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Frrie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, PP. 6491-6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v/ 2022 Lwm.l-mmiin5522 URL https:
//aclanthology. org/2021 wttgg/21..emmmmmaaiii. 522/.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint or Xio:2407.217833 2024.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Trans. Assoc. Comput. Linguistics, 9:1012-1031, 2021. doi: 10.1162/TACE
{ 00410. URL https: /ttp::://do..oo org.i..oogg// 11 1162/tacl 00410.
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 134-139, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.1863/vv1/W162255 https://aclaanthologg. wwtp:////o..oo///////5//00 2524/.
URL
Tom Fawcett. An introduction to ROC analysis. Pattern Recognit. Lett., 2766))861-55442006 URL https: https:://doi.ooo org.1/......//... 10. 1016/j http:/000e0e.pppaeeee0.00000 www.ec.2005.......
doi: 10.1066/PPATREE2205.000 010.
Constanza Fierro, Ruchira Dhar, Filippos Stamatiou, Nicolas Garneau, and Anders Segaard. Defining knowledge: Bridging epistemoloyy and large language models. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 16096-16111 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: Ip-main.900 900. URL https://acce antthoooogy or org/2024 emnlp-main.9e 900/.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning LLMs new knowledge encourage hallucinations? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7765-7784, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 /v1/2024. /v1/2024.mmll--main emnlp-main.444 URL https: httpss//aclanthologgy org/2024. emnlp-main. 444/.
Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without generating a single token. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3994-4019, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 2003//112222.... www.....mllpmmmm....mm. 232. URL https: h/tps//htlaaaann.....o/.. org/2 024.emnlp emnlp-main main. 232/.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arviv preprint Arx:::2501. 12948,, 2025.
J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29-366 1982. doi: 10.1148/ 10.1148/radiologe /rriiooooooy1133....666 747. URL https: / pwwmed. ncbi. .nlm. ww....ii................ gov/76637777.
Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The larger the better? improved LIM code-generation via budget reallocation. In First Conference on Language Modeling, 2024. htte.httss//oppeeeiiiee wttp://wwww//ooooo////////////a/
Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T. Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representa- fions, 2025. URL https: /ttp:///vpp.....e.///m wttpiiewww../tfooom//////aaaalae
12

---

Published as a conference paper at COLM 2025
Dieuwke Hupkes, Sara Veldhoen, and Willem H. Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif. Intell. Res,, 61:907-926, 2018. doi: 10.1613/JAIR.111996 URL
https://doi org/10. 10 11196..
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint ar arXit:2310006825, 2023.
Zhengbao Jiang, Frank F. Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423- 438, 2020. doi: 10.1162/tacl...000 ta022 URL pittps://aclinn https://aclaanthology 2tt.aaanntoooooroor/222200
cacl-1.22 28.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arv preprint Xiv:2200 Siff:2207705221,, 2022.
Nora Kassner, Benno Krojer, and Hinrich Schütze. Are pretrained language models symbolic reasoners over knowledge? In Raquel Fernández and Tal Linzen (eds.), Proceedings of the 24th Conference on Computational Natural Language Learning, pp. 552-564, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/20200 -1.45. URL https: /acla/tthlla org 2220 2tt///2.c.cc11..555555 con1l-1.45/.
Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. BeliefBank: Adding memory to pre-trained language model for systematic notion of belief. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8849-8861, Online and Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. doi: 10. 10.18653/ org/2021 http//221.mmmnmmaaaa...mm wwn.e-main.neeeememmmmmmmmm 697/.
m9.1.697. URL
Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a. URL http: /pppers/pppeeepeaeaaaeee nips. ctt.cccppaaaa cc/paper__.. httppapppleeeeaalllllallallaeeee www.feeeeleee/e/eee http:p/////////hh/a//aaaaa hattssh////bbb/03bbbb (tt.c999999c66688866633883
html.
Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL http: / / http:/pppprrr...... cttpppccc//ooooooccpoe sc/paper__a waape_files/pppeeaaprrppppesseee pttpl///pper//223///// hww.////h/hhha/eeeee 818s9////bbb99bbbbb frreeeee
fernnce.eeeee.eem html.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022, 2022. URL https: // htpp//ppnervvveeeeeeeeeeeee wwwpeeeeiweeeeeeeorriii.... 8s8K2UZGTZ.
Kevin Liu, Stephen Casper, Dylan Hadfield-Menell, and Jacob Andreas. Cognitive dis- sonance: Why do language model outputs disagree with internal representations of truthfulness? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4791-4797, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023.emm 1p-main.291. URL httpp:://ace /tntto////allatthollogyy org.//23333333 arg/2023.emnlp-maii.. 291/
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large false datasets. In First Conference on Language language model representations of frue/false Modeling, 2024. URL. https: httper/wwww..eeeoooo//aaaaaaaaa.
13

---

Published as a conference paper at COLM 2025
OpenAL Introducing openai ol preview, 2024. poppenai-o--revviie https://openai. http:////w//ooooooooooooooo/ooaa /.
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. Lims know more than they show: On the intrinsic representation of LLM hallucinations. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL httpps://pppneeee Set//oorumiioKKRRXEEEnna
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from 1lms? objectives for defending against extraction attacks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: /ttp:///e///eeeee//////eee.//eee net 1e
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 2463-243222222 2473, Hong Kong, China, November A 2019. Association for Computational Linguistics. doi: 10.18653/ 00.18653/v1/D19111 v1/D19-1250. URL https://aclanthologg. wwwphhhooooooooo///-//222 org/D19-1250.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Miriam Rateike, Celia Cintas, John Wamburu, Tanya Akumu, and Speakman, Weakly supervised detection of hallucinations in LLM activations. In Socially Responsible Language wwt. net/foorm?iddz2gddoll
Modelling Research, 2023. URL
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering ilama via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15504-15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1 //2////////224aclllooo 828. URL https: / hwwph/oloollttoooo.oooooooo org/2024.4. acl-long. 828/.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, 2020.
Juan Diego Rodriguez, Wenxuan Ding, Katrin Erk, and Greg Durrett. Rankalign: A ranking view of the generator-validaoor gap in large language models. arXiv preprint drXiv:2504.1138) 2025.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric questions challenge dense retrievers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. pp. 6138-6148. Association for Computational Linguistics, 2021. doi: 10.18653/ /V1 /213//1122222.22 1.0D1...LLP-MMIII.CM 496. URL https://doi... http////...rr////////////////// 18653/v1/2021 emnlp -main.490 496.
Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. Distinguishing ignorance from error in Ilm hallucinations. ar Xiv preprint ar XrX::22410.22071, 2024.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172-118, 2023.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
14

---

Published as a conference paper at COLM 2025
Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar. On early detection of hallu- cinations in factual question answering. In Ricardo Baeza-Yates and Francesco Bonchi (eds.), Proceedings of the 30th ACM SIGKDD Conference Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pp. 2721 -2332. ACM, 2024. doi: 011145/363728..6111 017752883677999 3671796.
Knowledgge Discovery and Data
URL hww.h:ttp:////etmm //dos.//////.. 10./10.111 1055/0.1115550 01t.1111///177888777777777 3671796.
Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 14379-14391, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 24. 14.findings-acl.844 URL https://aclanttholog/ wwwp:thhhooooooooo/eogeeoeeeeeee 2222 madings-acl.88.
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in llms. arXiv preprint ar XrXoo:2502.06233, 2025.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arv preprint arXiv:2488.00138 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 5433-5442, Association for Computational Linguistics, 2023. doi: 10.18653 /V1/ DD23.EMNIP--MAN...
URL /tt.httss://oii.rrgg//
18653/v1/2023 emm.p-main.nmm.mmmmm 330.
Rebecca Treiman, Charles Clifton Jr, Antje Meyer, and Lee H Wurm. Language compre- 2003.
hension and production. Handbook of psychology, pp.
Eduard Tulchinskii, Laida Kushnareva, Kristian Kuznetsov, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, and Serguei Barannikov. Listening to the wise few: Select-and-copy attention heads for multiple-choice qa. arXiv preprint arXiv:2410.02343 2024.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't t always say what they think: Unfaithful explanations in sfain-of-thougee prompting. Advances in Neural Information Processing Systems, 36.74952-74965, 2023.
Denny Vrandečić and Markus Krötzsch. Wikidata: free collaborative knowledgebase. Commun. ACM, 57(10):78-88 85, sep 2014. ISSN 0001-0782. doi: (0.1145/262944 URL
https://doi. http://oo..rr///..1/ 10. 1145/2629489
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- met/forum?id=1PP
sentations, 2023. httphttts////peereee
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. ar preprint ar Xrx:::2411 .04368, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint ar arXiv:2412.15115 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. Xiv preprint ar 0x00:2505500388, 2025.
15

---

Published as a conference paper at COLM 2025
Gal Yona, Roee Aharoni, and Mor Geva. Narrowing the knowledge evaluation gap: Open- domain question answering with multi-granularis answers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computationnll Linguistics (Volumee 1: Long Papers), pp. 0737-6751, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v /v /2024.ac.-.ong. 365. URL https: /tcp:////llaaahhlloggooo or2/2024. acl acl-long.366 365/.
Gal Yona, Or Honovich, Omer Levy, and Roee Aharoni. Keep guessing? when considering inference scaling, mind the baselines. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 5979- 5991, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7, doi: 10. 10.18653/v v2222/5/2222222/// htt.//22......nss111.. findings- URL https: //aclanthologyyorg/////
findings-naacl. 332/.
Mert Yüksekgönüil, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint- satisfaction lens on factual errors of language models. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, nett
2024. URL https:
net wtttt/ormmmiideeeee dogoum?md:ggfVVAffP..
Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large language models in truthful space. In Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 5908-8949 Association for Computational Linguistics, 2024. doi: URL https://don org/10. 00011....666555 /653/v112222 acl- 1al1-long.44 483.
Eric Zhao, Pranjal Awasthi, and Sreenivas Gollapudi.
Sample, scrutinize and scale: Effective inference-time search by scaling verification. In Forty-second International Conference on Machine Learning, 2025. URL https: /ttp////eeeeeeeeeeeeeeeeeeeeeeee wtwpeeewwwwwweeeeeeeeeeeeee net/forum?id=wl.. 4wiE5.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang, Can we edit factual knowledge by in-context learning? Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 1862-4876. Association for Computational Linguistics, 2023. doi: 10.18653/VI https://doi, org/ 1etpg/....16.... 18653/v1 222222023322222 htt.//2..m..mmmmi.......
URL BEN.EEMNPP-MAA
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 15686-15702. Association for Computational Linguistics, 2023. doi: 10.18653/V /V URL https://dol org/ /0. 18653 /v1 1 22223 emnlp- waallppmminn.. 971.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach ai transparency. arXiv preprint xix::23310001455, 2023.
A Appendix
A.1 Data Creation Process
As discussed in $3.1, we build on EntityQuestions (Sciavolino et al., 2021), which provides triplets from Wikidata (Vrandečić & Krötzsch, 2014) that have already been converted into QA pairs. We categorize relations based on two criteria: (1) they should be difficult to guess, and (2) they should have a single, unambiguous answer, making them easier to grade. This categorization is presented in Table 2.
16

---

Published as a conference paper at COLM 2025
We consider question easy to guess when the object's entity type has relatively small set of unique values, and when the same value can apply to multiple subjects for a given relation, making it possible to guess a more prevalent value with reasonable success. For example, guessing a capital city is easier than guessing a spouse since there are only about 200 possible capitals, compared to billions of potential individuals. Similarly, professions can be easier to guess because certain professions are more common, whereas each spouse relationship is unique to single individual.
We define a relation as well-defined if it has a single, specific answer. In contrast, some relations allow multiple levels of granularity, making them anbiguous. For instance, when specifying a location, responses may vary in detail (e.g,, city, state, or country), making it less well-defined
We use P26 (spouse), P176 (manufacturer), P264 (record label), and P50 (author). While P40 (child) could also be included, answering it often relies on knowledge closely related to P26 (spouse). Given the computational expense of our experiments, we decided to exclude it to manage resources more effectively.
Test and Dev Sets. We use the test split of Enti tyQuestions for each of our four relations. We filter out questions with more than one gold answer and questions that contain the gold answer. We also deduplicated questions that appeared more than once. We then sample 500 questions from each of our four relations. Afterwards, we generate the greedy answer and sample additional 1,000 answers with a temperature of To this end we prompt the model with an instruction to answer the question, as described in $A.2. Next, we label each of the sampled answers using the LLM judge as described in SA.3. Some answers get labeling errors (see $A.3) in which case we filter the whole question out. We considered filtering the answers but ultimately chose to filter entire questions instead, as answer-level filtering could introduce noise into our evaluation. Our sampled answers are intended to approximate the full set of plausible responses, and we aim to utilize this entire set when estimating knowledge. Finally, we reserve 10% of the remaining questions for a developments set. Full statistics can be found in Table 3.
Train set. We use the train split of EntityQuestions for each of our four relations and apply the same filtering steps described above (for the test and dev sets). To ensure that there is no knowledge leakage between the train and test sets we also apply the following filtering steps: (1) Filter questions that appear in the test set. (2) For relations where the subject and object have the same entity type, e.g., P26 ("married to"") we ensure that the train subject does not appear as object in the test and vice versa.
We focus on questions for which the greedy answer is correct (see $A.4). To do this, we first predict greedy answers for all examples and retain only those with an exact match to the gold answer. Using exact match allows us to avoid running the judge across the full test set, which is large. This approach is justified as our goal is to find a sufficient number of examples where the greedy answer is correct, not to exhaustively identify all such cases. For these selected examples, we treat the greedy answer as the positive case and sample 200 additional responses using a temperature of 2.7 We label the sampled answers using our LLM judge and discard questions for which no incorrect answers were generated. For the remaining questions, we randomly select one incorrect answer along with the correct greedy answer for our final dataset. From this set, we randomly sample 500 questions per relation, resulting in a total of 2,000 examples.
A.2 QA Prompts
This section describes the prompt that we used to sample answers from our evaluated LLMs. In designing our prompts, our goal was instruct the model to generate plausible answers without unnecessary descriptive words while keeping the instruction natural. We did not specifically optimize the prompt instructions. It is possible that carefully crafted
A higher temperature increases the likelihood of sampling incorrect answers, which can be difficult when the greedy response already correct.
17

---

Published as a conference paper at COLM 2025
Relation
Question Template
Hard to Guess?
Well Defined?
P176
Which company [X] produced by?
P264
What music label is [X] represented by?
P50
Who the author of [X]?
P26
Who is [X] married to?
P40
Who [X]ss child?
P106
What kind work does [X] do?
P112
Who founded [X]?
P127
Who owns [X]?
P131
Where [X] located?
P136
What type music does [X] play?
P159 Where the headquarter of [X]?
Which country is located in?
P170
Who was [X] created by?
Who performed [X]]
P19
Where was [X] born?
die?
P276
Where is [X] located?
P36
capital [X]?
P407 Which language was [X] written in?
What position does [X] play?
P495
Which country was [X] created in?
Where [X] educated?
P740
Where was [X] founded?
What [X] famous for?
Table 2: Overview of the relations, their corresponding question templates, and metadata about difficulty and entity type definition. "Hard to Guess" refers to questions where the possible answer's ' space is large. For instance, person names considered hard to guess, whereas professions are not, as there are relatively few professions, and the model can default to more common ones. "Well Defined" assesses whether the entity type and answer granularity are unambiguous. For example, location can refer to a city, country, or exact address, making it less well-defined. Similarly, ownership may refer to person or corporation, adding ambiguity. In contrast, capital city or company name are well-defined entity types with a clear level of granularity.
Llama-3-8B
Mistral-7B
Gemma-2-99
Test
Dev
Train
Test
Dev
Train
Test
Dev
Train
P26
445
50
500
400
45
500
425
48
500
P264
447
50
500
430
48
500
442
50
500
P176
430
48
500
412
46
500
421
47
500
P50
448
50
500
441
50
500
447
50
500
Total
1770
198
2000
1683
189
2000
1735
195
2000
Table 3: Dataset statistics across different models.
instruction could yield significantly better performance. Yet our goal is to sample plausible answers from the model. We iterated through several versions until we observed that the model consistently outputs entities of the correct type (e.g., a person for "married to"). Once we reached this point, we did not observe meaningful variance in performance. The following system and user prompts were used for Llama and Mistral.
System Prompt (Llama and Mistral):
Your job is to answer an entity-centric question.
You need to answer with the correct entity, without any additional information.
User Prompt (Llama and Mistral):
Here is the question. Simply reply with the correct entity. If you cannot answer for any reason, output None. But do try your best find the correct answer.
18

---

Published as a conference paper at COLM 2025
Question: {question}
Just return the answer, with no text around
Gemma does not support system prompt, and concatenating the system and user prompts from above did not work well enough. We ultimately used the following prompt.
User Prompt (Gemma):
Answer the following entity-centric question, reply with the correct entity without any additional information.
Question: {question}
Just return the answer, with around
A.3 LLM Judge
We select Qwen2.5 14B Instruct (Yang et al., 2024) as the grader for two reasons. First, it belongs to a different model family than the ones we evaluated. Second, it is larger. Larger models contain more factual knowledge, which can be useful for the judge task. In early experiments, we found that the performance both zero-shot and vanilla chain-of-thougle prompting was insufficient. Therefore, designed program-guidee chain-of thoughht prompt. Essentially, we construct a mini decision tree and prompt the LLM to follow specific steps until it reaches verdict. Our program also includes self-verification step, which may result in an error label. For very small number of questions we found an issue with the gold answer, so we also let the judge verify that the gold answer has correct entity type. We design the prompt per-relation with small adaptations to account for specifics of the entity type and the question format. Below an example of our prompt for P26 (" ("married to")):
will give you question about the spouse of a person (e.g., "Who is Umberto I of Italy married to?"), gold answer, and proposed answer. You need to compare the proposed answer to the gold answer and assign one of the possible grades using the steps below.
Possible grades are:
A: CORRECT
B: INCORRECT
C: WRONG GOLD
D: ERROR
Spelling errors, synonyms, abbreviations, hedging (e-g., is possible that") should not alter the grade the person referred in the proposed answer matches the gold answer.
The steps are:
Step 1: If the gold answer does not refer to person, output "C" and finish. Otherwise, proceed to Step 2.
Step If the proposed answer does not refer person, output "B" and finish. Otherwise, proceed to Step
Step 3: the proposed answer refers to the exact same person as the gold answer, output "A" and finish. Otherwise, proceed to Step 4.
nttps: htttps://hugiingaaace. wttpae//ecccoe/e/en..accmm
19

---

Published as a conference paper at COLM 2025
Step Double check that both answers reflect person and the proposed answer refers different person from the gold answer. If it does, output "B". Otherwise, output "D" and finish.
Question: {question}
Gold answer: [gold_answer)
Proposed answer: (answer)
Output your thinking steps. After that, finish your response with "Output"" and the letter (A C or D). Do not provide any explanations.
To save inference calls we observed that while the judge executes the program correctly and reaches the intended conclusion, it nonetheless produces incorrect output. Those cases could be addressed by applying simple heuristics that checked "step 4". For instance the output is "A" (correct), then step 4 should not contain the word "different", or if the output is "B" (wrong) then it should not contain "refer to the same entity". Eventually, this procedure lead to a very high labeling quality as demonstrate our human evaluation next.
the judge only when exact match is false. In certain instances,
Accuracy
F1
Precision
Recall
P26
99.96
98.16
96.39
100.00
P50
99.92
96.93
94.04
100.00
LLM
P176
99.76
95.16
90.76
100.00
P264
100.00
100.00
100.00
100.00
P26
99.12
32.66
100.00
19.52
P50
99.20
51.86
100.00
35.01
EM
P176
98.11
31.32
100.00
18.56
P264
99.42
56.72
100.00
39.58
Table 4: Judge Performance Comparison: LLM vs. Exact Match (EM). The score values represent average measured across all predictive models (M) per each relation.
Estimating Judge Quality. To validate the reliability of our findings, we perform a hu- man evaluation for the performance of our LLM judge. The judge model, denoted by J, determines whether a predicted answer a, for factual question q, provided by predictive model M is equivalent in meaning to a gold answer g. Authors of the paper manually annotated 1, 1,080 examples for correctness. Each example consists of the triplet (q, a, a.g), along with Vj, a predictive verdict made by J.
We categorize all evaluation examples into three distinct groups based on how closely matches g and the verdict provided by the judge:
Group (Exact Match): The predicted answer exactly matches the gold answer g. All these cases automatically labeled correct (true positives), and are not part of the 1080 examples we manually annotate.
The judge verdicts these answers as correct even though they do not exactly match the gold answer
Group (Judge-Positive, Non-exact match):
Group 3 (Judge-Negative Non-exact match): The judge verdicts these answers as incorrect, and they do not exactly match the gold answer g.
Due to significant class imbalance with incorrect predictions by M vastly outnumbering correct ones (approximately 60:1) - random sampling would yield insufficient represen- tations of examples with correct predicted answers. E.g., if we randomly annotate, 1000
We normalization process on before comparing it g.
20

---

Published as a conference paper at COLM 2025
examples, we likely get ~16 correct answers. To address this, we employed the following sampling approach:
All examples from Group (exact matches) were automatically considered true positives, and also not included in our annotated set of 1, 080 examples.
For Groups and 3, we sampled an equal number of cases, 540 from each group. This 540 examples are evenly distributed among the three predictive models and four relations (i.e., 45 examples per model and relation).
All selected examples were manually labeled for correctness. However, because our sam- pling strategy does not reflect the actual distribution of classes, we applied re-weighting step after annotation to accurately represent the true proportions of Groups 2 and in the complete dataset. Table illustrates the sampling and re-weighting clearly for relation P26:
Group
Sampled Size per relation
Actual Dataset Size
Est. correct
Est. incorrect
1 (Exact Match)
669
669
669
2 (Judee-osiiive)
135
2887
2757.1
129.9
3 (udge-negative)
135
311324
311324
Table 5: An example of sampling and re-weighting approach for judge quality evaluation for relation P26.
Consider the relation P26 as an example (Table 5). Our dataset has 669 examples in Group 1 (exact matches), 2887 in Group (judge-positive, non-exact match), and 311324 in Group 3 jjudge-negative, non-exact match). All examples from Group 1 are true positives. To estimate true positives from Group we manually annotated a sampled subset of 135 examples, finding 129 (95.5%) correct. Thus, we estimate that 0.955 2887 2757.1 additional examples in Group are true positives. Combining these, we have an estimated total of 669 + 2757.1 3426.1 true positives. Following this approach, we estimate the amount false-positives and false negatives. We then compute precision, recall, accuracy, and Fl-score from these re-weighted counts for each relation and report these scores.
We present the results in Table Notably, the judge achieves very high accuracy of more than 99%. This is expected most cases are straightforward in nature, reflected by the high accuracy of the exact match alternative. Nevertheless, we look into the non-straightforward cases, where determining correctness is less obvious. To quantify the benefits of LLM- based approach, we compare its precision and recall to the widely-used exact-match metric. Exact-match has a precision of 100% by definition, but it may suffer from low recall when classifies paraphrases of the correct answer as incorrect. As seen in Table 4, our LLM judge successfully identifies many correct answers missed by the EM judge, achieving a notably higher recall while maintaining a very high precision. This improvement can be attributed to multiple valid formulations of correct answer that EM fails to capture. Taken together, these result provide evidence of a high performance of our labeling mechanism and support the validity of our primary findings.
M knows the to
M dovsa't know
the ansiver
connect
Known and
Carnect
(C) Unizzown
and Cornect
is wrong
(8) Known and
Wrong
(D) Unkmown
and Wirengg
Table 6: All possible conditions for given question & and candidate answer In knowledge- aware probing we train exclusively on categories (A) and (B) to ensure that the model (M) knows that the correct answer is correct and the wrong answer is wrong.
We note that true recall is likely slightly below 100%. Yet our human evaluation strongly suggests that it is close to 100%.
21

---

Published as a conference paper at COLM 2025
A.4 Knowledge-aware Probe (on M's hidden states)
In this section, we provide an extended discussion of our intuition for training the probe on questions for which it mostly knows the answers. We primarily describe possible risks associated with alternative choices but did not empirically validate whether these risks manifest in our specific setup. This does not affect our conclusions since, as discussed in $3.2, our goal is to explore the existence of hidden knowledge, and demonstrating it with a single internal function is sufficient. Further investigation of this aspect could be an interesting direction for future work.
The main intuition is that we want ensure that we train mostly on questions for which we have high certainty that M knows the answer. Since we want to train the probe to distinguish between multiple answers to the same question, we want to ensure that when we train on (9. a) pair, M encodes the information about a's correctness. If M does not know the answer to 9, its representations are unlikely to contain useful discriminative information about the correctness of different answers, and thus our trained probe may be less effective.
To train a probing classifier for TM, we need a dataset of (g. a) pairs labeled for correctness. Prior work follows two main approaches to create such a dataset. Approach (i) is to pair each question & with its gold answer 4g for positive examples and using fubricated answers as negatives (Marks & Tegmark, 2024; Azaria & Mitchell, 2023; Su et al., 2024; Rateike et al., 2023; Li et al., 2023a). Since fabricated negatives are often unlikely according M, this risks the probe learning M's likelihood rather than correctness. To disentangle likelihood from truthfulness, we instead use plausible -mneerggeeer incorrect answers. Approach (ii) is prompting M to generate answers and labeling correct and incorrect responses as positive and negative, respectively (Zou et al., 2023; Orgad et al., 2025; Snyder et al., 2024; Yüksekgönüil et al., 2024). To illustrate the risk associated with it, note that for given question and a candidate answer a, we can examine two key aspects: (1) does M know the answer to g? and (2) is a correct answer to q? Table 6 categorizes the outcomes of all possible responses to these questions. Since there is considerable correlation between the correctness of the generated answer and the model's knowledge, in (ii) we are likely training the probe mostly on categories (A) and (D). This may train the probe to identify whether M knows an answer rather than assessing the correctness of specific answers, weakening its ability to distinguish between answers to the same question. Instead, we introduce knowledge--twar probing, focusing on categories (A) and (B). To approximate data from these categories, we make relaxation and assume that if M generates the correct answer via greedy decoding, it likely knows the answer. Thus, we focus exclusively on questions for which greedy decoding produces correct answer. We then use this (correct) greedy answer as a positive example (A). To obtain a negative example (B), we induce a hallucinated response from M, even though it likely knows the correct answer (Simhi et al., 2024), by sampling additional responses at high temperature until an incorrect answer is found.
A.5 Training the Probe (on M's hidden states)
As described in SA.1, we create training set of 2,000 questions, 500 from each relation, applying multiple filtering steps to ensure no factual knowledge overlaps with the test set. We then merge these datasets and use the resulting data to train our probe. In early experiments, we also trained a separate probe for each relation, but since the conclusions were similar, we opted for a simpler approach using a single probe. We also examined the effect of data size. We found that the probe's performance with 250 questions per relation (1,000 total) was very close to that with 500 (2,000 total), ensuring the probe is not under-trained. The probe is trained with logistic regression objective, receiving as input M's hidden state HM (q, a), obtained by encoding g and a, and classifying a as correct or incorrect. To represent a relative to q, we follow similar procedure to that described in SA.8.1, simulating a sequence where M generates a when prompted with q. In early experiments, we also tested the sequence from SA.8.2, where the model is prompted to verify a as the answer to 9, but the classifier's performance was similar in both cases. Finally, we use the probe's 'ss output probability representing the likelihood that is correct as
22

---

Published as a conference paper at COLM 2025
TM- We train one probe per layer and select the layer with the best performance the development set. We also present the per-layer performance in Figure 5. We observe that scores tend to improve and then mostly stabilize in the final two-thirds of the network, typically starting around layers 11-12 out of 32. This suggests that the upper layers encode higher-quality and more consistent representations compared to earlier layers. Interestingly, we also see that there is often tiny drop in the last layers, which are used during decoding. However, this drop is relatively small. Apart from that, did not observe any interesting insights from analyzing the different layers. However, it was not our focus during the research, so is possible that we missed some interesting insights there which could be explored in future work.
K Values Per-Layer Llama 88
K Values Per-Layer Mistral 7B
K Values Per-Layer Gemma 98
0.00
3
0.80
10
30
30
13
12
-.70
2.13
13
0.70
15
10
30
9
19
10
0.70
20
22
29
23
28
-660
28
0.03
25
28
0.5
78
78
78
Fib
P24
PX64
P276
86
F66
P24
(a)
(b)
(c)
Figure 5: Per-layer K scores for Llama-3-88 Baae--3-8B--nstree (a), Mistral-7B-Insteee (b) and Gemma-2 2- 9B-Instruct (c).
A.6 Evaluating Memorization in the Probe
As we discuss in $3.2 and SA.1, we ensure that the factual information present in the training set is not useful for classifying test examples by careful data curation. We now complement this by empirically verifying that the probing classifier's 's performance does not result from memorizing training examples. To this end, we compared the Probe's performance to baselines trained on alternative input representations.
We define f(g a) as the concatenation of a question q and its corresponding answer a. our probing setup, classifier trained hidden representation (q.a)) extracted from the LLM.
As baselines, trained classifiers using two alternative input representations:
1. TF-IDF features derived directly from the input text, denoted (f,
2. Embedding-based features obtained by mean-pooling the token embeddings from the LLM's input embedding layer, denoted as EMBED BERIDDMEA((((( (q.
For each of these representations, we trained a logistic regression classifier and two fully connected neural networks with varying depths. The idea is simple, if the training set contains facts that are useful for answering test questions, then we should see that leaning from these examples results in better than random performance on the test set.
This specific experiment was done with Qwen3-32B, to leverage the strongest embedding among all L.IMs.
23

---

Published as a conference paper at COLM 2025
Table 7 summarizes the accuracy results. Classifiers trained on both TF-IDF and embedding- based representations perform around chance level on the test set. 12 If the train set would contain useful information, we would expect at least minor gains on the test set. In contrast, the probe classifier, trained on internal LLM representations, significantly outperforms these baselines. This strongly suggests that the probe's performance is driven by genuine internal knowledge representations rather than memorization of textual content from the training examples.
Input Representation
Model
Train Acc (%)
Test Acc (%)
Random
50.0
TFIDF(f(g,a))
Logistic Regression
95.6
50.5
IFIDF(((q.a)
MLP (256, 256)
99.9
50.7
a))
MLP (512, 512, 256, 128)
99.9
50.2
EMBED EMBED_MEAN (qq.a))
Logistic Regression
98.8
49.8
EMBED_MEAN
MLP (256, 256) MLP (512, 512, 256, 128)
99.9
49.1
EMBED MEAN(((q.a))
99.9
50.9
h()(q.a))
Logistic Regression
99.9
64.0
Table 7: Accuracy of classifiers trained on different representations of the input (g, a) pairs.
A.7 Statistical Significance
In Figure 2, we report the outcomes of statistical significance tests comparing the K and K values obtained using our internal scoring method against the best-performing external method for each model-relation combination. To this end, we shuffle all examples the relevant test set, split them into 50 approximately equal-sized subsets, and compute K and K for each subset. We then apply a paired-sample t-test with p < 0.05. We also report statistical significance in Table 1, where we compare each answer selection method to the greedy decoding baseline. The same procedure is applied, but we use 200 bins since we mix all four relations together.
A.8 External Scoring
A.8.1 P(alq) and P Promm (alq)
To compute P(alq) = II=-1 P(a; 9, q,),,, as well as its length-normalized variant Poom (a (a|q)
= P(T) 9, q.a=)))" exp E,_nlog log P(a; q.a=;) we need to obtain the token-level probabilities of the answer conditioned on the question: {P(a; 9, i- 1, n}. For each answer a, we use the relevant prompt from SA.2 as g and construct sequence S that simulates the generation of a given 9. Instead of simply concatenating to 9, we ensure that all special tokens match their expected form as if the model had actually generated following 9. We then perform a forward pass through the model with S as input and use the resulting logits to compute the token-level likelihoods a. This procedure is particularly useful for scoring the gold answer in cases where the model did not generate it at all.
A.8.2 P(True)
We use the system and user prompts from below for Llama and Mistral, and concatenate them for Gemma. We do forward pass with the model on the resulted input and then use the logit of the next token to compute the likelihood of " AA", Rather than applying the softmax over the entire vocabulary, compute it only over "A" and "B".
12To make interpretation easier, we balance the test labels by sampling correct and one incorrect answer per question, resulting in a 50% random baseline (the training set already balanced).
24

---

Published as a conference paper at COLM 2025
System Prompt:
Your job is to evaluate if a proposed answer to an entity-centric question is correct.
User Prompt:
Here is the question and the proposed answer.
Question: [question]
Proposed Answer: {answer)
Is the proposed answer:
A: CORRECT
B: INCORRECT
Just return the letters "A" or "B", with no text around
A.9 Extended Definition of Knowledge
We now discuss the full definition of knowledge, which introduces the sanity-check expres- sion ((q: [q:Su] that handles "implausible" answer candidates (ones that are not in A())). We omitted those details from the main definition (Definition 1) in order to make it easier to follow.
Definition 3 (Extension to Knowledge of a Model w.r.t Scoring Method).
As in Definition 1, we consider a model M, and fact represented as a (subject, relation, object) triplet (,, e.g., ("France", capital, "Paris"). We also denote the vocabulary of the tokenizer used by M with V.
Then, in addition Q(,,r), A(o) and A(o), we define:
Am: The (infinite) set of all possible answers that M can produce, produce,formallyy defined as V Le., it is equal to the set of all finite sequences that can be formed using tokens from It may include phrases such as "Paris", "Hello", "#% elc.
We can then define the scoring function to be SM Q(s,r) x AM + R instead SM Q(s,r) x A(o) + R.
Next, we define the sanity-check indicator ((q:Sx) that ensures that any plausible answer is scored above any non non-plausible one:
Sm) I(VaE A(o), a Am ((o) Sm(q,a)
Sx)(q,
We then adjust the definition of the per-question score (s, r, Sm) to consider the sanity check:
Kg (,(,,r,,;Sm) = Sm) [(ss
Σ
I(Su(q,a) > Sx(q,ā))
(5)
(2,)))((srr.)
We note that ((q: Sm) should be consistently 1 for reasonable scoring methods, but can also verify that by creating a focused challenge set.
A.10 Hidden Knowledge in Larger Models
25

---

Published as a conference paper at COLM 2025
0.8
-15%
+14%
+5%
-106
0.6
To make findings reliable, put special emphasis on sampling a large set of 1,000 answers per-question to approximate (o) (see $3.2), which made our exper- iments computationally intensive. We use 7-9B models, which balance between model size and our compute budget. To provide initial evidence of hidden knowl- edge in larger models we run a smaller scale experiment with Qwen3-32B (Yang et al., 2025), using 200 sampled answers per-question. Figure 6 presents the results. For all relations, both the K and K" measures are consis- tently higher when computed using the internal scoring function compared to any external one, suggesting that the hidden knowledge phenomenon persists even when scaling the number of parameters by a factor of ~-4 The average gap in K for Qwen is 12.5%, compared to 14%, 48%, and 57% for Llama, Mistral, and Gemma, respec- tively. Given that the gap for the larger Qwen model remains relatively close to that of the much smaller Llama, it remains an open question whether further scaling alone can be sufficient to reduce this gap, and which other factors affect the existence of it. Importantly, since 7-32B remains a commonly used capacity range, we argue that additional mitigation strategies beyond mere scaling up are needed to better expose model knowledge.
0.4
02
P26
P264
F178
P50
0.3
0.2
+13%
+18%
+47%
P26
1264
F176
1750
((aig)
P.True)
Probe
Figure 6: Qwen3-32B.
Avg.
K and K* for
A.11 Kg vs AUC-ROC
We note that Kg is conceptually related the area under the ROC curve (AUC-ROC), as interpretation of AUC-ROC is the probability that randomly chosen positive instance is ranked higher than a randomly chosen negative one (Hanley & McNeil, 1982; Fawcett, 2006). One difference is that AÚC-ROC averages over continuous thresholds, whereas our approach relies on direct pairwise comparisons, which also offers an intuitive interpretation of Kg as the fraction of correctly ranked answer pairs. More importantly, AUC-ROC is typically calculated across examples in a dataset, whereas Kg is computed separately for each question. This distinction is significant, as scoring methods performance can vary substantially when comparing answers to the same question, as opposed to scoring question-answer pairs independently (Taubenfeld al., 2025).
A.12 Choosing the LLMs For Our Study
We chose the following three popular open-weight instruction-tuned LLMs for our study; Beas---tB-
Llama-3-8B Instruct (Dubey et al., 2024), 13 Wistral-7B-Instreee (Jiang et al., 2023)14 and Gemma- Inst-lnst (Team et al., 2024)15 We used the largest size that could afford, as our experiments are compute heavy. We focus on instruct models as they are the ones that the users interact with, the question of hidden knowledge much more relevant them. As discussed in $A.10, we also perform a smaller scale experiment with Qwen3-32B (Yang et al., 2025)16 to provide evidence of hidden knowledge in larger capacity LLMs.
A.13 Analysis of K Values When Manually Adding The Gold Answer to A())
In Figure 7 we compare the K values between setup that uses only the answers that were sampled as Á(o) and our main setup (used in $4.1) where the gold is manually added to Ā(o). On average, in 64% of the cases we do not sample the gold answer, in which case adding the gold can change K.
13 14 httpp://wuuginneee.coomttrrrssrs In--nnrruct--a 15, http:::://ungnnggae...oo/o/gg/// 2-9-i- 2-9b-1t hhttps::/hugginee co/Quen/Qwen3-330
co/neta- 1laaa/ata- Instruct
26

---

Published as a conference paper at COLM 2025
Llama 3 8B
Mistral 7B
Gemma 2 9B
P(a/q)
0.8
0.6
46%
440%
+84%
+91%
9.%%
+90%
+25%
+23%
+26%
02
150
150
++148%
+162%
+164% +152%
+189% +248% +274% +163%
+104%
+101% +119% +112%
Probe
0.8
0.6
0.4
P26
P264
P176
150
126
P264
P176
P50
P26
P264
P176
P50
Sampled Only
Sampled + Gold
Figure 7: Comparison of average K values (see Equation (2)), under two conditions: without manually adding the gold (Sampled Only, left bars) and with manually adding the gold (Force Gold, right bars).
If we look at P(alq), we observe consistent increase in K scores. We then further analyze the nature of the examples that lead increase in K. In 97% of those cases, not only that the gold answer ac was not sampled, but there was no other correct answer sampled, and thus K was manually set to 0. Accordingly, it is rather expected that in such cases adding the gold will lead it to a "win", increasing K. Yet, Figure 3 shows us that we never observe an increase in K* for P(alq). In fact, in 7 out of 12 setups even decrease in K* In $4.2 we explain why K* results are expected for P(alq) q) and we now show that some increase in would expect.
K are also
When looking at Probe, not only it also shows a consistently increase in K scores, this increase is substantially higher than for P(alq). As discussed in $4.2, unlike P(alq), Probe shows consistent increase in K*
A.14 How K Affects Our Chances of Success in Inference Scaling?
We begin with a theoretical discussion on the relationship between our definition of knowl- edge and inference scaling. One interpretation of K(+;SM) the probability of ranking randomly chosen correct-vs-incorreed answers pair correctly. Let = Pra-M E A(o) be the probability to sample correct answer from M. Then, when drawing n i.i.d. samples, the probability that the top-ranked candidate is correct is:
Probability that all correct answers fail to out-rank all wrong
Probability that correct answer out-ranks all [n-1) wrong ones
P'1'1 p)"
((,
SM)]
Σ
1
1
K(s,
Probability to sample correct and (ı-i) wrong
Probability that at least one correct answer out-ranks all (ı-i) wrong ones
66)
27

---

Published as a conference paper at COLM 2025
MrraJjama--8S--nniiu
gemma---00-t
10
10-
Mera Limma-3 00 instruct
Mintral 713 11--ttrttuctt0... v0.3
gemmin 99-iit
307
33'
337
10'
33
300
337
16.
120
Figure 8: Statistics for the number of unique answers (top) and number of unique correct answers (bottom) per-questions. For very small number of questions we observe significantly high number of correct answers. This reflects cases where the model failed to provide a short-form answer and added additional diverse suffixes, e.g., "Helen. This answer relates to the Greek mythology context.". ". Those cases are rare and we can either filter-out such cases with length threshold, or leave them and require the scoring functions to score them higher than wrong ones. Both options are legitimate and we validated that they do not affect our findings. For simplicity we report results without filtering.
evident that as K increases, so does Pr[Success K)). Demonstrating a practical useful- ness of our definition of knowledge: the more knowledge M has according to our definition, the higher is its chance to benefit from inference scaling.
That said, it is also important to note that two models may exhibit identical inference scaling behavior despite significant differences K. While, in principle, higher K and K* scores increase the chances of success under inference scaling, they do not guarantee it. In fact, there are cases where a model M1 has a higher knowledge gap than another model M2, yet M2 benefits more from inference scaling. The key intuition is that K scores reward robustness across multiple correct phrasings, whereas inference scaling only requires the model to rank one correct answer above all incorrect ones.
To illustrate this, consider the following hypothetical example. Both M1 and M2 have the same five candidate answers: two correct (2), and three incorrect (w1, w2, w3). Suppose M1 ranks them as (c1, w1, w2, w3,c2, and M2 as (w1,cl, c2, w2, w3), where the leftmost is the top ranked answer. Under inference scaling, where the top-ranked answer selected, M1 returns correct answer (c1) while M2 returns wrong one (w1). However, M2 achieves a higher K score. For M1 K 3/10 due to > w2, cl > w2 and cl > w3, while for M2 K 4/10 due to cl > w2, cl w3, c2 > w2 and c2 > w². This illustrates how differences in scores do not always predict differences in inference scaling outcomes. Same holds for K*. Suppose we now have model M3 that ranks the answers as (c1, w1, w2, w3). Both M1 and M3 will succeed under inference scaling, but only MB achieves K* all correct answers are ranked above all incorrect ones), while M1 gets K* 0 (as c2 is ranked below several incorrect answers). This illustrates that even K may not fully explain inference scaling performance.
(as
28

---

Published as a conference paper at COLM 2025
We believe that choice to include rankings of cifferent phrasings of the correct answer in our metric is important, even if it is not required for inference scaling. For example, model might be used to verify responses, either in response to user request to generate a reward score, in which case recognizing alternative phrasings is crucial.
A.15 Alternatives to QA Format
As discuss in $2.1, choose to work with QA format but other alternatives exist. Specifically, we could examine the scores that M assigns to claims reflecting the relevant fact. For instance, if our fact is ("Empire State Building", location, "NYC"), then, instead of scoring alternative answers to a related question, such as "Where is the Empire State Building located?", we could score different claims, measuring whether correct claims (e.g., "The Empire State Building is located in NYC") score higher than contradicting claims (e.g., "The Empire State Building is located in Paris"). However, ensure meaningful comparison, claims must follow a fixed template and differ only in factual content. The QA format naturally facilitates this by providing an environment where this factor is easily controlled since we can keep the question fixed and compare alternative answers.
A.16 Discussion On Possible Reasons And Mitigations For Hidden Knowledge
An interesting direction for future work understanding the reasons for hidden knowledge. One hypothesis is that post-training sharpens the probability distribution, which may concentrate the probability mass on a specific answer. This may lead to a condition where correct answers that were plausible before post-training are assigned with low likelihood by the model even though the model encodes knowledge on their correctness. Another possible reason is that post-training may emphasize style, leading the model to choose fluent but less factual answers. A possible mitigation could be to encourage robustness to different phrasings by exposing the model to multiple correct answers during training, which would require modifying both the dataset labels and the optimization objective to jointly learn from them. Another solution may lie in adapting the model during test time using decoding methods that use internal signals to help surface known facts, e.g., Rimsky et al. (2024). Lastly, another possible direction to address the issue during the reinforcement learning phase by designing reward signals that prioritize factuality over style. This is challenging goal, as we do not want to compromise the stylistic fluency that makes these models so effective.
B Limitations
A key limitation of our framework is its high computational cost. For each question and model, we must generate many candidate answers, label each candidate using LLM judge, and then score them using multiple methods. This is also the reason we focused on 7-9B models and did not experiment with models of larger capacities. Another limitation (discussed in $5) is that our definition of knowledge does not consider knowledge of related facts. For example, knowing that Paris is the capital of France may also require knowing that Paris is located in France. We choose to leave this aspect out of scope and hope future work will explore corresponding extensions to our definition. Finally, a limitation of the K* metric, which reflects full knowledge, is its sensitivity to labeling errors: an incorrect label assigned to a candidate answer can flip its score from 0 to 1 or vice versa. To address this issue, we put significant effort into ensuring high labeling quality by using LLM judge, carefully designing its prompt, and performing extensive human evaluations to confirm its accuracy (see $A.3). This approach shows clear improvements over the commonly adopted exact-match method. Additionally, we introduce the contimuou metric K, which is less sensitive to labeling errors, as our main evaluation measure.
29

---

Published as a conference paper at COLM 2025
C Acknowledgements
This research is collaboration between the Technion and Google Research. It was sup- ported in part by grant from Google. Part of this research was also supported by Open Philanthropy and an Azrieli Foundation Early Career Faculty Fellowship and part of it was funded by the European Union (ERC, Control-LM,1001654000 expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. Hadas Orgad was supported by the Apple AIML PhD fellowship.
Views and opinions
30


Reviews:
[{"review_id": "1vWQNYfCOl", "reviewer": "Reviewer_959U", "summary": "The paper introduces a formal and measurable notion of hidden factual knowledge in large language models (LLMs) and shows that internal signals in the models’ hidden states rank correct answers better than common probability-based external signal, revealing a knowledge gap and cases where the model “knows” an answer. The experiments are conducted on three LLMs (Llama-3-8B-Intruct, Mistral-7B-Instruct, Gemma-2-9B-Instruct) using specifically constructed closed-book QA dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **(A1)** Convincing methodological setup: authors provide precise definition distinguishing internal vs external knowledge and metrics to quantify them. They compare multiple external scores, using 1 000 diverse samples, an LLM judge with >99 % estimated accuracy, and statistical significance thresholds. \n- **(A2)** Strong empirical evidence: across all settings the probe outperforms every external scorer with significant margins. Hidden-knowledge gaps differ across models (e.g. 62% Gemma vs 14% Llama), but it is statistically significant. \n- **(A3)** Clear presentation and structure: the materials are presented in a very clear and structured way, and all experiments are accompanied by clear illustrations. \n- **(A4)** Practical takeaway: authors provides a way to rank answers based on probes and demonstrates a 12% average accuracy boost in closed-book QA. However, there is some arguable moments (see R2)", "reasons_to_reject": "- **(R1)** Limited scale: only 7B-9B models are tested, so conclusions may not hold for frontier 70B-175B models. It would made results more convincing if at least one 70B model will be included.\n- **(R2)** Limited practical availability: based on my understanding, authors provide results on increasing test-time compute only on their own dataset. I believe, it would be more interesting to see other (probably even out-of-domain) datasets and corresponding results. \n- **(R3)** Insufficient analysis of internal knowledge probes: authors didn't analyse the constructed probe classifiers to inspect its reliability – i.e. metrics on training set, the visualisation of activations manifold and the analysis of metrics per layer.", "questions_to_authors": "Questions based on reasons to reject:\n- **(Q1)**: Did you attempt even a small-sample study on 30B or bigger models? If not, do you think the results will be qualitatively different?\n- **(Q2)**: Do you think that your results will be applicable to other datasets for increasing test-time compute?\n- **(Q3)**: Could you provide more results, showing the reliability of constructed linear classifier like in (R3)?\n\nOther question:\n- **(Q4)** Have you experimented with alternative decoding (e.g., nucleus sampling, temperature sweep, MCTS) to see if “never-generated” but known answers can be surfaced?\n- **(Q5)** I don't completely understand, why the increase for Gemma is so small compared to other models in Table 1, while Gemma has the biggest gap between internal and external knowledge. Could you please comment on this?", "extra_scores": {}}, {"review_id": "AUhjUg0Vdx", "reviewer": "Reviewer_6s8b", "summary": "This paper studies an important question on if LLM encodes more knowledge (internal knowledge) than it outputs. They propose a definition of internal knowledge and external knowledge. Concretely, to scoring internal knowledge, they train a linear classifier to predict correctness from the model’s hidden representations of question-answer pairs. For external knowledge, it’s based on the model’s token-level probabilities. Experimental results show that LLMs consistently encode more factual knowledge internally than externally (around 40%) based on the QA dataset. \n\nThough, the paper studies an very important question, the original research question has been significantly simplified during the research formation, and have too much assumption and hypothesis (such as internal and external knowledge definition, and using QA to validate this), which still raise questions on if the findings from this simplified setting can used for the original question.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper studies an important research question, and provides definitions to some concepts to quantify the hidden knowledge in LLMs. \n* The paper conducts experiments and demonstrates important findings on gap between internal knowledge and external knowledge.", "reasons_to_reject": "* The original research question is significantly simplified, which raises questions on if the research findings are still applicable to the original question. For example, when training a scoring function, then a new knowledge will also be learnt during the training process, does the capability of the scoring function still represent the capability or knowledge of the original model? \n* The paper only conducts experiments on QA task, the generalizability of the finding to other problems is unknown.", "questions_to_authors": "* What is the implication for practical real-world applications? \n* Will we have the same findings when using larger size LLMs?", "extra_scores": {}}, {"review_id": "mN0CGdkyP6", "reviewer": "Reviewer_T7HJ", "summary": "The paper explores and provides evidence for an interesting phenomenon in LLMs - hidden knowledge: the encoding of more factual knowledge in the model parameters than what expressed in their outputs. The author(s) build from explicit definitions such as that of knowledge (external and internal) and used them to define experiments aimed at testing for the presence of hidden knowledge. Using a QA task, they contrast the factual knowledge emerging from model outputs with that emerging from hidden states (as extracted using trained probes), demonstrating that through the latter the model can show awareness of the correct answer in spite of never generating it through large-scale sampling.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very well-written and well-structured. The research questions are well explained, with clear motivation for the experiments and the methodology adopted. \n- The paper addresses a very interesting phenomenon (hidden knowledge), connected to an important topic for the NLP community (factual knowledge in LLMs). The results provide insights on how LLMs behave and can inspire new research on better leveraging the knowledge in LLMs. \n- The paper provides convincing evidence for the existence of hidden knowledge in LLMs, as well as a formal framework to investigate it (in terms of definitions and formulas).", "reasons_to_reject": "- The paper could benefit from a discussion - even if speculative in nature -  on what the results could entail for our understanding of LLMs, why this phenomenon of hidden knowledge would arise in the first place or how we could push LLMs to better use their knowledge. I encourage the author(s) to expand a bit on this in the camera-ready version of the paper.", "questions_to_authors": "\"We train probes for all layers and choose the best layer based on a development set.\" -> What are your observations on what hidden layers tend to encode better factual knowledge and what encode more hidden knowledge? Were there any trends? There might be some interesting insights there to include in the paper. \n\nHidden knowledge is an interesting but puzzling phenomenon (since model outputs are based on hidden states, why is certain knowledge not leveraged for the final task?). I'm curious if you have any speculative hypotheses in terms of why this phenomenon would arise in LLMs and what could be done to align more external knowledge to the internal one that remains hidden.", "extra_scores": {}}, {"review_id": "fZPCq3N5Gv", "reviewer": "Reviewer_LjuE", "summary": "The authors investigate the phenomenon of hidden knowledge within LLMs. Namely, they introduce a formal definition of \"hidden knowledge\" and provide a formal framework based on two distinct types of scoring functions: \"external scoring\" (using observable token-level probabilities) and \"internal scoring'\" (based on intermediate computations within the model). The paper applies this framework to evaluate three LLMs: LLaMA-3-8B, Mistral-7B, and Gemma-2-9B, demonstrating that using probes from their intermediate layers to score the answers consistently outperforms more traditional \"external scoring\" functions. The authors highlight significant gaps between the models' potential and their generative performance and perform a case study. Finally, they utilize these probes to get better results in Question Answering task for the same models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors address an important problem of finding hidden knowledge inside LLMs and extracting it. This may be potentially helpful both to understand these models better and to improve their performance in the future.\n- The formalism of the framework that the authors introduce looks reasonable.\n- The design of data collection and sampling \"plausible answers\" also seems reasonable and well thought-out.\n- The authors provide an insightful analogy between hidden knowledge in LLMs and a psycholinguistic phenomenon in humans, where people may understand words yet fail to retrieve them.", "reasons_to_reject": "- **The paper lacks clarity in its presentation**. In particular, certain crucial details are scattered across several little appendices (A.5, A.6, A.7, A.10), which hinders the clarity of the paper. The most importantly, I suggest to integrate the general description of the probe training process (Appendix A.5) into the main text, and explain it better (refer to the specific questions for the authors below).\n- The LLMs used in the paper are somewhat outdated. **The paper would benefit from adding at least one more recent model into the experiments**.\n- The topic of hidden knowledge has already been explored in previous works from different perspectives. In particular, in the paper Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA by Tulchinskii et al., it was found that answering MCQA tasks based on query-key interactions from specific attention heads of the LLaMA model outperforms the model itself on many MCQA tasks. In that work, they concluded that some parts of the model 'know' the answer, but this information sometimes is lost before it reaches the output layer. Additionally, there is a body of earlier work on extracting topological features from specific heads and layers of transformer models (Topological Data Analysis for Speech Processing by Tulchinskii et al., Acceptability Judgements via Examining the Topology of Attention Maps by Cherniavskii et al., Artificial Text Detection via Examining the Topology of Attention Maps by Kushnareva et al). In these works, topological features extracted from specific heads and layers also outperform the model itself in a number of task, serving as a sort of whimsical internal probe as well. **Summing up, the authors should better clarify the novelty of their work in light of other studies**. For example, you could emphasize a formal definition of hidden knowledge and a new method for identifying it as a clearer novelty of this particular work.\n\n**UPD**: The authors have committed to improving the presentation, satisfactorily addressed my questions, and provided additional examples illustrating their point in the rebuttal. As a result, I have increased my score.", "questions_to_authors": "- The design of the experiment with probes (for internal scoring) is not fully explained. A major point of ambiguity is the selection of the development set for identifying the optimal layer. At line 222 you write: _\"We train probes for all layers and choose the best layer based on a development set\"_. But what is development set exactly?\n- You choose local 14B model as a Judge. Did you consider using a better model via API, such as GPT-4-O?\n- What does \"oracle\" refer to in Table 1?\n- Are there additional case study examples beyond the one presented in Figure 4? It would be helpful to see more.\n- **What is the accuracy of scoring on probes from other layers** of LLaMA, Mistral, and Gemma, aside from the best layer?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates whether an LLM encodes moore knowledge internally (in embedding) than its generated tokens. All reviewers find the problem interesting and important. The technique proposed in the paper to estimate internal and external knowledge (using trained linear classifier based on latent embeddings, and using model's token level probability) are reasonable and valid. The paper's experimental results show that LLM encoder more factual knowledge internally than externally. Most reviewers have a major concern regarding the evaluated model size. The author addressed the concern by adding QWen 32B result. I highly encourage authors to add those in the revision. 

The pape could be improved if the authors add: 1) a discussion on tasks other than QA; 2) probing other layers in LLaMA, Mistral, Gemma etc; 3) reliability analysis.
Decision: Accept