HARP: Hallucination Detection via Reasoning Subspace Projection
Authors: Junjie Hu
Gang Tu
Cheng Shengyu
JinXin Li
Jinting Wang
Rui Chen
Zhilong Zhou
Dongbo Shan
Venue: ICLR 2026 Poster (2026)
Keywords: Hallucination detection
Subspace
Projection
SVD
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Hallucinations in Large Language Models (LLMs) pose a major barrier to their reliable use in critical decision-making. Although existing hallucination detection methods have improved accuracy, they still struggle with disentangling semantic and reasoning information and maintaining robustness. To address these challenges, we propose HARP (Hallucination detection via reasoning subspace projection), a novel hallucination detection framework. HARP establishes that the hidden state space of LLMs can be decomposed into a direct sum of a semantic subspace and a reasoning subspace, where the former encodes linguistic expression and the latter captures internal reasoning processes. Moreover, we demonstrate that the Unembedding layer can disentangle these subspaces, and by applying Singular Value Decomposition (SVD) to its parameters, the basis vectors spanning the semantic and reasoning subspaces are obtained.
Finally, HARP projects hidden states onto the basis vectors of the reasoning subspace, and the resulting projections are then used as input features for hallucination detection in LLMs. By using these projections, HARP reduces the dimension of the feature to approximately 5% of the original, filters out most noise, and achieves enhanced robustness. Experiments across multiple datasets show that HARP achieves state-of-the-art hallucination detection performance; in particular, it achieves an AUROC of 92.8% on TriviaQA, outperforming the previous best method by 7.5%.

Published as a conference paper at ICLR 2026
HARP: HALLUCINATION DETECTION VIA REASON- ING SUBSPACE PROJECTION
Junjie Hu
Gang Tu"
ShengYu Cheng
Jinxin Li
Jinting Wang
Rui Chen
Zhilong Zhou
Dongho Shan
School of Computer Science and Technology Huazhong University of Science and Technology Wuhan, China
(hujunjie, tugang) (hagan @hust. @hust.edu.uu............ edu.on cn
ABSTRACT
Hallucinations in Large Language Models (LLMs) pose major barrier to their reliable use critical decision-making. Although existing hallucination detec- tion methods have improved accuracy, they still struggle with disentangling se- mantic and reasoning information and maintaining robustness. To address these challenges, we propose HARP (HAllucination detection via Reasoning subspace Projection), a novel hallucination detection framework. HARP establishes that the hidden state space of LLMs be decomposed into direct sum of mantic subspace and a reasoning subspace, where the former encodes linguistic expression and the latter captures internal reasoning processes. Moreover, we demonstrate that the Unembedding layer can disentangle these subspaces, and by applying Singular Value Decomposition (SVD) its parameters, the basis vec- tors spanning the semantic and reasoning subspaces are obtained. Finally, HARP projects hidden states onto the basis vectors of the reasoning subspace, and the resulting projections are then used as input features for hallucination detection in LLMs. By using these projections, HARP reduces the dimension of the feature to approximately 5% of the original, filters out most noise, and achieves enhanced robustness. Experiments across multiple datasets show that HARP achieves state- of-the-art art hallucination detection performance; in particular, it achieves AU- ROC of 92.8% on TriviaQA, outperforming the previous best method by 7.5%.
INTRODUCTION
Large Language Models (LLMs) have recently demonstrated remarkable generative capabilities and broad applicability across various natural language processing tasks (Yang et al.) 2024 Grattafiori al. 2024 Minaee et al. 2024). However, hallucinations-ii.e model-generated information in-
consistent with objective facts remain a major obstacle to their deployment in critical decision-
making scenarios (Ji et al. 2023 Huang et al., 2025). Consequently, efficiently and accurately
detecting hallucinations during LLMS generation has become a pressing challenge.
From a cognitive perspective, the hallucination behavior of LLMs to some extent similar to hu- man's "nonsense" behavior. When answering complex questions, humans typically follow a "Rea- soning + Expression" process: they first perform internal reasoning and then express part of the thought outcomes in language (Johnson-Laird] 1986). Therefore, although assessing the veracity of the answer challenging when based solely on linguistic output, can substantially improved by observing the complete reasoning process (Frank & Goodman 2012). By analogy, achieving high-precision hallucination detection LLMs requires placing greater emphasis on the reasoning information encoded within the hidden states, rather than primarily the semantic content of the outputs.
"Corresponding Author

---

Published as a conference paper at ICLR 2026
Trunsforn
Calculate
Hallucination
Manmih
Score
Where capital United
Autermante
capital
Washington
Accepe
Asemantte
dv
Konsoning
4001
-0.01
-0.01
-0.01
0
Decoder
40
copital
Shanghai
X Hallacimeeim
Figure Illustration of the proposed HARP framework for hallucination detection. HARP separates the reasoning information h Reasonin from the hidden state fy to compute token-level hal- lucination scores, with the maximum score taken as the hallucination score of the entire response.
Inspired by this cognitive insight, propose novel hallucination detection framework, HARP (HAllucination detection via Reasoning subspace Projection). Specifically, HARP decomposes the hidden state space into direct sum of the seman- tic subspace and the reasoning subspace. The se- mantic subspace captures linguistic information of the generated content, while the reasoning sub- space reveals the model's 's internal reasoning process. As illustrated in Figure 2 comparing humans and LLMs "Reasoning + Expression" behaviors reveals that LLMs discard reasoning information in the Un- embedding layer while compressing semantic infor- mation into generated tokens. This suggests that the Unembedding layer inherently distinguishes be- tween semantic and reasoning information. Based on this, we perform Singular Value Decomposi- tion (SVD) on the parameter matrix of the Unembedding layer to identify the basis vectors of the semantic subspace, which dominates token prediction, as well as those of the reasoning subspace, which is orthogonal to the semantic subspace.
Transfor
Aurwer
Transfom
10 Semandic
10
Semandic
Ressoning
Reasowing!
Deooer
Enteshiing
Queatin
Quostion
Figure 2: Comparison of the "Reasoning -> Expression" behavior between humans and LLMs
Finally, HARP projects hidden states onto the basis vectors of the reasoning subspace and uses the resulting projections as input features for hallucination detection in LLMs. Since the reasoning subspace basis vectors account for only about 5% of the hidden state dimension, the input features are highly concentrated in reasoning information while largely eliminating noise. This allows HARP to achieve strong robustness while maintaining high detection accuracy. The main contributions of this work are:
We establish that the hidden state space of LLMs can be decomposed into a direct sum structure composed of semantic subspace and reasoning subspace.
We verify that the Unembedding layer has the capability to distinguish between the mantic subspace and the reasoning subspace. Furthermore, by performing SVD on the parameters of the Unembedding layer, the basis vectors that span the semantic subspace and the reasoning subspace are identified.
We introduce novel approach that explicitly constructs input features by projecting hidden states onto the basis vectors of the reasoning subspace. This projection drastically reduces the feature dimensionality to about 5% of the original, suppresses most noise, and achieves highly accurate hallucination detection in LLMs.
RELATED WORK
Mechanistic interpretability of LLMs.. Research on mechanistic interpretability mainly focuses two aspects: model parameters and hidden states. For the former, several works analyze weight ma- trices uncover structural properties and interactions among modules. For instance, Merullo et al.
2

---

Published as a conference paper at ICLR 2026
2024) and Cheng al. (2024) employ SVD to characterize attention-head structures and investigate their roles in downstream tasks. However, such approaches remain largely at the structural level, of- fering limited semantic interpretabilityy To address this, recent work has shifted toward analyzing hidden states directly, probing the predictive relationship between intermediate representations and
model outputs (Gurnee et al. 2023
Lv al. 2024 Ju et al. 2024 He et al. 2024 Jin al. 2025).
The success of probing methods has motivated researchers to adopt simi- lar ideas in hallucination detection (Marks & Tegmark| 2023 Bürger et al. 2024 Park al. (025). For instance, HaloScope (Du et al. 2024) leverages unlabeled embeddings and applies SVD to identify key subspace directions, followed by probing to link these directions to hallucinations. Yet. probing- methods often rely on predefined supervised labels, making them less general- izable when feature dimensions are large or category priors are incomplete. Another line of work approaches hallucination detection from the perspective of output consistency. FigenScore (Chen al. 2024) quantifies semantic agreement through covariance eigenvalues, while Farquhar et al. (2024 employ clustering and semantic entropy for hallucination detection. These methods are ef- fective practice but may suffer from misclassification due to their inability to exploit internal reasoning information.
Hallucination detection.
Different from these approaches, our method explicitly separates semantic and reasoning subspaces, and projects hidden states onto the basis vectors of the reasoning subspace construct compact and interpretable features for hallucination detection.
PRELIMINARIES
In this section, we first formulate a mathematical model to characterize the hallucination behavior of LLMs. Then, we analyze how the hidden state space evolves across decoder layers during genera- tion, and subsequently decompose it into the direct sum of the semantic subspace and the reasoning subspace. This theoretical framework forms the foundation of HARP and provides essential support for hallucination detection via reasoning subspace projection.
3.1 MATHEMATICAL MODELING OF LLMS' HALLUCINATION
To model LLMs' hallucination mathematically, we first define the knowledge set known to the LLMs. Given an input sequence æ and its reference answer y", the LLMs generate multiple re- sponses y = {y' y2 ........ y¹ for æ. If any generated response closely matches the reference answer, the knowledge about is considered known to the LLMs, denoted as known(x:) (x) = 1. Formally:
known(x) 1, By Y, sim(y, v²) A
0, otherwise
where sim(y, y*) measures the similarity between y and y², and À is similarity threshold. Let Kknown = {x known(x) = 1} denote the set of all inputs whose knowledge is known to the LLMs. For each Mknown, let y LLM (x) denote the response generated by the LLMs. The hallucination indicator G(y |x) is then defined as:
1,
sim v²) S otherwise
G(y x))
(2)
0,
When G(y x) 1, the QA pair [z, v] exhibits hallucination.
3.2 DIRECT SUM DECOMPOSITION OF HIDDEN STATE SPACE
Let the token vocabulary be T. For LLMs with decoder layers, an input token T is mapped by the embedding layer to an initial hidden state ho containing purely semantic information. As the hidden states propagate through successive layers, semantic and reasoning information are pro- gressively integrated into their representations. Finally, the Unembedding layer projects only the semantic component to generate the output token Eyen E T. Thus, the final hidden state hy simul- taneously encodes: (1) Semantic prediction information: To accurately generate the next token, h must retain sufficient semantic features. These features are primarily captured by the parameter matrix W uemmb of the Unembedding layer and play dominant role in predicting the next token. (2)
3

---

Published as a conference paper at ICLR 2026
Reasoning trajectory information: To support multi-step reasoning and intermediate state compu- tation, hy also encodes intermediate reasoning information that does not directly affect the output. This information is typically not explicitly captured by W Wunemb and exerts minimal influence on the final output.
Denote the hidden state space at layer I as H4. To disentangle these two signals, decompose Hi into the direct sum of two orthogonal subspaces:
H4 SSemantic
S Steasoming
(3)
where Semaantic and S Reasoning represent the semantic and reasoning subspaces, respectively. The final hidden state hy E M4 is projected token logits by the Unembedding layer:
logits W unemm hu
(4)
where W Wunemb denotes the Unembedding parameters. Let hu Semeemnticc and f4, Reasoning denote the components of ht in the semantic and reasoning subspaces, with Semanntii exerting primary influ- ence on the logits for token prediction, while Retttonng encodes the model's ' reasoning processes.
To empirically validate the existence and functional role of the reasoning subspace, we design a Rea- soning Patch experiment in Appendix E This experiment demonstrates that the reasoning subspace Reasoning indeed captures critical intermediate reasoning information by showing that patching reasoning components from correct solutions can effectively rectify erroneous reasoning trajectories while preserving semantic coherence.
4 METHOD
In this section, we detail the proposed HARP framework for hallucination detection, as illustrated in Figurr 1] First, in subsection 4.1] we validate the Unembedding layer's capability to effectively disentangle the semantic and reasoning subspaces. Then, in |subsection 4.2 and subsection 4.3] we present practical strategy for subspace decomposition. Finally, in subsection 4.4, we introduce the HARP algorithm, which performs hallucination detection based on reasoning subspace projection.
4.1 SUBSPACE DECOMPOSER - UNEMBEDDING LAYER
As shown in Figure 3. during token generation, the Un- embedding layer of LLMs compresses only the seman- tic information hu, Semeemantic in hidden states into the gen- erated tokens, filtering out the reasoning information hy, Reasoning used in intermediate computations. There- fore, by analyzing the basis vectors that interact with the Unembedding layer parameters W Wunemb. we can deter- mine the mathematical representations of the semantic subspace and its orthogonal reasoning subspace.
Agen E
Uneeneedding
Ay Somantte
Branessonigg
fü E Ssemantic C Speasening
Remsning
Based on the properties of the semantic and reasoning subspaces, their interactions with W Wuzemb can be defined as:
Decoder
E Sseeceatte
W Wunemb SSemantic & W Wunemb Hi
(5)
W Wunemb S SRensoning Rs
(6)
Embedding
In other words, Ssemantic aligns with the principal acting directions of W Wunems, while the orthogonal SReasoning contributes negligibly to prediction scores. [subsec- tion 5.3] we demonstrate the validity of our definitions for these subspace properties, laying the foundation for subsequently identifying the subspace basis vectors.
LET
Figure 3: Flow of semantic and rea- soning information within LLMs hid- den states.
4.2 DETERMINATION OF SURSPACE BASIS VECTORS VIA SVD
Given that the Unembedding layer can filter reasoning information, we first perform SVD on its parameter matrix W W.eembb By analyzing which hidden state components interact with W Weembb
4

---

Published as a conference paper at ICLR 2026
identify the basis vectors of the semantic and reasoning subspaces. As shown in Equation 7 decompose W anemb via SVD:
W Wnnemb USV Σ i=1 4:0:":
where U E RI E RITllx V E Rdxd, |/u1 | = and the singular values in E are sorted in descending order 0n 2 02 2 ... 2 oh > Jk+1 = 0kt2
da 0.
For any hidden state h = Ei= 444 E R", its interaction with W Wnemm is expressed as:
Σ U, a.3 E, (o:a;)u
(8) Since the vectors u, mutually orthogonal, it follows that Wunemb h 0 if and only if Ei= lo:a; O, in which case the vector filtered out by the Unembedding layer. In other words, h belongs to the reasoning subspace Reasoning if and only if all singular values correspond- ing non-zero a; vanish, Accordingly, we define orthogonal basis for the reasoning subspace as VR {2; og 0}, while the remaining directions Vs foz o; > 0} constitute the semantic subspace S Semantic Since Jisk 0, the semantic and reasoning subspaces can be expressed as:
W Wunemb h
Semantic = Span (fv), U2...., 1x
S Rtasoning Span (fr6+1, Pk42.-..,
(10)
Let =u. by denote the projection coefficients of the hidden state h onto the basis vectors. Then the components in the semantic and reasoning subspaces are h4,Semantic Exm 034; and Eizko1 a&y, respectively, with interactions with W Wnemm given by:
hy, Reasoning
E oi(auu,) W Wanemb hı
W Wunemb h4,Smmantic
(11)
W Wunemb hi, Reasaning
az (:(auu)) =0
(12)
This partitioning of the hidden state space aligns precisely with the definitions of semantic and reasoning subspaces in in(Equation 5) aa [aand[[quatien Equation 6) and provides a theoretical basis for constructing low-rank approximation-basee bbasd subspaces in real models.
4.3 CONSTRUCTION OF SEMANTIC AND REASONING SUBSPACES VIA LOW-RAND APPROXIMATION
ideally construct the semantic and reasoning subspaces, practice, the condition o = for singular values rarely holds. To address this, we perform a rank-k approximation of W unemb: extracting the & most representative semantic directions from its row space to define the semantic subspace under realistic conditions, and determine the reasoning subspace using orthogonal relationships.
While the method described in subsection 4.2
Specifically, based on Equation 7 for any l < raank(Wanmmtn the Eckart- Young- -Misky theorem (Bekart & Young [936] Greenacre et al. 2022) gives the best rank-k approximation We of W Wunemb under the Frobenius norm as:
W arg min | [Wnnemb rank(A){k
AlLF
(13)
To ensure that this approximation does not significantly degrade prediction accuracy, the following information- preservvtonn condition should hold:
V of < V
|| umemmb W
(14)
This condition implies that W₂ retains the majority of W Wunemt's 's information in the Frobenius norm, ie., the first singular values account for most of the total energy.
Figure 4a illustrates the singular value distribution of the Unembedding layer parameters. We ob- serve that the trailing 5% singular values are markedly smaller than the others, and the infor- mation loss associated with these minor singular values be safely ignored. Accordingly,
5

---

Published as a conference paper at ICLR 2026
Qwen-225- (woe-115-70--nnnmm in1--instm
Seramii
Projecton
Ressoning
1290
2001
1091
4091
Index
Layer
Layer
Layer 01
LLaMA 3.1.8
044
00
Layor
Layer
Layer
(a)
(b)
Figure 4: (a) Singular value distributions of W Wunemb after SVD, with hidden state dimensions of 3584 for D0ven-2.5-7B-Instuue and 4096 for LLMMA-3.1-8B. (b) Projections of hidden states onto the basis vectors of the semantic and reasoning subspaces across layers, where the first row shows the first three layers and the second row shows the last three layers. Further details are provided in Appendix B
set h d 95%. By analyzing W and incorporating it into Equation 9 and [gquation 10, derive the corresponding subspace representations. Denoting the basis of the reasoning subspace as VR- [0&+1, Pk42.-.... E Rdx(d-k), the projection of hidden states hy onto the reasoning subspace is:
projn (1+) V hu
(15)
In subsection 5.3 we experimentally demonstrate that replacing W Wanemb with We in the token prediction task introduces negligible error. This finding provides the basis for subsequently using proj n. (hz) as the input feature to construct the hallucination detector.
4.4 HALLUCINATION DETECTION VIA REASONING SUBSPACE PROJECTION
As shown in Figure 4b universal representations of hidden states are extracted from different layers of the LLMs and projected onto the basis V [Vs, VR]- We observe that shallow hidden states primarily enhance information in the semantic subspace, while deep hidden states exhibit stronger features in the reasoning subspace. This observation is consistent with definitions of the two subspaces. Based on this, we propose novel hallucination detection framework- detects hallucinations using projections of hidden states onto the reasoning subspace.
which
During training, HARP employs beam search strategy to generate multiple candidate answers [ (
7 7 nations. For QA pair [z. y] composed of n tokens, HARP computes the projection of each token's s hidden state onto the reasoning subspace and calculates its hallucination score. The maximum score among all tokens is taken as the hallucination score of the QA pair:
..... y }. for given question æ, and annotates whether each candidate contains halluci-
(yl (z) = max Do (proj n (M(T))
(16)
ISiS
where 0 denotes the parameters of the hallucination detector. go (projR (1, represents the hallucination score of the i-th token, and (y|ær) E [0, 1] is the score for the entire QA pair. We optimize the detector using the Binary Cross-Entropy Loss (Goodfellow et al. 2016):
f =fllg-- log(go) (1 flag) log(1 -go)
(17)
where flag E {0, indicates whether the QA pair [ar, y] contains hallucinations. Minimizing this loss trains hallucination detector G:
G(g|x) [go()/z) >
(18)
where a [o, 1] is the detection threshold. When (()lx) = 1, the QA pair considered halluci- nated. Beam search is used only during training to construct diverse supervision samples, whereas during testing, G relies solely on the projection of single sampled answer onto S Snesonnngg-

---

Published as a conference paper at ICLR 2026
As shown in Figure answer "The capital of the United States is Shanghai!" assigns a hallucination score of 0.73 to the token "Shanghai", whereas all tokens in the correct answer "The capital of the United States is Washington!" have scores below 0.01. This demonstrates the effectiveness of G.
for the question "Where is the capital of the United States?", the hallucinated
5 EXPERIMENTS
In this section, we first describe the experimental setup and demonstrate HARP's advantages over other hallucination detection methods across multiple models and datasets. We then analyze the validity of our proposed direct-sum decomposition of the hidden state space and the necessity of the projection operation, followed by an evaluation of the detection performance under varying reasoning subspace dimensions and hallucination score thresholds. Finally, we discuss HARP's cross-dataset generalization capability.
5.1 EXPERIMENTAL SETUP
Datasets and models. Our experiments cover four generative question answering (QA) tasks, in- cluding three open-domain dialogue QA datasets- NN Open (Kwiatkosski et al. 2019). Truth- fulQA (Lin et al. (022a) (generation task), and TriviaQA (Joshi et 2017) and reading com- prehension dataset, TyDiQA-GP (English) (Clark et al. 2020). To assess the effectiveness and gen- erality of our proposed framework, we conduct evaluations using widely adopted open-source foundation models: Qwen-2.5- 2---rrtn (Yang et al. 2024) and LLaMA-3 --1-BBBBBBBBBB (Grattafiori et al. 2024). More dataset and inference details are provided in Appendix A
Evaluation Metrics. AUROC (area under the ROC curve) is employed as the primary evaluation metric. AUROC measures binary classifier's ability distinguish positive and negative samples across different thresholds, ranging from to 1, with higher values indicating stronger discriminative power. AUROC equal to indicates perfect classification, while value of 0.5 corresponds to random guessing.
Baseline Methods. HARP compared with several mainstream hallucination detection meth- ods, including Perplexity (Ren et al., (003), LN-Entropy (Malinin & Gales] (021). Semantic En- tropy (Farquhar et al.] (024)), Lexical Similarity (Lin et al. 2022b), EigenScore (Chen et al. 2024).
and HaloScope (Du et al. 2024)).
Correctness Measurement. Following Chen et al.] (2024), correctness determined based on ROUGE-L and semantic similarity between generated and reference answers. Semantic similar- ity is computed using the BLEURT model (Sellam et al. 22200; Park et al.) 2025). An answer is considered correct if its ROUGE-L score exceeds 0.7 or its semantic similarity exceeds 0.5.
5.2
MAIN RESULTT
Table summarizes the AUROC scores (in %) of various hallucination detection methods across four QA datasets, using Qwen-2.5-7B-Innstrne and ILLMMA3.1-8B as backbone models. Several key findings emerge from these results. (1) HARP consistently outperforms all baseline methods across all datasets and models, often by a significant margin. For instance, on TriviaQA, HARP achieves AUROC scores of 92.8% on Qwen and 92.9% on LLaMA, yielding improvements of +7.5% and +16.6%, respectively, over the second-best method, demonstrating its robustness and scalability across architectures and data characteristics. (2) Baseline methods such as Perplexity and HaloScope perform competitively on simpler datasets like TriviaQA. where answers are often lim- ited to one or two tokens, but their performance deteriorates sharply on more complex datasets such as TyDiQA, which contains long contexts and accompanying documents. contrast, HARP main- tains high AUROC scores of 88.4% on Qwen and 86.6% on LLaMA in these challenging settings, highlighting its ability to handle reasoning-intensive intensive and context-rich inputs. (3) Sampling-baat based methods, such as Semantic Entropy, Lexical Similarity, and EigenScore, incur higher computational costs but still fail to achieve comparable performance, whereas HARP's single- pass approach pro- vides both superior efficiency and accuracy.
In addition, Table repprrs the number of known and unknown questions for Qwen- 2.5-7B-Instruct Instruct across the four datasets, reflecting the model's 's varying answering capabilities on these benchmarks.

---

Published as a conference paper at ICLR 2026
Table I: Main result. Comparison of different methods hallucination detection performance across multiple datasets. All values are AUROC percentages. "Single" indicates whether multiple samplings are required for hallucination detection.
Models
Methods
Single
NQ Open
TruthfulQA
TriviaQA
TyDiQA
Perplexity
76.5
64.4
30.5
LN-Entropp
X
77.7
63.6
80.2
47.1
Semantic Entropy
X
77.7
60.0
76.1
68.6
Qwen-2.5-7B -a-ttt-----nstt
Lexical Similarity
77.8
63.9
76.9
60.3
EigenScore
78.9
63.8
74.8
HaloScope
60.7
63.0
85.3
69.0
HARP(Ours)
84.0
88.1
92.8
88.4
Perplexity
50.3
71.4
76.3
53.4
LN-Entropy
52.7
62.5
48.8
Semantic Entropy
60.7
59.4
68.7
62.2
LLaMA-3.1-8B
Lexical Similarity
X
60.9
49.1
71.0
69.5
EigenScore
56.7
45.3
69.1
82.4
HaloScope
62.7
70.6
76.2
53.3
HARP(Ours)
89.4
88.5
86.6
Collectively, these findings validate the effectiveness, robustness, and practical utility of HARP for hallucination detection in diverse QA scenarios.
Table 2: Distribution of known and unknown questions across four QA datasets. A question classified as Known if the model's s knowledge state contains the correct answer according to the criterion in Equation I and Unknown if none of the 10 candidate responses contain the correct answer.
Dataset
Known
Unknown
TruthfulQA
636
181
TyDiQA
402
38
TriviaQA
6225
3735
NQ-open
293
3317
5.3 MORE ANALYSIS
Rationality of Direct Sum Decomposition in Hidden State Space. To validate this direct sum decomposition, we conduct a comparative experiment: removing the reasoning subspace compo- nents of hidden states and examining their effect token prediction scores and rankings. Mathe- matically, this operation can be formulated as:
logits' WR h4 = W Wanemb b4, S1.emmantic
(19)
As shown in Figure 5a computing token prediction scores using Equation 19 instead of the original logits maintains the top rankings of greedily generated tokens. This result aligns with our theoretical design: the hidden state space can be decomposed into semantic and reasoning subspaces, and token prediction mainly influenced by the semantic subspace component hy, Semaetitt This experiment confirms that the proposed direct sum decomposition exhibits clear representational disentanglement and functional partitioning, providing theoretical support for building hallucination detection models based on the reasoning subspace.
Ablation Study. We tested the importance of projecting hidden states onto the reasoning subspace by comparing hallucination detection performance under different projection strategies. "HARP (w/o)" denotes completely removing the projection, while retaining hidden state features of the same dimensionality as full HARP: "HARP (random)" denotes randomly selecting a set of bases from the projection basis V = {U1, 02 ....... Va} for projection. The results in Table 3 show that both removing the projection and using random projection significantly degrade hallucination detection performance, confirming the necessity of projecting hidden states onto the reasoning subspace.
8

---

Published as a conference paper at ICLR 2026
25%5-75%
NQ Open Qwan
within 1.5KRR
TrubfuuQA Qwen
NO Open
16-
TruthfulQA aas
14-
12-
ATROOC (6)
Rank
128
Reasoning Dimension
Ressoning Dimension
(a)
(b)
Figure 5: (a) Greedy token rankings logits' under different reasoning subspace dimensions. (b)
Effect of reasoning subspace dimension on hallucination detection performance.
Table 3: Hallucination detection performance under different projection strategies
Owen-2.5-77 IBsttruttt
LLaMA-3.1-8B
Methods
NQ Open
TruthfulQA
NQ Open
TruthfulQA
HARP (w/o)
62.9
70.7
70.4
73.5
HARP (random)
67.6
68.6
59.5
75.8
HARP
84.0
88.1
89.4
88.5
Impact of Reasoning Subspace Dimension on Hallucination Detection. The reasoning sub- space dimension affects hallucination detection two ways: (1) its influence on logils scores: when the dimension is too large, [Equation 14] gradually breaks down, which impairs the model's next-token prediction capability: (2) its effect on detection accuracy and generalization: increas- ing the dimension may improve training accuracy but also increases the risk of overfitting, reducing generalization. We evaluated dimensions from 32 to 1024 using Qwen-2.5-7B, 2isenn2.5-7B-Instiie Instruct and LLaMA- 3.1-8B models. As shown in the Figure 5b] dimension of 256 yields the best performance. This di- mension accounts for only about 5% of the original hidden state dimensionality, preserving sufficient reasoning information while filtering most redundant noise, satisfying the information-preservatii constraint in Equation 14
Accuracy (%)
NQ Open
50
NQ Open
TruthfulQA
TrathfulQA
TriviaQA
40
TriviaQA
TyDiQA
TYDiQA
30
Thresholds
Thresholds
(a)
(b)
Figure 6: (a) Effect of hallucination score threshold on detection accuracy. (b) Effect of hallucination score threshold on detection Fl score.

---

Published as a conference paper at ICLR 2026
Selection of Hallucination Score Threshold. In practice, it is necessary to set hallucination score threshold a that G(yl produces clear binary decision. As shown in Figure 6a and and Fig- ure 6b) when a is between 0.2 and 0.8, both detection accuracy and FI score remain high, indicating a substantial separation between normal and hallucinated answers under G. To align with common expectations for binary classifier, we set 0.5, where Ggylz)) 11 (y(lz) >> 0.5].
Robustness Experiments. To apply HARP in real-world scenarios, examined its performance under distribution shifts between training and test sets. We trained the hallucination detector on source dataset s and evaluated well across multiple target datasets. Notably, when trained on TriviaQA, its accuracy on NQ Open nearly identical to directly training on NQ Open, demonstrating HARP's strong robustness and cross-distribution adaptability.
different target datasets t. Figurr 7 shows that HARP generalizes
92.9
NQ Open (6)
89.4
77.9
91.7
81.6
89.6
TruthfilQA
76.2
88.5
91.9
81.2
86.2
TriviaQA (6)
89.2
84.9
92.9
80.7
82.9
79.5
TyDiQA
82.0
80.9
86.6
76.2
NO (je ())
TrubfulQA
TriviQA e
TyDiQA
Figure 7: Cross-datasct generalization. "(s)" indicates the source dataset used for training the hallu- cination detector; "(t)" indicates the target dataset.
CONCLUSION
In this study, we introduced HARP, a novel hallucination detection method that leverages only rea- soning information input features, achieving high detection accuracy while maintaining strong robustness. First, we showed that the hidden state space admits a direct-sum decomposition into a semantic subspace and reasoning subspace, and that the Unembedding layer effectively sepa- rate these two components. Building on this, we applied singular value decomposition to the param- of the Unembedding layer and, following the Eckart-Young- W Wanemb with its best rank-k approximation W₁. Setting = d x 95%, we identified basis vectors for both the semantic and reasoning subspaces that align with empirical observations. Furthermore, we empirically validated that the reasoning subspace effectively captures intermediate reasoning information through the Reasoning Patch experiment detailed in Appendix Finally, HARP con- structs an accurate and efficient hallucination detector by using the projections of hidden states in the reasoning subspace as input features. Experiments show that HARP significantly outperforms existing mainstream hallucination detection methods and maintains robustness under distribution shifts across datasets. In addition, we present a proof-of-concep demonstration of ballucination mitigation using framework in (Appendix D and aim to inspire future research in this direction. The authors acknowledge OpenAl's ChatGPT (OpenAl 2025) for its support in language polishing of this manuscript.
theorem, approximated
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lennart Bürger, Fred Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in Ilms. Advances Neural Information Processing Systems, 37:138393-138411 2024.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Llms' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744, 2024.
Pei Cheng, Xiayang Shi, and Yinlin Li. Enhancing translation ability of large language models by leveraging task-related layers. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp- 61 6110-6121, 2024.
Jonathan H. Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. Tydi QA: benchmark for information-seeking question an- swering in typologically diverse languages. Trans. Assoc. Comput. Linguistics, 8:454-470, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14 14168, 2021.
Xuefeng Du, Chaowei Xiao, and Sharon Li. Haloscope: Harnessing unlabeled LLM generations for hallucination detection. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Anmual Conference Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (3):211-2183 1936.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nat., 630(8017))625--630) 2024.
Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Sci- ence, 336(6084):998-99)) 2012.
Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep Learning. Adaptive computation and machine learning. MIT Press, 2016. ISBN 978-0-262-03561-3.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Michael Greenacre, Patrick JF Groenen, Trevor Hastie, Alfonso Todice d' d'Ezza, Angelos Markos, and Elena Tuzhilina. Principal component analysis. Nature Reviews Methods Primers, 2(1):100, 2022.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.016610 2023.
Linyang He, Peili Chen, Ercong Nie, Yuanning Li, and Jonathan R. Brennan. Decoding probing: Revealing internal linguistic structures neural language models using minimal pairs. In Nico- letta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 4488 4497. ELRA and ICCL, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
11

---

Published as a conference paper at ICLR 2026
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang. Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng. Kaize Ding, Fan Yang, Mengman Du, and Yongfeng Zhang. Explor- ing concept depth: How large language models acquire knowledge and concept at different layers? In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Lingnistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pp. 558-573 Association for Computational Linguistics, 2025.
N. Johnson- Laird. Mental models: towards a cognitive science of language, inference, and con- sciousness. Harvard University Press, USA, 1986. ISBN 0674568826.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. Regina Barzilay and Min- Yen Kan (eds.). Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Pupers, pp. 1601-1611. Association for Computational Linguisties, 2017.
Tianjie Ju, Weiwei Sun, Wei Du, Xinwei Yuan, Zhaochun Ren, and Gongshen Liu. How large lan- guage models encode context knowledge? a layer-wise probing study. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Pro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re- sources and Evaluation and ICCI.
2024), pp. 8235-8246 Torino, Italia, May 2024. ELRA
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Weig Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. Lingnistics, 7:452-466, 2019.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volune 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3214-3252. Association for Compu- tational Linguistics, 2022a.
Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural- syubol-ymbooli graph semantic parsing via uncertainty. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACI. 2022, Dublin, Ireland, May 22- 27, 2022, pp. 4160 Association for Computational Linguistics, 2022b.
Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in transformer-based language models. arXiv preprint 2tt.::20031192.... 2024.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference Learning Representations, 2021. URL httph:ttt///. //ppnn/vepee..eoo//e/e
JN5y 2b507m
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer com- munication in transformer language models. Advances in Neural Information Processing Systems, 11.61372-614133 2024.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: A survey, arXiv preprint arXiv:2402.06196, 2024.
12

---

Published as a conference paper at ICLR 2026
OpenAl. Chatgpt. lattps://openai.. chat 2025.
Seongheon Park, Xuefeng Du, Min-Hsuan Yeh, Haobo Wang, and Yixuan Li. How steer LLM latents for hallucination detection? In ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI, 2025. /tLLittpss//ppeneeiier forum? 1 d= [ZRxk0gnC4
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter Liu. cat-of-distrbbutinn detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2023. URL
https:/ http:///pppeeeee...emmmm net oepeenreiee....ttfoooooooo.dsds fo
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. BLEURT: learning robust metrics for text gen- eration. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7881-7892. Association for Computational Linguistics, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of tf-thughht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1rXiv:241..55115. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 093388 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A DATASETS AND IMPLEMENTATION DETAILS
Input prompts. In experiments, datasets were categorized based whether additional sup- porting information is provided. For datasets without context, including NQ-Open, TruthfulQA, and TriviaQA, used prompts that contain only the question. Specifically, the prompt format is:
Prompts for datasets without context
Q:
(question)
A:
For datasets with context, including TyDiQA, the prompt includes both the task description and the relevant context:
Prompts for datasets with context
Concisely answer the following question based on the information in the given passage:
Passage: [context)
Q: [question]
Implementation details. Using the formulations in subsecctn 3.1 we select LLMs' known knowl- edge set Mnoow {z known(z) 1} and unknown knowledge set Kunknown {o known(x) = 0}. 75% Kenown is used for training, while the remaining 25%, together with Kunknownn: used test the hallucination detector unseen data. For dataset questions, the tem- perature is set to 0.5, and beam search is used to generate 10 answer paths per question. The hallu- cination detector G is two-layer MLP with hidden dimension 1024 and ReLU activation. Training is conducted for 50 epochs with the Adam optimizer, initial learning rate le-4, cosine decay, batch size 128, and weight decay 3e-4.
B EXTRACTING A UNIVERSAL REPRESENTATION VIA UNCENTERED PCA
Given collection of n hidden vectors (1(0) into a matrix:
Ji=1 from LLMs, each of dimension d, we arrange them
(T
M
E R nnd
(20)
(T
From an maxiiizaaiii perspective, the "universal representation" of these hidden vectors can be interpreted as their dominant direction variation in the feature space. To extract this direction, perform SVD:
M ==U'EN
(21)
where U Raxn E' diag(o (' o'' Raxd R V [uy, ... on E Rdxd and the singular values satisfy of > og > 2 0. The dominant right singular vector vı provides the principal direction of the row space of M, which equivalent to the first principal component in uncentered Principal Component Analysis (PCA). We define the universal representation direction
h = of E d
(22)
By collecting hidden states from the i-th layer, we can derive the corresponding universal repre- sentation h; following the steps above. Projecting it onto the basis vectors V/ [s. Vnl E gdxd yields the projections of the i-th layer's hidden state onto the semantic and reasoning subspaces:
proj (14a
(23)
14

---

Published as a conference paper at ICLR 2026
In Figure 4b, we normalize the lengths proj (1:) and visualize the projections the universal representations of hidden states from the first three and last three layers of the twen-2.5-7B-Instmun model onto the semantic and reasoning subspaces. We observe that shallow layer vectors are pri- concentrated in the
marily represented in the semantic subspace, while deep layer vectors reasoning subspace.
C ANALYSIS OF LAYER-WISS WISE CONTRIBUTIONS IN LLMS
Although previous analysis has characterized the hidden states after processing through multiple decoder layers, it remains important to understand the individual contributions of each layer and how they differ. To this end, we define the contribution the i-th decoder layer as dh, h4 h4-1, and, following the method described in Appendix B compute the universal representation direction dh;- Since singular vectors obtained via SVD can have arbitrary signs, we compute the absolute cosine similarity between dh; and dhg to measure the similarity between the universal representations of the increments the layers and j.
Layer-24
Layer-23
Layer-22
Layer-21
Layer-20
Layer-19
Layer-18
Layer-17
Layer-16
Layer-15
0.6
Layer-14
Layer-13
Layer-12
Layer-11
Layer-10
0.4
Layer-09
Layer-08
Layer-07
Layer-06
Layer-05
Layer-(4
Layer-03
Layer-02
Layer-01
Layer-01
Layer-02
Layer-03
Layer-04
Layer-0S
Layer-06
Layer-07
Layer-08
Layer-09
Layer-10
Layer-11
Layer-12
Layer-13
Layer-14
Layer-15
Layer-16
Layer-17
Layer-18
Layer-19
Layer-20
Layer-21
Layer-22
Layer-23
Layer-24
Figure Similarity between universal representation directions of layer-wise increments
Figure 8 illustrates the cosine similarity between the universal representation directions of layer- wise increments in the 2.ve--2.5-0.5BB-Inmmn
model. We observe that the first six layers behave in broadly similar manner, however, the first two layers are relatively independent of the remaining ones, while layers 3, 4, and exhibit almost identical directions. Interestingly, the direction around layer 22 is remarkably similar to that of layers 3, 4, and 6. We hypothesize that the first two layers primarily parse the shallow semantic structure of the input, layers 3, 4. and encode this information into internal representation space (a "language" specific to the LLM), the intermediate layers perform various reasoning operations over this representation, and layer 22 decodes it back into sumann-iterrpretblle semantic space before unembedding.
15

---

Published as a conference paper at ICLR 2026
Based on this observation, argue that mitigating hallucinations especially those induced by suboptimal training patterns or aggressive answer-generation strategies requires intervention the decoding behavior around layer 22. Following this intuition, Appendixx D provides demonstration of hallucination mitigation, with the goal of inspiring future research in this direction.
D MITIGATING HALLUCINATIONS BY MODIFYING REASONING SUBSPACE COMPONENTS
Based on our previous analysis of LLM behavior, we explore mitigating hallucinations by interven- ing on the components of hidden states within the reasoning subspace. To this end, we introduce fictitious city, "Epsilon", and pose the question to the LLM: "The capital of Epxilon is ?". We then investigate the effect of removing the reasoning subspace components from hidden states at different layers and observe the resulting impact on the LLM's outputs.
Table 4: Impact of interventions across layers and reasoning subspace dimensions on hallucination mitigation for the query. Green responses indicate correct mitigation of hallucinations.
Reasoning Subspace Dimension
Intervened Layer
Output
None
None
Epsilon's 's capital is likely ** "*aiyuan**, which is the capital city of the Kingdom of Kaiyuan in the Eastern Regions.
Epsilon's 's capital likely be the city town where its government and administrative center is located. Epsilon's 's capital likely be the city town where its government and administrative center is located.
Layer-01
16
32
Epsilon's 's capital likely be the city or town where its government and administrative center is located.
64
The capital of Epsilon is Elea.
8
Epsilon is the capital city of the planet Alpha. apologize, but don't have enough context to determine the spe- cific name or location of the capital city in question. apologize, but I'm not able to determine the capital city of Epsilon as it appears be fictional planet or alternate universe.
16
Layer-02
32
64
Epsilon is the capital city of the European Union (EU).
I'm sorry, but don't 't have enough context to accurately answer your question about the capital city of Epsilon.
Layer-22
16
I'm m sorry, but I need more context to accurately answer your ques- tion.
32
I'm sorry for any misunderstanding earlier. Epsilon currently not specified in my knowledge base for now,
64
8
I'm sorry, but don't 't have enough context to accurately answer your question about the capital city of Epsilon. m sorry, but I need more information to accurately answer your question.
Layer-23
16
32
Epsilon is currently not in my knowledge base am AI lan- guage model created by Alibaba Cloud based on publicly available information...
64
Epsilon is currently unknown due to lack of information about its current status in relation other planets in our solar system neighboring celestial bodies...
Table 4 presents the outputs of the LLM under interventions in various layers and with different subspace dimensions of reasoning. We observe that interventions in shallow layers, such as layers and 2, produce limited improvement, whereas interventions at deeper layers, such as layers 22 and 23, lead the LLM to explicitly acknowledge its lack of knowledge about the fictitious city "Epsilon"
16

---

Published as a conference paper at ICLR 2026
and refuse to answer. This phenomenon aligns with earlier analysis of the behavior of LLMs. We hope that this callucination-mittigen demo can inspire further research in this direction.
E VERIFICATION OF REASONING INFORMATION IN THE REASONING SUBSPACE
To verify that the components of hidden states lying in the reasoning subspace indeed encode inter- nal reasoning information, we design a controlled experiment consisting of three input conditions Figure 9). These conditions isolate the effect of the reasoning subspace while keeping all other factors unchanged.
(A) Normal:
Decoder
Question
Embodding
Unembedding
Wrong
N Normol
It Normal
Maintain
(C) Reasoning Patch:
Decoder
Reasoning steps
Question
Embedding
x
Unembedding
A Semoante
A Recsoning
Correct answer
Replace
(B) CoT:
Decoder
Reasoning steps
CoT Question
Embedding
x
Unemhedding
Semonnte
R Reasoning
Correct answer
Figure 9: Experimental design illustrating the three conditions used to verify the causal role of the reasoning subspace. (A)Normal: the model receives only the question and produces an incorrect answer. (B)CoT: chain-of-thought is prepended, enabling multi-step reasoning and correct answer. (C)sennnnn Patch: no CoT is provided, but the reasoning-subspace components of hidden states at all layers are replaced with those from the CoT run, causing the model to generate reasoning steps and arrive at the correct answer.
(A) Normal: direct question input. In the first condition, we feed the model only the question without any chain-of-thought (CoT) guidance. The model typically produces incorrect answer. Let the hidden state be
h. Normal
h Normal Semanticc
+ h. Normal "easoning
prepend chain-of-thought. In the second condition, we prepend chain-of- thought (Wei al., (022) to the input. The model now first generates intermediate reasoning steps and then outputs the correct answer. The hidden state is
(B) CoT:
h C.TT
CoT Semantic
+
CoT Reasonnnn
(C) Reasoning Patch: replace reasoning components at all relevant layers. The third condi- tion serves the key causal intervention. The input text is identical to condition (A); however, every decoder layer that contributes to the representation of token, we replace only the reasoning- subspace component of the hidden state with the corresponding component extracted from condition (B), Formally, for all layers along the forward-pass trajectory of token f, we apply:
h Patch
Normal Se.aanttii
+ CoT "eeaoson
Thus, semantic information preserved at every layer, while the reasoning components across all intermediate layers are substituted with those from the CoT run. This ensures that the patched forward pass follows the CoT reasoning trajectory throughout the entire decoder stack.
Key result. We evaluate the effectiveness of the proposed Reasoning Patch mathematical rea- soning benchmarks such as GSMSK (Cobbe et al. 202T), using both few-shot CoT and zero-shot Col' to extract the reasoning-subspace components.
17

---

Published as a conference paper at ICLR 2026
(A) Normal
CoI
Imput:
Input:
Q: Mishka bought pairs of shorts, pains pants, and pairs pair of shorts costs S 16. 50. One pair of pants costs $22.50 and one pair shoes costs S42. How many dollars did Mishka spond on all the clothing items? Ac
C..eewsstoo CoT prompt...)
Misháka bought 3 pairs of sborts, 3 pairs of pants, and 5 pairs of shocs. One pair of shorts costs $16.50. One pair of pants costs $22.50 50 and one pair of shoes costs $42, How many dollars did Mishia spend A:
clothing items?
LLM Output:
Calculate each type clothing:
LLM Ouiput: 247.5
Shorts: x $16.50 $49.50
Punte: 3 -22.22500 50 $67.50
2. Add up the costs find the tocal amount spent
Tottal cos
$243.00 .000
Therefore, Mishka spent $243.00 on all the clothing items.
(C) Patch
Input:
Q: Mishia bought pairs shorts, pains pants, and pains shoes. One pair shorts costs $16.50. One pair pants costs $22. 20 and one pair of shoes costs $42, How many dollars did Mishka spend on all the clothing items?
A:
LLM Output:
1. Calculate the socal cost for each type of clothing item:
Shorts: 3 S16.5 $49.5
Pants] $22.5 $67.5
Shoes] 3 1
2. Add up the tocal cost for all types of cloching frems:
Total - S49.5 $67.5 + $126 $243
Therefore, Mishka spent total of $243 on all the clothing items.
Figure 10: Reasoning Patch experiment using.few-slle chain-of-thought supervision.
outputs under the three conditions (A)-(C) when the reasoning components condition (C) are derived from few-shot CoT, with the full prompts shown in Table 5 We observe that, even though condition (C) receives no CoT text in the input, injecting the CoT-derived gaasoning-subspaee components reliably triggers the model to follow a "reason- then-answer" generation pattern. As a result, the model transitions from an incorrect answer in (A) correct, multi-step reasoning process in (C), demonstrating that the patched reasoning trajectory causally determines the emergence of correct step-by-step reasoning.
Figure 10 presents the
Table 5: few-shot chain-of-thouge prompt.
Q: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? robe needs bolts of blue fiber. The amount of white fiber needed is half of the blue fiber.
Half of 2 bolts is bolt of white fiber.
The total bolts needed the sum blue and white fiber. 2 bolts plus bolt equals 3 bolts. Therefore, the final answer is 3.
In..utut...)
Figure shows the corresponding results when the reasoning components are extracted from zero- shot CoT. Remarkably, even though condition (C) does not contain the zero-shot instruction (c.g., "Answer the following question step by step to the best of your ability."), the patched model nonethe- less produces a coherent step-by-ster reasoning chain before giving the final answer. Interestingly, in this setting the original CoT run in condition (B) makes arithmetic mistake and outputs incor- rect final answer; however, condition (C>-which inherits only the saasoning-uubspaie components rather than the explicit token sequence- doees not reproduce this error and instead produces the cor-
18

---

Published as a conference paper at ICLR 2026
(A) Normal
(B) CoI
Input:
Input:
besswax candles. For every pound of beeswas, he can make 10 tapered candles, One pound of beeswax and the wicks cost $10.00 in supplies. be sells each candie $2.00 cach,
Charlic
Anewer the followinng quertion step by the best your ability
Q: Charlie wants to sell beeswax candles. For every pound of beeswax, can make 10 tupered candles. One pound of beeswax und the wicks cost $10.00 in supplies. he sells each candle for $2.00 each, what his net profit if he makes and sclls 20 candles?
profit if he makes and sells 20 candles? A:
LLM Outputt
1. First, let's colculate the total cost of supllies for makingg 20 candlex:
that pound beeswax wick vos $11.00
find many pounds
LLM Outpul:
booswax ane needed 20 candles, we divide the number of camdlles by the number cundles per pound:
20/ 10 -2 pounds. in Therefore, the tocal cost of supplies 2 $11.00 $22.00 Next, calculate the
3.6
from selling candles:
We know that each candle sold for $2.0 00.0 Therefore, total revenue 3. Finallyy let's calculate the not profit:
$2.00 $44.00
Net prufit totall revenue
total cost supplies Net profit - $44.00 $22.00 22.00-mm $22.00
Therefore, Charlie's is net profit makes and sells 20 candles $22.00.
(C) Putch
Impitt:
Q: Charlie wants to sell beeswax candles. For every pound of boeswax, he can make tapered candles, One pound of beeswax and the wieks $10.00 in supplies. be sells each candle for $2.00 00 each, what his net profit if he makes and sells 20 candles?
A:
LLM Outpurt
Calculate the lonal cost of supplies for makinge
Total 20
$1.00 per cundle $20.00
Calculate the lotal revenue from selling the candles:
Total revenue 20 candles x $2.00 per candle - $-10.00
Calculate profit subtracting total from the total revenue:
Net profit $40
$30.00
Therefore, the profit for Churlie he makes and sells candles $20,00
Figure 11: Reasoning Patch experiment using zero-shot chain-of-thought prompting.
rect result. This highlights that the reasoning subspace captures the structural reasoning trajectory without being constrained by the semantic information in the CoT prompt.
Together, these results provide compelling evidence that the reasoning subspace encodes causally meaningful internal reasoning information, and that injecting its components is sufficient to induce coherent multi-step reasoning even in the absence of explicit CoT prompting.
F COMPUTATIONAL COMPLEXITY OF SVD
To construct the reasoning subspace, we perform singular value decomposition (SVD) on matrix M E Rnxdd where n denotes the vocabulary size and d is the dimensionality of the hidden rep- resentation. In typical large language models, the matrix is tall and skinny with > d (e.g., for Qwen2.5-7B. n 152,064 and d = 3,584). The computational complexity of SVD depends on these matrix dimensions as well as whether a full or truncated decomposition applied.
Time Complexity. For a full SVD on an n x matrix, the time complexity is
(minnnnd² n² n²d))
Since the vocabulary size is typically much larger than the hidden dimension, the dominant term becomes
O
which makes full SVD computationally expensive in practice. For truncated SVD that retains only the top-k singular directions, the complexity reduces to
O(nd)),
particularly when using iterative or randomized SVD algorithms. Such approximations are crucial for scaling to vocabularies of realistic size.
19

---

Published as a conference paper at ICLR 2026
Table 6: SVD computation cost on the unembedding layer using an H100 80GB GPU. We report wall-clock time, memory required during the SVD computation, and the additional memory by SVD.
Unembedding Shape
Peak Memory
Extra Memory
Model
Time
Qwen2.5-7BBIInstrue
152,064 3,584
1.30s
8.37GB
0.02GB
LLaMA-3. I-B
128,256 x 4,096
1.60s
8.08GB
0.03GB
Owen2.5-72B-Ile
152,064 8,192
9.83s
19.22GB
0.13GB
Swwe3-2235-A2222
151,936 4,096
1.70s
9.55GB
0.03GB
Instruct-2507-FPS (MoE)
Space Complexity. Storing the matrix M requires
O(n² + +nd + df)
memory. The truncated singular vectors U E Bunk and V Baxk introduce an additional
()k+)d)k)
space overhead. Because n is very large in modern LLMs, the memory is dominated by storing U.
SVD Resource Consumption. To quantify the practical resource requirements of performing SVD the unembedding layer, Table 6 summarizes the wall-clock time, the peak memory con- sumption during the SVD computation, and the additional memory introduced by truncated SVD across several representative models. The evaluation covers models of different scales including and the MoE model Qwen3-235B-
Qwenn..5-7B-Instrruet, LLaMA-3. 1-8B, Swwen2.5-72B-Instruus A22B-Instruuct2557-FPPP (Yang al. 2025 using an H100 S0GB GPU.
Singular Value Distribution in Larger Models. Figure 12 shows the singular value distribution of the unembedding layers for larger models, including Qwen2.5. (MoE). The trend of singular value decay is consistent with that observed for Qwen2.5-7B- --re-ttc-BBl and LLaMA-3.1-2 (Figure 4a), indicating that our method can be directly applied to larger models.
and Owen3-235B-
A22B-Insttruct2500--1
Singnlar valuess
1.0
100
Index
Qwen3-235t 22..2-325-2------ Instrnnttttttt000 2507-FFP
Singular valuues
10
107
100
2000
4000
6000
5000
Index
Figure 12: Singular value distributions of Wunemb after SVD, with hidden state dimensions of 8192
for Qwen2.5-72B- 7-2---trrtttttt and 4096 for 232B---2BBBBBB-------
-22---2nstttttt225555555555 Instruct-2507-FPS (MoE).
20


Reviews:
[{"review_id": "g19twhkcl1", "reviewer": "Reviewer_VdiQ", "summary": "This paper introduces HARP, a novel framework for hallucination detection in LLMs. The core hypothesis is that an LLM's hidden state space can be decomposed into a \"semantic subspace\" and a \"reasoning subspace\". The authors propose to identify these subspaces by applying SVD to the unembedding matrix. They define the \"reasoning subspace\" as the one spanned by the basis vectors corresponding to the smallest singular values. The method then projects the final-layer hidden states onto this low-dimensional \"reasoning subspace\" and uses these projections as features to train a simple MLP classifier for hallucination detection. \n\nThe paper reports state-of-the-art results on several QA datasets, suggesting high accuracy and efficiency.", "questions": "Please refer to the previous section", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The core idea of separating semantic and reasoning information by analyzing the unembedding matrix is novel and insightful.\n\nThe reported experimental results show a significant improvement over existing baselines, which is impressive.", "weaknesses": "The evaluation is confined to simplistic tasks that are misaligned with modern, critical LLM use cases.\n\nThe paper's entire empirical validation rests on short-form, factual question-answering datasets like NQ Open, TruthfulQA, and TriviaQA. While HARP demonstrates strong performance, these tasks are arguably \"solved\" problems for state-of-the-art LLMs and do not represent the true frontier of the current hallucination challenge. If this paper were submitted two years ago, its contribution would be significant, but the field has rapidly advanced. The critical applications where robust hallucination detection is most needed today are in:\n\n- Long-form generation (e.g., summarization, report writing)\n\n- Complex reasoning (e.g., multi-step math or logic problems)\n\n- Agentic workflows (e.g., software development and tool use)\n\n- ......\n\nIn these complex scenarios, hallucinations manifest as subtle logical fallacies, fabricated evidence, inconsistent reasoning steps, or non-existent API calls—not just the simple factual errors tested in this paper (e.g., \"The capital... is Shanghai!\").\n\n\nThe core assumption of HARP is that a \"reasoning subspace\" can be isolated and used for detection. However, the paper provides no evidence that this method can capture these more complex, structural, and procedural forms of hallucination. It is highly unclear whether the proposed technique can generalize beyond the simple, factual-recall domain to address the far more difficult and relevant problems facing LLMs today.\n\n----\n\nThe paper appears to be missing a critical and directly relevant baseline: Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models.\n\nThis work is highly relevant as it proposes a method for hallucination detection that also leverages the model's internal states. Specifically, the authors of that paper develop a Classifier that receives the contextualized embeddings of each token during the LLM’s inference process. It then outputs a probability indicating the probability of hallucination in the LLM’s output.\n\nGiven the strong methodological overlap (i.e., using internal representations/embeddings during inference for classification), this paper should be considered a key baseline.\n\nI strongly recommend that the authors:\n\n- Discuss this paper in the related work section.\n\n- Include it as a baseline in the experimental comparison and discuss how the current submission's performance and methodology differ.\n\n\n\n\nI am open to increasing my score if the authors address these points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lQ22ojvUI8", "reviewer": "Reviewer_EBLo", "summary": "This paper presents HARP (HAllucination detection via Reasoning subspace Projection), a framework for hallucination detection that mathematically decomposes the hidden state space into semantic and reasoning subspaces. The approach employs SVD on the Unembedding layer to derive basis vectors representing these subspaces, enabling hidden states to be projected onto the reasoning subspace, which then serves as the input to a hallucination detector. HARP achieves substantial feature compression (to about 5% of the original dimensionality) while isolating reasoning-related signals crucial for identifying hallucinations. Experiments across multiple datasets (NQ Open, TruthfulQA, TriviaQA, TyDiQA) and models show state-of-the-art AUROC performance, surpassing existing approaches and baselines.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Research Innovation: The paper offers a novel perspective by interpreting LLM hallucinations through the decomposition of the hidden state space into semantic and reasoning subspaces. This conceptual framing provides new insight into the underlying mechanisms of hallucination generation.\n2. Computational Efficiency: By projecting hidden states onto the reasoning subspace, the method effectively compresses feature dimensionality to about 5% of the original space, significantly improving both computational efficiency and robustness.\n3. Methodological Effectiveness: As shown in Table 1, HARP achieves substantially higher AUROC scores than competing baselines across four QA datasets and two LLMs. The results demonstrate strong consistency and robustness across different data domains and model architectures. The paper also conducts comprehensive ablation studies and analyses to support its conclusions.\n4. Clarity of Presentation: The methodology is clearly described and easy to follow.", "weaknesses": "1. Overreliance on SVD and orthogonality assumptions. HARP relies heavily on the assumption that the hidden representations can be effectively decomposed via SVD into orthogonal semantic and reasoning subspaces. In practice, neural network weight matrices rarely yield strictly zero singular values, and the orthogonality assumption may not strictly hold. Although Section 4.3 mitigates this issue using an approximate low-rank solution, the reliability of this decomposition requires more rigorous empirical validation and theoretical justification.\n2. Limited experimental scope and insufficient diversity. The current experiments are primarily conducted on QA datasets, which may not fully reflect the generalization ability of HARP. It would strengthen the paper to evaluate the method on a broader range of tasks—particularly those involving more complex logical reasoning (e.g., mathematical reasoning datasets) and non-QA settings. Moreover, experiments on additional model architectures and scales are necessary, since the compatibility of some methods may vary with model size and vocabulary distribution. Including additional evaluation metrics such as ECE or F1 would also provide richer insights. Finally, the paper does not clarify whether results are averaged over multiple runs or report variance, which raises concerns about the robustness and statistical significance of the findings.\n3. Lack of analysis on overconfidence and hallucination dynamics. LLMs are known to exhibit overconfidence, particularly on datasets with high accuracy. It would be valuable to analyze how HARP behaves under such conditions—e.g., comparing hallucination patterns and performance changes across models and datasets with varying accuracy levels. A differential analysis between correct and incorrect responses could also offer deeper insight into how HARP mitigates hallucinations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UXjKYwBjJr", "reviewer": "Reviewer_NZ6Q", "summary": "This paper proposes HARP (Hallucination Detection via Reasoning Subspace Projection), a framework for detecting hallucinations in Large Language Models (LLMs).", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Comprehensive Experimental Validation: The experiments cover diverse datasets (open-domain QA, reading comprehension) and models.\n\n2. Novel Theoretical Insight: The paper introduces a meaningful decomposition of the LLM hidden state space into semantic and reasoning subspaces.", "weaknesses": "1. Limited Generalization to Larger Models: The experiments only use medium-sized LLMs (Qwen-2.5-7B-Instruct, LLaMA-3.1-8B). It remains unclear whether HARP can maintain its performance on larger-scale models (e.g., 70B or 175B parameters), as the structure of hidden state spaces and the role of the Unembedding layer may change with model scale.\n\n2. Ambiguity in Reasoning Subspace Interpretation: While the paper defines the reasoning subspace as capturing internal reasoning processes, it lacks a more concrete, interpretable validation (e.g., linking specific components of the reasoning subspace to specific reasoning steps). This makes it difficult to fully confirm that the projected features truly reflect \"reasoning\" rather than other irrelevant hidden state information.\n\n3. Sensitivity to SVD Parameter Selection: The paper sets the semantic subspace dimension k=d×95%, based on singular value distribution, but it does not thoroughly explore how different k values (e.g., 90%, 98%) affect detection performance across diverse datasets and models. This may limit the adaptability of HARP to different LLM architectures.\n\n4. Lack of Comparison with Recent Advanced Methods: The baseline methods include relatively early works (e.g., Perplexity from 2023) but lack comparisons with the latest hallucination detection methods (post-2024) that also focus on model internal states. This makes it hard to fully position HARP’s competitiveness in the current state of the art.\n\n5. Practical Deployment Challenges: Although HARP reduces feature dimensions, it still requires accessing the hidden states of LLMs—a requirement that may not be feasible for closed-source LLMs (e.g., GPT-4, Claude 3). The paper does not discuss solutions for such scenarios, limiting its practical application scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pVFnGAcDXI", "reviewer": "Reviewer_Vnht", "summary": "This paper introduces HARP, a novel framework for detecting hallucinations in Large Language Models (LLMs). The central claim is that the final hidden state space of an LLM can be decomposed into a direct sum of two orthogonal subspaces: a \"semantic subspace\" and a \"reasoning subspace.\" The authors posit that the model's unembedding layer parameter matrix, W_unemb, inherently separates these two spaces. By applying Singular Value Decomposition (SVD) to W_unemb, the authors identify the basis vectors for these subspaces. Specifically, right singular vectors corresponding to large singular values are proposed to span the semantic subspace, while those corresponding to near-zero singular values are claimed to span the reasoning subspace.\nThe proposed method, HARP, projects the final hidden states onto this identified \"reasoning subspace\" to generate low-dimensional features. These features are then fed into a two-layer MLP classifier to produce a token-level hallucination score. The final hallucination score for a given response is the maximum score across all its tokens. The authors conduct experiments on several open-domain QA datasets (NQ Open, TruthfulQA, TriviaQA, TyDiQA) using Qwen-2.5-7B and LLaMA-3.1-8B models, reporting state-of-the-art performance in terms of AUROC.", "questions": "To be more confident in my assessment, I would like the authors to address the following:\n1. Can you provide any theoretical justification or cite prior work that supports the claim that the hidden state space naturally decomposes into orthogonal semantic and reasoning components, and that the unembedding layer acts as the separator? Beyond the analogy, what is the mechanistic basis for this hypothesis?\n2. How do you defend the specific label \"reasoning subspace\"? Have you conducted any analysis to show that activity in this subspace correlates with explicit reasoning steps (e.g., in chain-of-thought prompting, mathematical problems, or logical puzzles) and is not just capturing other forms of non-semantic information?\n3. Could you elaborate on the computational cost of applying SVD to the unembedding matrices of much larger models, such as those with vocabulary sizes exceeding 100,000 and hidden dimensions exceeding 8,192? Is the proposed method practical for state-of-the-art foundation models?\n4. Why should the community believe that findings on 7B/8B models are representative of hallucination phenomena in significantly larger models? Have you considered any experiments or analysis to suggest this generalization holds?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Novelty of Approach: The core idea of leveraging the spectral properties of the unembedding matrix to derive features for a downstream task is novel and interesting. It moves beyond standard probing or output-based consistency checks by attempting to ground the detection mechanism in the model's internal structure, which is a commendable direction for mechanistic interpretability.\n2. Simplicity and Efficiency: The proposed method is technically straightforward and computationally efficient. It relies on a one-time SVD pre-computation and a simple projection, avoiding costly iterative sampling or complex architectural modifications. If validated and efficient at scale, this would represent a practical and scalable approach for hallucination detection.\n3. Strong Empirical Results on Tested Benchmarks: The paper presents compelling empirical results, consistently outperforming several established baselines across multiple datasets and two different model architectures. The reported gains, such as a 7.5% AUROC improvement on TriviaQA, are significant and suggest that the extracted features are indeed highly informative for hallucination detection in the tested settings.", "weaknesses": "The paper suffers from fundamental weaknesses in its theoretical grounding and argumentation.\n1. Unsupported Foundational Premise: The cornerstone of this work—the decomposition of the hidden state space into orthogonal \"semantic\" and \"reasoning\" subspaces—is presented as a discovery but is, in fact, an unsubstantiated assertion.\n  ○ The paper transitions from a high-level cognitive analogy directly to a formal mathematical claim (Eq. 3) without providing any theoretical justification, empirical evidence from prior work, or logical deduction. Statements like \"we establish that the hidden state space ... can be decomposed\" are misleading; the property is not established, but is postulated.\n  ○ The properties defined in Eq. 5 and Eq. 6 are the very definitions the authors introduce. The subsequent claim \"Based on the properties...\" is a form of circular reasoning. The validity of the entire framework rests on this unproven assumption, making the subsequent technical development an exercise built on a fragile foundation.\n2. Misleading Terminology and Lack of Conceptual Rigor: The choice to label the near-null space of W_unemb as the \"reasoning subspace\" is highly speculative and lacks evidence.\n  ○ The authors provide no causal or correlational evidence linking this subspace specifically to what is commonly understood as \"reasoning\" (e.g., chain-of-thoughs, self-reflection, etc.). An alternative, and perhaps more plausible, hypothesis is that this subspace captures any information orthogonal to the final token prediction, which could include contextual noise, computational artifacts, or simply redundant features.\n  ○ Attaching the loaded term \"reasoning\" to this mathematically-derived space overstates the paper's findings and risks misdirecting future research. A more scientifically cautious term, such as \"output-orthogonal subspace,\" would be more appropriate.\n3. Arbitrary Formulation and Disconnected Sections: The mathematical modeling in Section 3.1, which defines hallucination via a similarity function sim and threshold λ, is both arbitrary and disconnected from the main technical contribution of the paper. This section does not inform the subsequent SVD-based methodology and its inclusion suggests a lack of a coherent, unified theoretical framework.\n4. Limited Experimental Scope and Unaddressed Scalability Concerns: The empirical evaluation, while strong on its specific benchmarks, is narrow and fails to address critical questions about the method's applicability to larger, state-of-the-art models.\n  ○ The experiments are confined to 7B/8B parameter models. It is unclear if the observed spectral properties of W_unemb and the effectiveness of HARP will generalize to much larger models (e.g., 70B+), where hallucination behaviors may differ.\n  ○ The paper does not discuss the computational feasibility of performing a full SVD on the unembedding matrix of a model with a very large vocabulary and hidden dimension (e.g., >128k vocabulary, >8192 hidden dim). This is a critical practical consideration for the method's scalability.\n  ○ Given the fact that Mixture-of-Experts (MOE) framework becomes status-quo choice in open-sourced and proprietary model developement, the experiment design does not reflect that its claim is applicable to a wide range of LLM design choices. The proposed method fails to generalize its impact in the mainstream research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)