Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViT
Authors: Guy Bar-Shalom
Fabrizio Frasca
Yaniv Galron
Yftah Ziser
Haggai Maron
Venue: NeurIPS 2025 poster ()
Keywords: Hallucination detection
Probing classifiers
LLM analysis
Primary area: deep_learning
TL;DR: We propose to treat LLM activations as images and detect hallucinations more accurately (and efficiently) across LLMs with a vision-inspired architecture.

Abstract: Detecting hallucinations in Large Language Model-generated text is crucial for their safe deployment. While probing classifiers show promise, they operate on isolated layer–token pairs and are LLM-specific, limiting their effectiveness and hindering cross-LLM applications. In this paper, we introduce a novel approach to address these shortcomings. We build on the natural sequential structure of activation data in both axes (layers $\times$ tokens) and advocate treating full activation tensors akin to images. We design ACT-ViT, a Vision Transformer-inspired model that can be effectively and efficiently applied to activation tensors and supports training on data from multiple LLMs simultaneously. Through comprehensive experiments encompassing diverse LLMs and datasets, we demonstrate that ACT-ViT consistently outperforms traditional probing techniques while remaining extremely efficient for deployment. In particular, we show that our architecture benefits substantially from multi-LLM training, achieves strong zero-shot performance on unseen datasets, and can be transferred effectively to new LLMs through fine-tuning.

Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViT
Guy Bar-Shalom"
Fabrizio Frasca"
Technion guyba990gmail.com com
Technion
fabriziof@campus technion. ac
Yaniv Gairon Technion
Yftah Ziser
Haggai Maron
University of Groningen, Nvidia Research
Technion, Nvidia Research
Abstract
Detecting hallucinations in Large Language Model-generated text is crucial for their safe deployment. While probing classifiers show promise, they operate on isolated boyer-token pairs and are LLM-specific, limiting their effectiveness and hindering cross-LLM applications. address these shortcomings. We build the natural sequential structure of activa- tion data in both axes (layers x tokens) and advocate treating full activation tensors akin images. We design ACT-ViT, a Vision Transformer-inspiinn model that be effectively and efficiently applied to activation tensors and supports training on data from multiple L.MSs simultaneously. Through comprehensive experiments compassing diverse LLMs and datasets, we demonstrate that ACT-ViT consistently outperforms traditional probing techniques while remaining extremely efficient for deployment. In particular, we show that our architecture benefits substantially from multi-LLM training, achieves strong zero-shot performance on unseen datasets, and can be transferred effectively to new LLMs through fine-tuning, Full code is available lttpt::///// // thub. com/ www./oonn/aauee Bet///BraauyAA--TTe
this paper, we introduce a novel approach to
Introduction
Despite their remarkable performance across wide range of tasks, the inner workings of Large Language Models (LL.Ms) remain poorly understood. This lack of transparency is particularly problematic in high-stakes scenarios, where L.MMs prone "hallucinations" cases in which the
model generates false or fabricated content 622
38
22
26
54
Accurate Hallucination Detection
(HD) is critical for the safe and reliable deployment of LLMS (51
35 (55). While existing methods
for detecting such hallucinations often rely on auxiliary LLMs or repeated prompting [34] 48 these approaches tend be slow and computationally intensive (up several seconds per instance (6)). A prominent class of methods suggest to overcome this additional computational cost via probing classifiers: simple models trained on internal LLM representations predict certain attributes [
These techniques have been central to interpretability research [20] 31
58
15]
[3
8), with linear
probes gaining popularity for their transparency and simplicity 25 37
Unfortunately, traditional probing classifiers exhibit two fundamental limitations. First, they operate isolated layer-token pairs, neglecting information from other activations within the model. This is critical: while predictive hallucination signals might indeed appear at specific "exact tokens", their po- difficult problem its own-practically requiring (multiple) LLM queries [48). In addition to this, these signals may also peak at different
sitions vary across responses, making the position-findin
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

X Richard Feynuaan
LLM speciiiiicc
Shared
Pb
Far
Pooling
Adapters
VIT-B
M
Albert Ensttei
H2
Dyg
Fa
MG
Hallucination prediction
:
Who developed the theory of general relativity?
Activation Tensors
Figure 1: -VT overview: we extract Activation Tensors from (multiple) LLMs, apply Pooling, and project them to shared space via per-LLM Linear Adapters. A shared ViT Backbone is then applied. ACT-ViT benefits from training on data from multiple LLMSs and can be easily fine-tuned to unseen ones.
layers per-dataset basis [5], underscoring the need for more holistic processing of the model's internal state. A second limitation is that probing classifiers are model-specific, thus unable to gener- alize across LLMs to benefit from hallucination--rlated insights found in other LLMs. This hinders sharing datasets across LLMs, transfer learning, and multi-model training. Crucially, each new LLM text generation task demands annotating a new dataset, a tricky and sometimes impractical procedure. Motivated by the above, we ask: (i) Can we design an architecture that processes the full internal state of an LLM and benefit from it?: and (ii) Can learn across multiple LLMs and benefit from it? In this paper, we positively answer these questions and propose a novel approach, dubbed ACT-ViT, addressing the aforementioned limitations in an effective and efficient way.
We start by providing further evidence on the limitations of On top of previous observations suggesting that the best probe-position varies across tokens and layers on a per-sample and dataset basis [48, [5) we show that the location of predictive signals for hallucinations also widely varies across LLMs, even for the same dataset. This further calls for the development of a more sophisticated approach that can adaptively attend to the most predictive "locations" within models' hidden states, especially if aiming to operate beyond the single-LLM regime. To this end, we propose leveraging, as our basic data type. what we term the Activation Tensor (AT) - comprehensive representation of the LLM's S internal state with respect to given input. Formally, for an LLM M and a N-long response to some input query, the AT is denoted A E L x N with La the number of layers, Dar M's hidden dimension. AT's are informative but challenging to process: they are large, high digheiimessional, and LLM-dependent. We process them efficiently and effectively by exploiting their structural similarity to images. An RGB image forms a H x Wxx x tensor with spatial dimensions (a grid of pixels) and feature channels. Similarly, Activation Tensors have "spatial" dimensions described by layer stacks (LM) and token sequences (N), and a feature dimension corresponding to the dimensions of the LLM's hidden states. This analogy enables applying proven computer vision techniques like spatial pooling and Vision Transformers [14].
static probing classifiers.
To unify ATs across different LLMs into consistent shape, we apply pooling over the spatial dimensions, followed by an LLM-specific Linear Adapter (LA) on the feature dimension. This transforms all tensors into shared shape of L,p x Np x D'. The resulting representation is then fed into a shared ViT-based backbone. See Figure for a complete depiction of our framework. These components are jointly trained for HD on multiple annotated AT datasets originating from different LLMs. Importantly, our use of linear adapters is inspired by recent work (24 46) suggesting that LLMSs converge towards a shared statistical model of reality, with the notion of universal truthfulness 43 60) positing they learn a generalizable, lincarly exploitable representation of factual truth. A key aspect of our architecture is its ability to be applied to new, unseen LLMs. In these cases, it suffices to only train the lightweight, LLM-specific LA, while the backbone pretrained for HD in a unified representation space across LLMs is kept frozen.
Results. our extensive experiments across 15 LLM dataset combinations, ACT- consistently outperforms traditional probing methods in both single- and multi LLMM training settings. Remarkably. exhibits strong zero-shot generalization to new datasets from LLMs seen during training and requires as little as 2 5% of the training data to improve performance further and in many cases- surpass traditional probes trained on the whole dataset. Furthermore, by training only the lightweight LA for new unseen LLM, model achieves superior performance than standard probes. ACT-ViT

---

trains on single GPU in less than three hours on all 15 L----tasset combinations together, and detects hallucination in less than 10- seconds per instance inference time.
Contributions. (1) We propose Activation Tensors (ATs) as a holistic, structured representation of LLM internals. (2) Inspired by the similarity of ATs to images we introduce novel architecture that can effectively train on ATs from multiple LLMs and datasets. We show it outperforms traditional probes, even when trained only on a single L.M. (3) We demonstrate strong transfer capabilities: zero-shot generalization to new datasets on LLMS that were seen during training, and fast adaptation to out-of-domain LLMS with only a newly trained linear adapter, keeping the remaining parameters frozen. (4) We show our method is highly efficient during both training and inference.
2 Related Work
Probing classifiers. In NLP, probing classifiers 7 are used to analyze model representations and reveal how features like syntax and semantics encoded [99 (0). Originating in interpretability research, they help identify which model components capture specific linguistic properties. E.g., lower layers capture local syntax in machine translation, while higher layers encode global structure [57]. Probes typically operate at the token and layer levels, with sentence-level tasks often using the final token's activation [T
29 61 While both linear probes and MLPs are common 25
linear probes are preferred for interpretability, as MLPs are often too expressive to distinguish whether the structure lies in the embeddings or the probe itself (2T) Although early interpretability studies favored simpler probes, more recent work uses probing as predictive or control tool and prioritize prediction efficacy. Probes have recently estimated L.LMs' downstream performance 770, 2 16 and
behavioral traits, e.g. sycophancy [49], or have guided models to desirable behaviors, e.g. reduced bias or improved factuality [65] 52 41 Still, many approaches use minimal setups, such as single token
from single layer fed into linear probe. Our work moves beyond these constraints by designing high- nigh-performing probes that better and more comprehensively leverage internal representations.
Error Detection in LLMs. LLMs produce diverse and complex mistakes and identifying these has become increasingly important. Hallucinations- a well-studied subset-ard are typically defined as outputs unfaithful to the input or external facts. Yet their definitions vary, extending to biases, We adopt broader view of error detection to account for this ambiguity, treating hallucinations one specific case. While we occasionally use the terms interchangeably. we recognize that hallucinations are only one facet of the broader error landscape. Most prior work approaches error detection using uncertainty estimates [29
reasoning flaws, and other failures [38 22 26 54)
[64]
33
42 or
probing internal representations- often focusing on the final answer token [29] 61
69
71
68
59
36 43
TO 53). Others probe the final prompt token to anticipate errors before generation
660 6 59
17 56
While effective, these methods under-exploit the richness of model activations.
We address
this by developing stronger methods that make fuller use of internal representations.
3 Notation, Problem Formulation and Setup
Given LLM M, input query and the I.L A..M-generated response ÿ. the goal is to predict whether I is correct or not. We assume a white-box setting, i.e. full access to the internal states of M, but disallow using external resources (such search engines auxiliary LLMs). We restrict the setting to single-pass prompting with no rounds of interactions due to their high cost.
We define the activation tensors the hidden representations within the LLM across all layers and all tokens in the output text Formally, the Activation Tensor (AT) of an LLM M is third-order tensor A Las x xDxr, where LMs is the number of layers in the model, N is the output sequence length, and D is the feature (hidden state) dimension; see Figure Throughout this paper, we focus on the hidden states each layer after the application of the feed-forward network and the residual connection, refer to Appendix A for details.
Activation Tensors.
Dataset construction. Following the setup of [48), we consider datasets D { 7.) consisting of m question-answer pairs, such as: ("hoo developed the theory of general relativity?",
TTii definition can be readily extended include the input text well.
3

---

"Albert Einstein"). For each query št, the model produces a response 9. and we extract the associated activation tensor A, yielding a collection -= We calculate binary hallucination labels, by comparing each response Ji with answer di to determine its correctness. We assign label = 1 for
correct response, 1/i otherwise. Our final activation dataset is A = {(A Yi. M.) 1m where M; is the LLM producing AT A; associated with hallucination label yt. We importantly note that multiple distinct M, 's from known collection M allowed within the same dataset.
Problem Statement. Our objective is to train parametric model mapping ATs to their corre- sponding hallucination labels. We additionally consider the following more general and challenging scenarios: (i) Off-domain data generalization: ATs are drawn from known model M MM, but correspond tasks or domains not seen during training (e.g.. training M's activation tensors from sentiment analysis, but testing on trivia question answering). (ii) Off-domain model generalization: ATs are drawn from previously unseen LLM M & M.
4 Towards Better, Non-Static Probing
In the context a single, target LLM, Orgad et al. [48 uncovered the importance of probing on "exact- token" position which may vary significantly on per-sample basis and generally requires external algorithms, such as multiple expensive queries to an LLM, in order to be identified. Complementary this, the results in 5 suggest degree of variability in the position of the best probing location also across layers. We consolidate these observations and further extend them by examining cross-LLM variability for fixed target datasets, further underscoring the need to go beyond static probing.
Inspired by the analyses by Or- gad et al. (48) on Mistral-7B- Instruct-v0.2 27 (Mis-7B), we train distinct logistic regression classifiers for each layer-token combination and construct ROC- AUC heatmaps to visualize the presence and localization of hal- lucination signals. We con- duct this analyses across three distinct LLMs and five differ- ent datasets. Since sentence lengths vary, we restrict tokens N {0, 1, -3. -2 -).. namely the first and last three in each sentence. Heatmaps are pre- sented in Figure 2 for Mis-7B, -lama-3-8B-Instruee
LiaMa Movies
leaihiiees 1417
0.750
Owen- 7B Movies
0.755
0.75
0.65
0.70
0.673
0.60
0.75
0.m
20
0.50
25
25
0.50
0.55
0.575
0.5
0.50
Tokere
Tokere
Tokens
Figure 2: Test AUC heatmaps across layer-token combinations; best layer-token combinations are boxed.
(Qwen-7B), over the Movies dataset [48 (we refer to Appendix AA5 for their full set). Figure shows predictive signals spread over multiple layer-token combinations, but it clearly indicates the most informative positions vary significantly across the three different LLMs, with distinct patterns characterizing each heatmap. In particular, the most predictive signal in terms layers appears around the middle of the model for LlaMa-8B and Mis-7B, whereas for Qwen-7B it is concentrated in the final layers. As for token positions, the signal tends to be strongest at the end of the sentence for both LlaMa-8P 8B and Qwen- 7B, while for Mis-7B it is more prominent towards the beginning. Concretely, note how the best probing position for Mis-7B, i.e., (14, 0), delivers suboptimal results on both LlaMa-8B and Qwen-7B, where the gap with their respective best probing locations is z 7% AUC both.
63 (LlaMa-88) and Qwwen2.5-B--. -n--truuttt
Towards Better Probing Techniques. Traditional probing methods rely on fixed layer/token selections. We claim this limits their ability to capture the dynamic nature of error signals, whose position vary across layers and tokens. This rigidity also hampers generalization across LLMs and tasks. We believe that more effective approach should adaptively attend to relevant activations while remaining efficient. To this end, we introduce and present next an architecture that processes the full AT A, enabling flexible and powerful HD across models and datasets.

---

4.1 Learning on Activation Tensors
Our starting observation is that Activation Tensors A E R are highly structured objects: they span two sequential dimensions and a feature dimension. The former two correspond LLM layers (Im) as well as generated tokens (N), forming structural map of neural activity across text positions and processing depths. The latter corresponds, instead, to the dimension of the LLM's hidden states (Dar). We note that this structure is remarkably analogous to that of images: layers and tokens are in correspondence with their vertical and horizontal spatial dimensions, the hidden states with channels (e.g.. RGB). Based on this analogy, we refer to the layer-token dimensions (LM, N) as the sputial activation dimensions, and each layer-token location as an activation pixel. Accordingly, we propose to process AT's similarly to images, designing our method using techniques inspired by modern vision architectures and guided by the principles outlined below.
Guiding principles. We follow two key principles in designing architectures for Activation Tensors: (i) cross-LLM generalization and (ii) computational efficiency. Cross-LLM, capability is especially crucial for hallucination detection, where annotated data is scarce, costly, or unavailable for new models. We aim to build model that learns from multiple LLMs and generalizes effectively to unseen ones, without incurring high computational cost. Principle (ii) poses a major challenge: Activation Tensors are large. For example, a transformer with L = 50 layers. N = 512 tokens, and D 4096 hidden dimensions produces a single AT of 0.2GB in float16: a batch of 64 totals s 12.8GB. Addressing this scale is critical for real-world use. The next section introduces ACT-ViT, an architecture designed to handle AT's efficiently while meeting the principles above.
4.2 ACT-ViT
Overview. ACT-ViT consists of the following components (see Figure D:
(1) a Pooling (Pool) layer that reduces the "spatial" dimensions of the activation tensor (layers and tokens) to fixed, predefined size (Lp, Np): (2) a per-LLM Linear Adapter (LA) layer that compresses and "aligns" their feature dimensions; (3) ViT-Based Backbone (ViT-B), which processes the resulting tensor using Vision Transformer architecture, enhanced with shared positional encodings within each patch. In what follows, expand on each of these components.
Pool. Similarly to image resizing, we propose to reduce the size of activation tensors before processing them with our architec- ture: as a preprocessing step, we apply a max-pooling operation across the spatial activation dimensions (see inset illustration). Specifically, we propose two possible pooling approaches. The first maintains full input resolution the level of tokens and applics 1d pooling on the layer dimension only. This approach preferred when expecting relatively concise L.M generations, or when their maximum generation length remains computationally manageable. Alternatively, a second approach adopts a more general 2d pooling both the spatial activation dimensions, allowing to more efficiently deal with potentially longer LLM responses. As we will detail next, we will mainly adopt the first approach, and additionally experiment with the second in Section 5
Dyg
Pooling
DH
LA
N
Note that, in our applications, not only the number of generated tokens N naturally varies de- pending on the input prompt, also, the number of LLM layers may vary across models in M. With guiding principle (i) in mind, it becomes important to ensure our architecture can robustly deal with both these varying parameters. Accordingly, we design the two above pooling opera- tions in a way that, regardless of the input size, their outputs, A" E R N2x Ds, always match predefined shape of (I,, Np) in their spatial dimensions (see Algorithm in Appendix B Importantly, this property ensures the downstream VIT back- bone is always fed with tensors of consistent spatial shape.
Linear Adapters
D
LA. Following pooling, we map the feature dimensions to a shared hidden size via a linear transformation. In particu- lar, we propose to instantiate and employ dedicated linear adapter (I.A) module for each L.MM M E M; see inset illus- tration. All adapters project to shared output dimensionality
LA
D
LAhy
5

---

D'. Formally:
L C-(LAM AP M WM M E RRDDDD M eMM.
where DM the hidden dimension of model M, and AP denotes the pooled activation tensor. The result is a tensor of shape Lp x N, x D' This appiin-g a single linear transformation, LAM jointly across all layers and tokens of the activation tensor for given LLM M, rather than using separate projections per layer or token- is motivated by the inherent alignment of feature spaces within the architecture of single transformer model. Specifically, residual connections promote consistency across layers, while the weight sharing of the transformer across tokens ensures a uniform representation structure along the token dimension.
Resorting to a set of LLM-dependent LAs is particularly relevant in relation to principle (i). Indeed, this expedient ensures that the downstream architectural components process objects which are of consistent shape and are meaningfully comparable across the internal representation spaces of the distinct LLMs in M. Indeed, motivated by recent literature on representation alignment 24 45 46 (2), we hypothesize that, despite differences in architectural design, training corpora, weight initializations and learning schemes, the hidden representations of LLMs can, in fact, be put in an approximate correspondence by some linear transformations, which we aim to learn via the adapters in L. This contributes to rendering our multi-LLM setup not only computationally possible, but also meaningful from a learning perspective and, as we show in the next sections, also beneficial in terms of generalization performance. For an illustrative motivating example of how a linear mapping can tackle misalignment in the representation space, we refer the interested reader to Appendix C where we specifically consider those induced by neuron symmetries, as recently studied in 37
VTT-B. In view of the proposed analogy between ATs and images, the output of any LA module is then processed à la VIT [14]. First, it is rearranged into a sequence of non-overlapping activation patches of size (PH.PW)) If Tup and N, are divisible by, resp.. ps and pw. this gives H = p/pp and W = N patches. The resulting reshaped tensor is denoted: Aatches E R(H ((HH-W)xpp) Xwt--W))xxxxxxxxx xpw xD'
We augment each patch with per-patch learnable Positional Encoding (PE), providing intra-patch ordering over (pooled) activation pixels. The Transformer's native PEs provide global ordering over activation patches. Combined with per- ver-patch PEs, this allows reconstructing each pooled pixel's original position. Similarly to VIT, the enriched activation patches are flattened and passed through shared linear layer. The resulting sequence is then processed by stack of Transformer blocks. Overall, using a vision-like backbone aligns with the aforementioned analogy between ATs and images and, will experimentally show next, it reveals to be effective inductive bias.
5 Experiments
Setup. We experimentally analyze various aspects of learning on ATs with ACT-ViT. We consider, in particular, two relevant setups: (a) in-domain and (b) out-of-domain.
The in-domain setup (a) is the setting commonly considered in HD literature: it assesses the gen- eralization performance on data from a known LLM on known generative tasks, i.e., it assumes training-timm access to annotated data for a target LLM and task of interest. In this standard sce- nario, we attempt answering the following questions: (Q 1) Does our method outperform standard methods, including probing classifiers? (Qa) What is the contribution of multi-dataset training? (Qa)3) Is the vision architectural inductive bias effective? (Qu)4) How scalable is our approach?
Harder but more compelling iss the out-of-domain setup (b). Here, we benchmark methods in cases where either the target dataset is unseen training time, or, even more challenging, both the target dataset and the target LLM are unseen. We study, in particular, the performance of our approach when either no data or only a small amount available for adaptation. Those correspond, resp., to zero-shot generalization, and to fine-tuning only the LA in our pipeline, keeping the other parameters frozen (see inset). These settings reflect relevant real-world use-cases
LLM-specific Pooling
Shared
Adapters
VTT-B
LAg
Frozen
:
LAX

---

pertaining to, e.g., newly introduced LLMs and/or generative tasks with hardly accessible halluci- nation labels. Our experiments revolve around the following questions: (Qa) Can our method effectively generalize on unseen datasets? Does it adapt in a sample-efficient manner? (Qm)) Can it handle unseen L.LMs by only training the corresponding Linear Adapters? (Qi))) Does this approach favorably compare with baselines?
LLMs and datasets.
Aligning with prior work, focus on the datasets and LLMs considered We experiment with --irral-----sttutt--o2222 [27] (Mis-7B) and Llama-3-8B 3-------sBttuue
in 48 6].
63
(LlaMa-8B) over the generation tasks encompassed by the following datasets: TriviaQA (28)
(question answering), HotpotQA with (HQA-Wc) and without supporting context (HQA) 67
(question answering), IMDB movie review [0) (sentiment analysis), and Movies [48] (actor role
retrieval). On top of this, we additionally include another LLM: (wwe2.55-BBIInstmun
[66] (Qwen-
7B). This setup yields total of 15 distinct LLM-dataset combinations. We use the area under the
ROC curve (AUC) to evaluate error detectors, a standard metric in this domain [8 61
Methods in comparison. We compare the performance of diverse set of methods: (1) Prob- abilities/loiits-aasse basee. To detect errors, several works [ 29 64 23 assess LLM confidence by aggregating output token probabilities or logits by taking the mean, max, or min across these scores. We refer to these with the naming aggic/Probas-mmaammnnnnne less-restrictive setting, and thus constitute important baselines for our method. They are especially relevant in the out-of-domain setup (b), since, being training-free, these methods are (degenerately) sample-efficient by construction. We also compare against LOS-Net [6). recent, learnable ap- proach on output probabilities. (2) Probing Classifiers (7) constitute natural and effective white-box baselines for our approach [29] 36.. For each benchmark, we learn logistic regression probes on each layer of the LLM and across a fixed set of token positions N (see Section 4. a superset of the top- pep-efforrmin (static) tokens identified in 48). The best layer-token combination, as well as the L2 regularization coefficient, is selected on the respective validation sets. We refer to this method as Probe[=]. With Token[n]. n .N, we refer, instead, to the probe specific to token position n. (3) Activation Tensors-Methods are those designed and introduced in this work. ACT-ViT, (Section 4.2) is trained on the corpus of all available training sets across LLMs and datasets. In the out-of-domain setup (b), some of these are purposely segregated from the corpus in an effort to study domain adaptation abilities; this will be appropriately specified. We also benchmark two variants of our approach to separately study the benefits of multi-dataset training and architectural inductive biases. We use ACT-ViT(s) to refer the single LLM-dataset counterpart of our method trained individually on each of the 15 ALM--ataset combinations. ACT-MLP and MMP-MLP(s) denote, instead, simplified variants in which MLPs operate on the output of Pool, flattened. These are trained, resp.. on the full training corpus or separately on each single training set. Across all these architectures, if not specified otherwise, we use the fixed pooling hyperparameter of (Tep. N,) = (8, 100).
These approaches operate in a
Next we present our main results, with additional experimental details and experiments in Appendix
5.1 In-domain Hallucination Detection
Comparing detection methods in-domain. Table ! reports the performance of the aforementioned methods across all the 15 LLLM-dataset combinations. The last row reports the gap between ViT and the best-perforning baseline from prior work. First, our methods outperform both probability- based and probing classifiers on all settings, with the only exception being IMDB over Mis-7B. There, they closely follow the best scoring Probe[*]. Additional comparison with LOS-Net is available in Table 7 in Appendix A.3 These results yield a positive answer to (Qia) underscoring how leveraging the full activation tensor can significantly benefit automated HD. Second, note that ACT-ViT outperforms -ii(s) in 12 out of 15 cases, sometimes with a remarkable margin. These outperform training on
results demonstrate that joint training, as implemented in ACT-ViT, individual LLM-dataset combinations, as in ACT-ViT(S) The notable performance of ACT-ViT suggests that the underlying signal for HD is shared across different LLMs and datasets, and that a simple linear adapter (module LA) can already effectively bridge these. Answering (Qia2), we conclude that training on Activation Tensors sourced from diverse datasets and LLMs holds significant potential, and that our method can effectively harness it. Last, we intriguingly note how the ACT-MLP
With appropriate padding to Dmxx the largest hidden size among the considered LLMs
7

---

Table 1: AUC HD performance comparison on all 15 LaMs-Datasets combinations. The best test AUC score is indicated in Bold, and the accond-best is underlined. The last row reports the improvement achieved by ACT-ViT over the best-performing baseline from prior work.
Mis-BB HR
LlaMn-811 HRU--WE
Quen-7B HRAA-W
Movies
IMD0
HI2A
Movies
IMDI
TIvaQA
HQA
Movies
IMD0
ThvisQA
Probus hased
Legisomon
61.00
65.00
55.00
57.00
65.00
65-00
75.00
56.00
59.00)
66.00
66
71. 10
67.40
74.80)
68.20
Logits max
53.00
36.00
51.00
47.00
54,00
39.00
07.00
55.00
51.00
54,00
60.43
65.10
60.00
50,70
63.90
Logis min
01.00
06.00
33.00
52.00
63.00
07.00
71.00
55.00
55.00
74,00
39.80
42.10
35.50
72.10
01.00
00
69. 70
probes max
72.90
protes
52.00
51,00
00.00
51.00
57,00
67.00
54.40
44.70
53.60
57,40
Learnigghhwrd
Tolenn (0)
$2.69
75.59
PM
97.61
79.95
79.00
75.84
02:22
93.58
80.13
60.45
86.
64.83
75.
Telern [1]
50.65
75.02
61.00
965.55
77.20
79.71
72.70
64:21
69.58
79.81
19.25
86.47
64.33
10..9
77.24
Telen [2]
79.38
75.46
611.90
97.46
76.76
79.60
77.75
66.24
99.51
70.90
79. 14
87.98
02.62
89.11
70.77
Toien 31
74.19
09.17
61.34
92.91
74.25
70.75
75.76
63.11
52.26
70.05
79.29
85.14
71.55
98.57
84.12
Toien (:2)
76.05
70.95
63.88
941.54
75.47
75.11
79.85
66.95
51.93
75.15
78.24
90.44
73.75
92.54
81.93
Toorn [-1]
77.89
14.87
$6.891
043.9
75.83
72.95
$3.61
20.55
90:49
$1.00
$0.57
95:22
75.01
93.76
87.08
Probet+]
79.85
74.87
65.80
97.61
79.95
79.71
33.61
70.36
93.58
$1.09
$0.57
93.22
75.01
93.76
67.08
AG7 HLP(3
78.76
70.66
39.51
93.05
78.25
77.95
76.55
65.15
91.22
75.23
$2.57
87.47
72.57
89.07
86. 10
AGT- HLP
78.41
14:25
60:57
78.11
77.77
76.35
63.25
90.23
75.61
82: 2090
87.45
72: 13
90.00
83-53
AC7-V1760
93.62
28.79
97.59
93.37
31.27
83.23
93.71
94.25
$8.03
94.59
76.44
94.71
91.01
AST-VST
70.25
8481
X1.02
95.08
78.26
Improvennnt
baselines are often outperformed by Probe[*]. This clearly answers (Qia)3) and shed light on the pivotal role of the design choice behind ACT- V.T.
Run-time and performance trade-offs. The inference run-time for ACT-ViT and ACT-ViT(s) is extremely contained, around 10 seconds. This is orders of magnitude faster than techniques resorting to LLM-querying [33 48). The training time of ACT -iiT on the full training corpus (15 LLM-dataset combinations) is below three hours on single NVIDIA L-40 GPU, Our competitive ACT-ViT(S) trains, instead, in 10 minutes. For detailed run-times see Table 9 Appendix (A.4.
One may wonder if it is possible to further reduce the run-time of our method with more aggressive Pool. We study this by testing different configurations on Qwen- 7B over HotpotQA. We train ACT-ViT(s) varying the number of pooled layers and to- kens: (L,, Np)
{[4, 20), (4, 100), (8, 20), (8, 100)]. Results, shown in Figure 3 are reported in terms of Test AUC (left) scores and corresponding training run-times (right). As expected, Npp) improves performance, but the cost of longer training. Notably, however, configuration (Ips Np)
increasing
Trais time
87:37
$8.03
124
$5.96
85.86
(4, 20) trains in around one minute while still achieving an AUC 86%, R-6 units higher than Probe[*]. In respect to (2,,)4), we conclude that our approach is indeed very
by, 16.300
U.
2200
He. 10. (9-23) U.,
12t
NO 1901
Figure 3: Ablation study on the pooling hyperparams (Ipp Np). Probe[*] AUC is indicated by dashed green line in the left plot. scalable, and that operating on simple design choices can provide optimal performance- trade-ofit
5.2 Off-domain Hallucination Detection
Hallucination Detection on unseen datasets. We implement (laave-one-datsett each LLM-dataset pairing, we train ACT-ViT on the corpus of the remaining 14 out of 15 combina- tions, and then evaluate on the target held-out test set zero-shof"] This something fundamentally not possible with standard probing methods and hence compare against the best probability-baas baseline, denoted Best-Probas. due to its training-free nature, and against -MP.
setup: for
Results are shown as bar plots in Figure 4 on all 15 LLM-dataset combinations, We observe strong generalization performance: ACT-ViT outperforms Best-Probas in 13 out of 15 cases, sometimes substantially. On the IMDB dataset with Mis-7B, ACT-ViT achieves gain of +37 AUC points.
*On huld-out LLM-data pair (M, this is possible M's LA is pretrained on the pairs (M, D'), D' DDD
8

---

ACT-MLP also shows reasonable performance, but it consistently falls short of ACT-ViT. These results provide a positive answer (Qm))).
ACTVTT
ACT-MLF
Biot Pcot-Pruous
Mis-78
Qwee--BB
(M
3L
AUic
30
Hogong
Brvagg
Mexied
Hopprt
Hopolga
Mevid
Happot
Holpopo
Bivage
Movidl
ANCE
HepoO4.90 Sec
Figure 4: Zero-shot generalization results across all 15 LLM-dataset combinations, in "leave- one-dataset-outt setup. Each bar shows the Test AUC score of ACT-ViT, ACT-MLP, and the best probability- based baseline Best-Probas on a dataset they were not trained on.
We next consider a more tolerant setting whereby only limited amount of samples from the target training set is available. In particular, we start with the pre- trained ACT-ViT and fine-tune the LA module keeping the VIT-B component frozen. We compare with Best- Probas and Probe[*]. fitted only the corresponding avail- able training data. Figure Shows Test AUCs for Mis-7B over HotpotQA across varying fine-tuning set sizes; re- sults for other 15 LLM-dataset combinations are in Ap- pendix A.6 ACT-ViT consistently outperforms all base- lines across all training fractions. Importantly, the perfor- mance gap between ACT-ViT and Probe[=] widens as the training set shrinks, indicating that the frozen ViT-B back- bone captures strong, transferable features learned from (pre ttraining other LLMs and datasets. Additionally, ACT-ViT requires only 10% of the available training data (1k samples) to surpass Probe[*) #] when trained the full 10k-sized training set.
Mehhod
BaakPrsbe
Prote!!
ACT-VIT
ACT9L7
O-shot
AUC
0-shot
g+
10%
20%
5e%
100%
Fine-tune tune sites % tor 10k)
Figure 5: Low-data regime results, for
Mis-7B over the HotpotQA dataset.
Hallucination Detection on unseen LLMs and datasets. We finally benchmark our method in the following challenging setup: we train ACT-Vit ViT on 2 out of 3 LLMs and leave the third one out. Specifically, our model is pretrained on corpus of 10 out of 15 training sets containing no data from target L.LM M. It is then (separately) adapted on the remaining datasets by keeping ViT-B frozen and only training the LAM module from scratch. In Table 2 we report the obtained Test AUCs, comparing ACT-ViT against Best-Probas and Probe[*]. ACT-ViT outperforms these baselines in 13 out of 15 cases, Recalling that the ViT-B module is frozen during adaptation, this further demonstrates how the features learned during (pre)training over different LLMs can strongly transfer to unseen one. Along with the above, these results answer positively to 21oi2 and 20y,.
Table 2: AUC comparison with only the LA fine-tuned (ViT-B frozen) on an off-oin LLM and
dataset. Last row shows ACT Vii'' gain over the best baseline from prior work.
Mi-7B HQQA-W:
LaMa--B HIQA-Wc
Qwen-7ft Iiqu-Wc
BetpurQA Mosice
BMDB
TrivaQA
Mexiox
DMBS
TriossAA
BetputQ
Mosicx
IMD0
'IrvorQ
Best Probe)
63,00
57.00
63.00
65.00
75.00
23.00
74.00
50.00
50,00
$0.00
50,00
87.69
24.87
97.41
79.95
7631
8141
70.56
91.56
$0.15
30.57
91.32
75.01
91.16
$7.00
ACT- MLL
86.22
22.83
50.6
0535
79.64
75.63
25.51
54.50
85.30
76.55
$3.40
85.03
73:64
$9.14
$6.34
ACT--117
83.75
7907
6887
97.58
63.04
98.33
80.31
70.66
96.71
$2.63
87.37
95.00
77.54
96.35
91.14
6 Conclusions
We introduce ACT-ViT, fast and effective method for L.M Hallucination Detection (HD) that relies solely a model's internal representation of single response. Unlike traditional probing,

---

ACT-ViT leverages the full Activation Tensor (AT) and, drawing on an analogy with images, it applies techniques from modern vision architectures, enabling both cross-LLM training and generalization. It is dramatically faster than multi-query methods (runtime R 10 seconds), making it suitable for real-time use. Across 15 L.LM-dataset pairs, ACT-ViT consistently outperforms prior white-box methods, supports zero-shot generalization to new datasets, and adapts efficiently to unseen LLMs. Our work opens several directions for future research. While focused on HD, ACT-ViT can be extended to tasks like data contamination and LLM-generated content detection. More advanced architectures also warrant exploration, e,g., replacing per-LLM adapters with a shared module that exploits permutation symmetries across ATs..
Limitations. Due the large size of the ATs, we begin their processing by conducting a simple pooling operation to reduce their dimensionality. While effective in managing computational costs, this step may discard potentially informative signals. Developing more sophisticated approaches to handle AT size without sacrificing signal fidelity remains promising future direction.
Ackowledgements
G.B. is supported by the Jacobs Qualcomm PhD Fellowship. F.F. conducted this work supported by an Aly Kaufman Post-Doctoral Fellowship. H.M. is a Robert J. Shillman Fellow and is supported by the Israel Science Foundation through a personal grant (ISF 264/23) and an equipment grant (ISF 532/23). F.F. extremely grateful the members of the "Eva Project", whose support he immensely appreciates.
References
[1] Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained anal- ysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv: 04207, 2016.
[2] Anum Afzal, Florian Matthes, Gal Chechik, and Yftah Ziser. Knowing before saying: LLM representations encode information about cfain-of-thought success before completion. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACI 2025, pages 12791-12806, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-891766 256-5. doi: 10. 18.18//3/v 1/255/////225..... /2022 findings--cll.666 URL aattps://aclantoon findings- acl. 662/
2025
[3] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint wwX.iv:22090488000 2022.
[4] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018. URL https: / www.wiiww.....a.eoo..ooe ort.:/..gg////b//r 1ttp///as/1610.1.111
[5] Amos Azaria and Tom Mitchell. The internal state of Ilm knows when it's lying. arXiv preprint arXivv2304.137. 2023.
[6] Guy Bar-Shalom, Fabrizio Frasca, Derek Lim, Yoav Gelberg, Yftah Ziser, Ran El- Yaniv, Gal Chechik, and Haggai Maron. Learning on Ilm output signatures for gray-box Ilm behavior analysis. arXiv preprint 2tt.::2503.1444:. 2025.
[7] Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48( (1)1)2207-219 2022.
[8] Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey, Transactions of the Association for Computational Linguisties, 7:49-72, 2019.
[9] Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https : //www www wandb.com/ Software available from wandb.com.
[10] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXhv:2212.03887, 2022.
10

---

[11] Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Lims' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.033 2024.
[12] Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single S&!# vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-11 1198.. URL httpt::://ccaaaaaaaaaaa httpa/acaaaatooogyyy.omoo http::////aaaaaaloooooooo/////-0
[13] Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can eram into single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv: arXiv:/805.00 /805.01070 2018.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /9999 1929, 2020.
[15] Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st workshop on evaluating vector-space representations for nip, pages 134-139, 2016.
[16] Shaona Ghosh, Amrita Bhattacharjee, Yftah Ziser, and Christopher Parisien. Safesteer: In- terpretable safety steering with refusal-evasion in 1lms. arXiv preprint urXiv:2506.042500 2025.
[17] Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without generating single token. arXiv preprint arXiv:2406. 12673, 2024.
[18] Nuno M Guerreiro, Elena Voita, and André FT Martins. Looking for needle in a haystack: comprehensive study of hallucinations in neural machine translation. arXiv preprint arXiv:2208.05309, 2022.
[19] Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. Distributional vectors encode referential attributes. In Lluís Mårquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 12-21, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1 1002. URL
[20] Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. Distributional vectors encode referential attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 12-21, 2015.
[21] John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. arXiv preprint arXiv: 00360.33...... 2019.
[22] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang. Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2023.
[23] Yuheng Huang. Jiayang Song, Zhijie Wang. Shengming Zhao, Huaming Chen, Felix Xue and Lei Ma. Look before you leap: An exploratory study uncertainty measurement for large language models. arXiv preprint 2tti::2337.1123366 2023.
[24] Minyoung Huh. Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint 040Xi:2200000788777. 2024.
[25] Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and' diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926, 2018.

---

[26] Ziwei Ji. Nayeon Lee, Rita Frieske, Tiezheng Yu. Dan Su. Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
[27] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06225, 2023.
[28] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.
[29] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint
2022.
[30] Hassane Kissane, Achim Schilling, and Patrick Krauss. Probing internal representations of wordi-word verbs in large language models. In Atul Kr. Ojha, Voula Giouli, Verginica Barbu Mititelu, Mathieu Constant, Gražina Korvel, A. Seza Dogruöz, and Alexandre Rademaker, editors, Proceedings of the 2 Workshop on Mulliword Expressions (MWE 2025), pages 7-13, Albuquerque, New Mexico, U.S.A., May 2025. Association for Computational Linguistics. ISBN 979-8-89176-243-5. URL htt..ttttt//cccaaatlllleeoee .r..org////00 wtrg/2025.cco mww- 1.2
[31] Arne Köhn. What's in an embedding? analyzing word embeddings through multilingual evaluation. In Lluís Mårquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2067- 2073, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: (0.18653/1/D15-1222 1246. URL ptctt:::///lllatthoooooooo////// 1246/
[32] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019.
[33] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.0900 2023.
[34] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation, 2023. URL htttp://///: org/ abs 0bs/2302.00m 2366449622.66
[35] Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449-6464, Singapore, December 2023, Association for Computational Linguistics. doi: 10.1 18633v 1/2022...mmppmmaaii... acl aclanthology org/2023. emnlp-main 397/
URL https://
[36] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024.
[37] Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855, 2019.
[38] Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. A token-level reference- free hallucination detection benchmark for free-form text generation. arXiv preprint 204.00204, 2021.
[39] I Loshchilov, Decoupled weight decay regularization. arXiv preprint 111.:11111.00001111 2017.
12

---

[40] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts, Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150,
2011.
[41] Sharan Maiya, Yinhong Liu, Ramit Debnath, and Anna Korhonen. Improving preference extraction Ilms by identifying latent knowledge through classifying probes. arXiv preprint arXiv:2503. 00003//7750 /7755, 2025.
[42] Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resoon kero-resource black-bor hallucination detection for generative large language models. arXiv preprint 0rXiv:2303.088%6 2023.
[43] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
[44] Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell. A tale of a probe and a parser. arXiv preprint arXiv:2005.01641, 2020.
[45] Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
[46] Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209. 15430, 2022.
[47] Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference Machine Learning, pages 25790-25816. PMLR, 2023.
[48] Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. Llms know more than they show: On the intrinsic representation of Ilm hallucinations. arXiv preprint arXiv:2410.02707, 2024
[49] Henry Papadatos and Rachel Freedman. Linear probe penalties reduce Ilm sycophancy, arXiv preprint arXiv:2412.00090 2024.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[51] Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay Cohen. Detecting and mitigating hallucinations in multilingual summarisation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8914-8919999 1932,, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.188811112223. eee.mmmmlppmmm. 551. URL https
//aclanthology. org/ 2023. emnlp main. 551/
[52] Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Maria Ponti, and Shay Co- hen. Spectral editing of activations for large language model alignment. Advances in Neural Information Processing Systems, 77:56958-56987 2024.
[53] Miriam Rateike, Celia Cintas, John Wamburu, Tanya Akumu, and Skyler Speakman. Weakly supervised detection of hallucinations in Ilm activations. arXiv preprint arXiv:2312.02798, 2023.
[54] Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, SM Tonmoy, Aman Chadha, Amit P Sheth, and Amitava Das. The troubling emergence of hallucination in large language models-an extensive definition, quantification, and prescriptive remediations, arXiv preprint arXiv:2310.04980 04988, 2023.
13

---

[55] Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. comprehensive survey of hallucination in large language, image, video and audio foundation models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguisties: EMNLP 2024, pages 11709-1 11724, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.1863//1/2024 findings-emmll.685.. URL https://aclantboology htt.a/aawwwooooooooo/2222/// findings emnlp. 685/
[56] Yeongbin Sco, Dongha Lee, and Jinyoung Yeo. Detecting hallucination before answering: Semantic compression through instruction, 2025. URLLpttps://oparreviee.. wttp:/rrwwiiiiieeefforrre. 1d-YFOg1LUGG1
[57] Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural MT learn source syntax? In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526-1534, Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10.18653/v1/D1-- 11599166911,,, URL fttps://accanthoonn
[58] Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural mt learn source syntax? In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1526-1534, 2016.
[59] Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. Constructing benchmarks and interventions for combating hallucinations in llms. arXiv preprint arXiv:2404.00997 2024.
[60] Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3607-3625, 2023.
[61] Ben Snyder, Marius Moiseseu, and Muhammad Bilal Zafar. On early detection of hallucinations in factual question answering. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2721-2732, 2024.
[62] SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXi::2401.0131. 2024.
[63] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 130..1991.. 2023.
[64] Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint wxXiv:2307.03987. 2023.
[65] Dimitri Von Rütte, Sotiris Anagnostidis. Gregor Bachmann, and Thomas Hofmann. A language model's guide through latent space. arXiv preprint arXiv:2402.1443 14433, 2024.
[66] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671. 2024.
[67] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William w Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 2018.
14

---

[68] Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. Characterizing truthfulness in large language model generations with local intrinsic dimension, arXiv preprint arXiv:2402.18048 2024.
[69] Mert Yuksekgonul, Varun Chandrasckaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint-satisfaction lens on factual errors of language models. arXiv preprint arXiv: 2023.
[70] Zining Zhu, Soroosh Shahtalebi, and Frank Rudzicz. Predicting fine-tuning performance with probing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11534- 11547, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 00555//1/2222..m1pmmmoee 7933. URL htttpp:://acaaattoologg.. org/ 2022. wmnlp-main. 793/
[71] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Katee Kathrin Dombrowski, et al. Representation engineering: A top-down approach to transparency. arXiv preprint urXiv:2310.014000 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are justified in Sections and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, see Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include any formal, explicit theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are provided in Section and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: The code to reproduce all our experiments provided in /https://githu: Bttp//arrSuyyyC-TTTSTS
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https Coc//ooodeeesiseeeeeeeeee hwtpbbbmsssiooooooooiooo for more details.
/nips..cc/pulli//uuui
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All necessary experimental details are available in Section 5 and Appendix A
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: As stated during submission, committed to including error bars in the final version upon acceptance. We have now added results averaged over 3 random seeds with 1-sigma error bars, reported in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All necessary experimental details are available in Section 5 and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper does not violate any aspect of the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix D
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All baselines as well as accessed code and data repositories have been cred- ited throughout Section and Appendix A For licenses and terms of please refer to Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper studies the task of Hallucination Detection in LLMs, but the core method development in this research does not involve LLMS as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, ntn-standard components.
Please refer to our LLM policy hattop///aauusgeccc///ooerrrrrrs for what should or should not be described.
22

---

refers to A Extended Experimental Section
Our experiments were conducted using the Py/Torch [50] framework (License: BSD), using single NVIDIA L-40 GPU for all experiments. We use a fixed batch size of 128 for all experiments, other than the ones with ACT-ViT(s), ACT-MLP(s), where used batch size of 64. We used 4 heads in the transformer part of ViT for all experiments. Hyperparameter tuning was performed utilizing the Weight and Biases framework
see Appendix A.I
Optimizer and Schedulers. For all datasets, we use the A optimizer 39 in combination with cosine learning rate scheduler, incorporating warm-up phase over the first 10% of training epochs.
LLMs. We consider the following L.LMs for our experiments:
1. -isstral77--insrruut--...2 [7 (License: Apache-2.0). Referred to as Mis-7B in the main text and accessed through the Hugging Face interface https::/hugggingaacee co/ mistralai/Mistrall 1at----I-nttuttt......
2. Llama-3-8b-Instruct 63 (License: Llama D Referred to as LlaMa-8B in the main text and accessed through the Hugging Face interface at https://huggingfee co/ alaaalmm-ma---- Instruct3
3. Qwen-2.5-7b-Instrues (License: Apache-2.0); Referred as Qwen-7B in the main text and accessed through the Hugging Face interface at htttpp:://uugingffaee ct/.co////aaooo co/Quen/
Qwen2.5- InnB-Instructt
Generating Activation Tensors. Given large language model (LLM) M and an output sequence of N tokens, we define the activation tensor A E IR f. XN Das where L is the number of layers in M. and DM is the hidden dimension. Let A ()) == All.; :| denote the hidden representations at the I-th layer across the entire sequence. In a standard transformer architecture, these representations are computed as:
A (1) A1-1)) + |RLLU (Attn(A(-)))) + +b/)] W? + b
where Attn denotes the output of the multi-head attention mechanism, followed by batch normaliza- tion. The matrices W,, W? E Pax P and biases b). by E RPM correspond to the feed-forward
subnetwork at layer L. Throughout this paper, the term activation tensor to refer A as defined in Equation
A.1 HyperParameters
This section outlines the hyperparameter search performed for our experiments. We employ the same hyperparameter grid for both our primary model, ACT-ViT, and our proposed baseline, ACT-MLF Hyperparameter grid search is performed to optimize the AUC on the validation set. The best hyperparameters are selected based on the run that achieves the highest validation AUC, for each of our two proposed methods, namely ACT-ViT, and ACT-MLP. The hyperparameter grids used for all experiments are outlined below, corresponding each experimental setup:
1. Training on single L..M-seccffic dataset see Table
2. Joint training on all 15 datasets simultancously - see Table 4
of 15 datasets) upper part of Table 4. Low-data regime adaptation on the held-out dataset lower part of Table slot which are inapplicable, taken from the pre-trained model.
3. Leave-one-out training (excluding
5. Leaving out LLM (excluding 5 out of 15 LLM-dataset combinations) see Table 6
For the probing basclines dasclines-specificclll Token [m] for n E N (see Section 4 and Probe[#] performed grid search over inverse regularization strengths C' {10000, 100, 1.0, 0.01, 0.0001).
For each token probe, we selected the best-performing model based on validation set performance, corresponding to the optimal value of C. In the case of Probe[+], we additionally selected the best token position using the validation set.
co/meta- waalllaamaaaa /Ntta Llama 3- 'hhe patch size parameter does not apply to these baselines, as it is not part of their contiguration.
23

---

Table 3: Hyperparameter search grid for ACT-ViT(s) and ACT-MLP(S), for each of the 15 LLL- combinations.
Hyperparameter Search Grid for each of the 15 LLM-dataset combinations
Number of layers
{1.3}
Learning rate
0.001
Embedding size Epochs
{128, 1024]
15
Dropout
0.3
Weight Decay
(1,0.001)
Patch Size
{L1, 1), (8, (4,
Table 4: Hyperparameter search grid for ACT-ViTand ACT-MLP, in the setting where training is performed jointly across all 15 LLM-datase dataset combinations.
Hyperparameter Search Grid for joint-training of the 15 LLM-dataset combinations
Number of layers
Learning rate Embedding size
{0.001, 0.0005]
128
Epochs
Dropout
0.3
Weight Decay
{10, 0.001]
Patch Size
(1,2), (2, (4,)
Table 5: Hyperparameter search grid used for ACT-ViT and ACT-MLP when trained on the combined of all 14 LLM datasets, in the "leaving one dataset out" setup. "_" denotes slots that inapplicable, as their values are inherited from the pre-trained model.
Num. layers
Learning rate
Embedding size Epochs
Dropout
Weight Decay
Patch size
Pre training (and Zero-shot)
3
0.001
128
15
0.3
0.001
(1,
Low-data regime LA adaptation (over (5%, 10%, 20%, 50%, 100%) of 10,000 test samples)
0.001
5
0.001
Table 6: Hyperparameter search grid used for ACT-ViT and ACT-MLP when trained on the combined set of 10 LLM datasets, in the "leaving one LLM out" setup. "_" denotes slots that are inapplicable, their values are inherited from the pre-trained model.
Num. layers
Learning rate
Embedding size
Epochs
Dropout
Weight Decay
Patch size
Pre training
(3,5)
0.001
128
0.3
0.001
[(1, (1, 1), (4,2))
LA Adaptation
0.001
15
(0.1,0.01,10, 20}
A.2 Dataset Description
In this section, we provide an overview of the five datasets used in our analysis; we mostly follow the framework given in 48 in constructing the datasets. We aimed to cover diverse tasks, reasoning skills, and datasets, highlighting each one's 's unique value and how it complements the rest.
For all datasets, we used a consistent split of 10,000 training samples and 10,000 test samples, unless otherwise specified. From the 10,000 training samples, 20% (i.e., 2,000) were selected in stratified manner for validation, using fixed random seed of 42.
24

---

1. HotpotQA with and without context (License: CC-BY-SA-4.0) [7]]
HotpotQA is multi-hop question answering dataset featuring diverse questions that require reasoning across multiple sources. Each instance includes supporting Wikipedia documents. We use two settings in our analysis: (1) Withou context, where questions are presented alone, testing factual recall and reasoning: and (2) With context, where supporting documents are provided, emphasizing the model's ability to leverage contextual information effectively.
2. Movies 48 (License: MIT): We use this dataset to evaluate generalization in scenarios regarding movies involving factual inaccuracies (i.e., hallucinations). This dataset contains 7857 test samples.
3. IMDB (originally released with no known license by Maas al. (40] ) This dataset consists
of movie reviews for sentiment classification. Following the method in [48], we employed
one-shot prompt to help the large language model (LLM) apply the predefined sentiment
labels accurately.
4. TriviaQA (originally released with no known license by (28): answer pairs presented to the LLM without any supporting context, relying only on its internal parametric knowledge. Multiple acceptable answer variants are provided to facilitate automatic evaluation of response accuracy.
): A dataset of trivia question-
A.3 Additional Results
Below in Table we provide a comparison of ACT-ViT with LOS-Net.
Table 7: Comparison of ACT-ViT with LOS-Net. For LOS-Net, we report the 1-sigma error-bars. Best is in Bold.
Method
HorporQA
IMDB
Movics
HotporQA
IMDB
Movies
HotporQ
IMDB
Movics
Mis-7B
LlaMa-8B
Qwen-7B
LOS-Net
72.92 0.45
94,73 8.58
72.20 a 0.66
72.60 a 0.34
90.57 0.3
77.43 0.66
73.71 1.21
88. 19 0.0
88.01 0.3
ACT-ViT performance average over three seeds. As outlined in the NeurIPS paper checklist, we provide additional results for ACT-ViT over the datasets and tasks provided Table I where we rerun ACT-ViT with three random seeds and report the mean and standard deviation. Those results are provided in Table
Table 8: ACT-ViT AUC HD performance averaged over three seeds with 1-sigma error bars.
HQA
Movies
HQA-Wc
IMDB
TriviaQA
Mis-7B
84.48 0.28
79.64 + 0.25
69.87 + 0.63
97.09
0.17
84.18 + 0.57
LlaMa-8B
82.45 + 0.36
84.05 + 0.58
71.82 + 0.34
93.57 + 0.34
85.25 + 0.35
Qwen-7B
88.03 + 0.32
95.25 4 0.14
78.23 + 0.59
96.24 0.05
91.12 0.10
A.4
Run-Time
Table 9: Training time (in minutes [m] and seconds [s]) for Probe[*] and ACT-ViT across all 15
combinations. Each ACT-ViT training run was performed on single NVIDIA L-40
GPU.
LLM
Method
HotporQA
Movies
HQA-Wo
IMDB
TriviaQA
Probe[=]
2[m] 45[s]
6[m] 23[s]
[m] 41[s]
[[m] 38[s]
[[m] 50[s] 12(m) 42(s)
Mis-7B
ACT-VST
13(m] 2[s]
11mm] 28(s]
12(m) 49[s]
14(m] 17[$]
Probel+]
2(m] 50[s]
2[m] 5[s]
1[m] 54[s]
2[m] 22[s]
2(m] 36[s]
LlaMa-8B
ACT-ViT
13[m] 48[s]
27(m) 5[s]
16[m] 34[s]
13 i[s]
27(m] 21[ss]
Probel+]
2[m] O[s]
[(m] 43(s]
[[m] 51[s]
1[m] 39[s]
1(m] 53[s]
Qwen-7B
ACT-ViT
10[m] 29[s]
13/m] 12(s)
11(m] 49[s]
11[m] 30[s]
10[m] 46[s]
25

---

We present the run-time of ACT -s) on a single NVIDIA L-40 GPU, compared with probing classifiers, on each of the 15
combinations considered in the paper: see Table
A.5 Layer/Token Visualizations
Below we present the layer-token heatmaps, corresponding to each of the 15 LLM-datasel combination considered in our paper.
26

---

LiaMa-88 HotporOA-W.
LiMM--BB HospotnA
LiaMa-8B-
Siyerstten IX3
Syperiiee IMP
Aupe Ditiiie (ME3)
0.90
0.73
10
0.
0.00
0.25
03
0.55
25
0.33
0.55
6-3
(0]
1-J1-43
LiMM 88
Mavies
LiMMo TriviDDA
Mis HotpotCA Wc
Bid Eeyeeaaaii
Bind
0.80
Rodkeperliee 140
0.00
0.75
0.64
10
10
0
0.70
0.
0.54
0.52
TAKONS
Mis:78 7B HotpotOM
Mis-7B 78 IMDB
Mis-76 Movies
0.7
0.708
Layos
1.5
0.600
20
220
20
0.625
0.575
Tokere
27

---

TrviagA
Syyor 103%
Qwen HotpotQA
Sipeciil 218
0.75
0.75
10
43
0.70
317
0.05
0.02
0.53
[1 6-1
M-]
T0tens
Owen 78 IMD8
Qwien-7E Movies
Owen-78 TrviDOA
Seyteittee 106
216
0.85
00
0.75
95
0.70
70
20
0.00
0.60
0.55
0.55
[2] 1-31
0.90
101 [4] [2] 1-31 (-1
0.50
101
(2113)
0.3
laxans
A.6 Low-Data Regime Adaptation
Below, we present additional results in the low-data training regime for all 15 L.LM- combina- tions considered in paper.
28

---

925
702
725
2725
4.0m
315
973
Figure 6:
LlaMa-8B
Figure 7: LlaMa-8B Hot- potQA
Figure 8: LlaMa-8B IMDB
HotpotQA-Wa We
730
404
650
733
700
525
1009
Aectuee shec 120
Predune abe No 1000
Figure 10: -laMa-8B-Tiin - aQA
Figure 11:
Mis-7B
Figure 9: LlaMa-8B- Movies
HotpotQA We
3,00
7
Figure 12: Mis-7B Hot-
Figure 13: Mis- IMDB
Figure 14: Mis-
Movies
potQA
3MC Na
Patatt
6."
Thaature 2311
Thature 2011
Testuue My tor 2001
Figure 16:
Qwen-7B
Figure 17: Qwen-7B Hot- potQA
Figure 15: Mis-7B TriviaQA
HotporQA-We
40
Deaiiuee 2041
Figure 20: Qwen- -Trivi aQA
Figure 18: Qwen-7B IMDB
Figure 19: Qwen-7B Movies
B
Pooling Algorithm
Our pooling algorithm is presented in Algorithm
29

---

Algorithm Pooling
Require: Tensor A
target sizes Tops N₂
Ensure: Tensor A E BRlxxN,DDar
1: A < permuteeA,, [DM, LM,NN]
2: Pad A that LM and N divisible by Lp and No
3: Compute patch sizes: fL = Lidd
JN
Npa N
Ly
4: Apply pooling:
A t- F. m. P.max_poo12d(A,
t kernekeeiiaaee = (Jt, [N))
5: A permute A, [p Nn, Dm))
6: return A
C Permutation Symmetries
Given the same number of layers and hidden dimensions, two LLMs can compute the exact same function and yet produce entirely different activation tensors. This discrepancy stems from internal symmetries in the model's weight space. To illustrate this, we consider simplified transformer architecture with L layers, omitting residual connections and batch normalization for clarity (though the analysis extends to those cases as well).
Recall that A' is the activation tensor at layer E [Z]. Let (Q), Kr, Vi) denote the query, key, and value matrices at layer i, computed as:
(Q., Ki Vi) =
1- (A'-1W9 A' A'-' A'-'W)).
(2)
Assume the feed-forward network (FFN) at layer / produces output:
A' RRLLU(AttnAA (A b?
bj.
(3)
Now assume the model uses d-dimensional hidden features, and let P Raxd be a permutation matrix, and define the following weight transformation:
(W by) + (WiP b)PT), 1+1 PW Ki+1 141
(W [1 W+1
W-
W Vi-1 (PW
(41
(5)
Although the new weights (the right hand side) are very different, it is straightforward to verify that the permutation matrices cancel out, leaving the underlying function unchanged. However, recalling Equation 33) the activation tensors differ significantly, as the transformation permutes the feature
dimension as follows:
A 11. n, d] - (d)),
where a is the permutation induced by P.
This illustrative example remarks the relevance of resorting to LLM-specific LA modules, as op- posed using single shared linear adapter. Although padding can side-step the dimensionality problem, such single adapter would be required to implicitly learn (to be invariant to) all feature permutations that may potentially arise (D!). While we note that principled approaches for handling such symmetries exist such as designing layers that are invariant to feature permutations by design we consider these less relevant to us given the limited number of considered LLMs. Exploring symmetry-awee architectures still remains promising direction envision explore in future endeavors.
Extended Analysis to Standard Transformers Activations. The implications extend naturally to a standard transformer that includes residual connections and batch normalization. Residual connections do not affect the analysis, since in our analysis the permutation symmetry is applies uniformly across layers-the same permutation used for weight matrices, so the residual addition remains consistent with our analysis. Batch normalization also preserves this symmetry, as is a
operation: applying permutation to the input features results correspondingly permuted output.
30

---

D Broader Impact
By enhancing the detection of hallucinations in Large Language Models, our work contributes the responsible development and deployment of generative Al by promoting greater transparency and trust. However, we acknowledge that our findings also reveal aspects of the predictive information embedded within LLM internals. This insight could be misused by malicious actors, potentially motivating restrictions on LLM internals access and thereby slowing progress open research.
31


Reviews:
[{"review_id": "k2X9ZEyKRU", "reviewer": "Reviewer_Brd8", "summary": "This paper proposes ACT-ViT, a novel architecture for hallucination detection in Large Language Models (LLMs) that leverages full activation tensors rather than individual token-layer pairs. Traditional probing classifiers are limited by their static, LLM-specific nature and fail to generalize across models. ACT-ViT treats activation tensors as image-like structures and applies a Vision Transformer (ViT)-inspired backbone, enabling joint training across multiple LLMs. The model uses linear adapters to align internal representations from different LLMs and supports efficient zero-shot generalization and fast fine-tuning for unseen models. Extensive experiments across 15 LLM-dataset combinations demonstrate ACT-ViT’s superior performance, scalability, and transferability compared to standard probes and logit-based baselines.", "questions": "1.\tPlease explain the importance and state-of-the-art relevance of probing classifiers in LLM Hallucination Detection (HD). Could you cite some directly related literature? In the related work section, the discussion does not seem to be directly relevant to HD (for example, citation [6]). It appears that probing classifiers are neither a cutting-edge nor a mainstream approach in HD.\n2.\tThe effectiveness of your method may be significantly affected by longer input and output text lengths. Please discuss and experimentally validate the impact of long input and output length on the performance of your approach.\n3.\tPlease provide a deeper analysis and explanation of the phenomena observed in Figure 2. Additionally, how does the proposed method adaptively focus on important activations? It appears that the approach simply relies on brute-force training rather than a truly adaptive mechanism.\n4.\tFor the linear adapter (LA), your method trains a separate adapter for each LLM. If your LA can generalize to different LLMs, then traditional classifiers should also be able to generalize across models. For a new model, how do you select the most suitable pre-trained LA from the existing ones?\n5.\tIn the experimental section, I believe the authors should include the following ablation studies for comparison: (1) Token[*]: Similar to Probe[*], but selects the optimal layer-token combination on the test set. This represents the oracle solution and should yield the best performance, serving as the upper bound for your method, even though it is not feasible in practice. (2) ACT-VIT without complete activations: Select the optimal layer-token combination on the validation set, then replicate this token to form the Activation Tensor (A). The rest of the process (model and training process) remains the same as ACT-VIT. (3) ACT-VIT without complete layers: Select the optimal layer on the validation set, then replicate all tokens from this layer to form the Activation Tensor (A). The rest of the process remains the same as ACT-VIT. (4) ACT-VIT with only the last layer: Select all tokens from the last layer, replicate these tokens L times to form the Activation Tensor (A). The rest of the process remains the same as ACT-VIT. These ablation studies can demonstrate that your motivation (adaptive selection) is responsible for the improvements, rather than additional parameters or training. Additionally, please explain why the Logits and Token methods are missing in the Off-domain Hallucination Detection experiments.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The strengths of this manuscript are summarized as follows:\n1.\tThe paper addresses an important and timely problem—hallucination detection in LLMs. This is a critical research area with growing relevance, given the widespread deployment of LLMs in high-stakes applications.\n2.\tThe paper provides a clear and accurate analysis of the limitations of existing methods and articulates guiding principles for improvement.\n3.\tThe proposed method is grounded in novel empirical observations revealed through insightful preliminary experiments, which effectively motivate the design of the approach. The method itself is thoughtfully constructed, leveraging structured activation tensors and vision-inspired architectures to address the identified challenges.\n4.\tThe paper is clearly written and well-structured, making it easy to follow.\nThe weaknesses of this manuscript are summarized as follows:\n1.\tThe paper focuses on a specific sub-direction of hallucination detection—probing classifiers—but does not sufficiently justify the significance of this direction. Although the authors cite several related works, many of them are only loosely connected to probing-based hallucination detection, which weakens the motivation and perceived importance of the chosen research scope.\n2.\tThe proposed method, while effective, appears somewhat straightforward and lacks deeper conceptual innovation.\n3.\tThe experimental section is somewhat limited and lacks strong ablation or comparative studies.", "was_revised": true, "final_justification": "The rebuttal can address most of my concern. The original rating is kept.", "rebuttal": "We appreciate your positive feedback and are glad you recognize the novelty and relevance of our work. Below, we address your comments.\n\n**“The paper focuses on a specific sub-direction of hallucination detection—probing classifiers—but does not sufficiently justify the significance.[...] explain the importance and state-of-the-art [...]”**\n\n\nWe believe probing for HD is important for several reasons. First, growing evidence shows LLMs encode strong signals about output correctness in their hidden states [1,2,3,6,8,9,10]. Second, we note, that several methods build on this by probing them—e.g., [1,2] for generic HD (as we deal with), to steer outputs [5], and to catch flawed reasoning or hallucinations in an unanswerable questions [9,10]. Third, importantly, probing hidden states is much more efficient than inferring the model’s correctness via semantic analyses of its generated text [7]. This makes probing an effective and cheaper alternative to, e.g., Semantic Entropy [11], as evidenced by [7] and our preliminary runtime analyses provided in response to Reviewer **gx9D** below. We thank you for this question — we'll revise the Related Work section to highlight this better and add any missing citations.\n\n\n**“The proposed method, while effective, appears somewhat straightforward [...]”**\n\nThe main contribution of this work consists of three parts. (1) We identify a specific limitation of existing probing techniques (Section 4, lines 134–167), namely their limited effectiveness in cross-LLM and cross-dataset applications. (2) We propose a solution to this limitation by introducing activation tensors as a new data type, and develop ACT-ViT—an architecture designed to leverage the structural properties of activation tensors to effectively process them. (3) We demonstrate that ACT-ViT not only offers a stronger solution to probing in standard settings (see Table 1 as an example), but also enables novel cross-LLM and cross-dataset applications for HD—including zero-shot generalization to unseen datasets—an area that, to the best of our knowledge, has received little attention (see Figure 4).\n\n**“The effectiveness of your method may be significantly affected by longer input and output text lengths [...]”**\n\nWe appreciate the opportunity to clarify. (1) our method detects hallucinations using only the LLM's response, not its input; and (2) ACT-ViT efficiently handles variable-length outputs via dynamic pooling, producing a fixed-size activation tensor. This makes it feasible to process even longer outputs while controlling computational cost.\nTo further justify this, and validate ACT-ViT’s robustness to longer outputs, we took your advice and conducted an additional experiment on the Math [4] dataset, which involves processing longer output text sequences [2].  We benchmarked ACT-ViT against all baselines, reporting only the best variant from each class due to space limits – best-probas for probability/logit-based methods and Probe[*] for probing-based ones – results are shown below. ACT-ViT remains the top performer, surpassing all baselines—by ~7 AUC points for LLaMA-8B. We will add this experiment to our paper, along with a proper discussion on the impact of the output length on ACT-ViT.\n|**Method** |**Qwen-7B – Math**| **Mis-7B – Math** |**LlaMa-8B – Math**|\n|-|-|-|-|\n|Best-Probas| 79.12| 56.00| 75.00|\n|Probe[*]| 92.13| 75.60 | 78.81|\n|**ACT-ViT(s) (Ours)**|**92.51** | **76.50**| **85.70** |\n\n**“Please provide a deeper analysis and explanation of the phenomena observed in Figure 2. [...] how does the proposed method adaptively focus on important activations? [...]”**\n\nThanks for the opportunity to clarify. We note that for a given LLM, the optimal probing token/layer position often varies across datasets [1,2]. On top of this, Figure 2 in our paper shows that even with a fixed dataset, the most predictive layer-token position for HD can differ across LLMs. As discussed in Section 4 (lines 134–174), this highlights a key limitation of standard probing methods in cross-LLM applications, as considered in our work. \nOur method is designed to overcome this limitation by side-stepping the reliance on a single, fixed probing position. By drawing an analogy with images and vision architectures (see Section 4.1), we propose to process the entire Activation Tensor. This allows the model to learn to combine predictive cues from multiple positions, like how vision models detect features across an image to identify faces. In this sense, our approach is “adaptive.” The ablation studies discussed below further validate this design and illustrate its effectiveness; see the relevant section below for details.\n\n**”For the linear adapter (LA), your method trains a separate adapter for each LLM [...]”** \n\nThanks for raising this point. As the reviewer notes, each LLM has its own Linear Adapter (LA). ACT-ViT is able to effectively learn on Activation Tensors from multiple LLMs as these LAs project their respective hidden states into a common embedding space. Here, the shared ViT Backbone (ViT-B) operates (see Figure 1) – HD is performed in this common space. This facilitates cross-model generalization: for a new LLM only its specific LA is trained, while the ViT-B remains frozen. Results in Table 2 demonstrate that this approach is indeed more effective than probing.\n\n**”[...] I believe the authors should include the following ablation studies for comparison:[...]”**\n\nThank you for the suggestion. We conducted the requested experiments and present the results below. Before that, we would like to note that selecting a single, optimal token-layer combination for probing across multiple LLMs is ill-defined, as they are not necessarily in correspondence.\n\nThat said, your proposed ablations offer valuable insights in a more controlled setting using a single LLM across various tasks, hence, we trained jointly on activation tensors from one LLM across all five datasets—HQA, Movies, HAQ-with-context, IMDB, and TQA—and report results below for Qwen-7B and LlaMa-8B.\n\nIn the table:\n- (1) Oracle Probe – probing applied to the best layer-token pair, when selected based on the test set;\n- (2) ACT-ViT (best token) — we select the best token across layers and replicate (broadcast) this 1D signal to form the whole activation tensor;\n- (3) ACT-ViT (best layer)— as above, but we select the best layer across all tokens.\n- (4) ACT-ViT (last layer) — as above but we select the last layer across all tokens.\n\n\n\n|**Method** |**Qwen -7B - HQA**|**Qwen -7B - Movies**|**Qwen -7B - HQA-WC** |**Qwen -7B - IMDB**|**Qwen -7B - TQA**|**Average**|\n|-|-|-|-|-|-|-|\n| Probe[*]   | 80.57  | 93.22  | 75.01 | 93.76  | 87.08 |85.93|\n| (1) Oracle Probe | 81.97  | 93.36 | 75.26  | 93.97  | 87.41 | 86.39 |\n| (2) ACT-VIT (best token) | 59.97  | 72.92  | 55.46  | 53.70  | 62.86 | 60.98|\n| (3) ACT-VIT (best layer)  | 84.86   | 92.38  | 75.06| 93.18 | 88.20 |86.74 |\n| (4) ACT-VIT (only last layer)| 82.36 | 92.90 | 71.70 | 92.52 | 86.32 | 85.16 |\n| **ACT-ViT**  | **88.20**  | **95.79** | **78.19** | **96.22** | **91.34** |**89.95** |\n\n\n| **Method**| **LlaMa - 8B - HQA** | **LlaMa - 8B - Movies** | **LlaMa - 8B - HQA-WC** | **LlaMa - 8B - IMDB** | **LlaMa - 8B - TQA** | **Average** |\n|-|-|-|-|-|-|-|\n| Probe[*] |79.71 | 83.61|70.56| 93.58|81.09 |81.71|\n| (1) Oracle Probe | 79.84 | 83.61 | 72.80| 93.68| 81.72|82.33  |\n| (2) ACT-VIT (best token) | 77.39| 74.01| 59.79 | 78.02 | 77.33 |73.31  |\n| (3) ACT-VIT( best layer)| 82.16  | **83.91** | 71.94 | 92.70| 83.83 | 82.91 |\n| (4) ACT-VIT (only last layer) | 80.51 | 80.57| 69.64 | 91.90 | 81.15 | 80.75  |\n| **ACT-ViT**| **82.22** | 83.70  | **72.71**|**93.55** | **84.68** |**83.37** |\n\n\n**Observations.** \n\n- (O1) Omitting the Oracle Probe, ACT-ViT is the best performing method overall – in 9/10 cases (see Bold in the Table). This suggests that processing the whole Activation Tensor is an advantageous strategy. \n- (O2) ACT-ViT(best layer) is second best, better than ACT-ViT (best token), indicating that committing to a single layer is more effective than focusing on a single token.\n- (O3)  As expected, the Oracle Probe outperforms the standard Probe. However, in most cases, it is notably outperformed by ACT-ViT. Note that the Oracle Probe is not an upper bound to ACT-ViT: ACT-ViT can, in fact, aggregate and combine information from different positions together (unlike standard probing). \n\n\n**”[...] please explain why the Logits and Token methods are missing in the Off-domain [...]”**\n\n\nTo clarify, both methods are used off-domain, but only the best variant of each is shown: best-probas (logit-based) and Probe[*] (probing). Full results will be added to the appendix.\n\n**References:**\n\n[1] The Internal State of an LLM Knows When It’s Lying. Azaria et al., EMNLP Findings 2023\n\n[2] Llms know more than they show: On the intrinsic representation of llm hallucinations. Orgad et al., ICLR 2024\n\n[3] Language Models (Mostly) Know What They Know. Kadavath et al., Arxiv 2022  (Anthropic AI)\n\n[4] Benchmarking hallucination in large language models based on unanswerable math word problem. Sun et al., CoRR 2024\n\n[5] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. Li et al., NeurIPS 2023\n\n[6] InternalInspector $I^2$: Robust Confidence Estimation in LLMs through Internal States. Beigi et al., EMNLP 2024\n\n[7] Efficient and Effective Uncertainty Quantification for LLMs. Xiong et al., SafeGenAI NeurIPS Workshop 2024\n\n[8] Discovering Latent Knowledge in Language Models Without Supervision. Burns et al., ICLR 2023\n\n[9] Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification. Zhang et al., Arxiv 2025\n\n[10] The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models. Slobodkin et al., EMNLP 2023\n\n[11] Detecting hallucinations in large language models using semantic entropy. Farquhar et al., Nature 2024"}, {"review_id": "JPxxWqx5yc", "reviewer": "Reviewer_hUsu", "summary": "The paper addresses a research question:\n- How to solve the dynamic existence of hallucination information depending on the location of tokens and layers?\n\nPrior work of probing classifiers has shown that the performance of hallucination detection depends on the selected location of tokens and layers. First, the paper shows that each LLM has a different location of tokens and layers for the best performance (See Fig. 2). This finding implies that we need to specific classifier for the given LLM, which is not generalizable to the different LLM.\n\nBased on the findings, the paper defines Activation Tensor (ACT) which has three dimensions: the number of layers, the number of tokens, and the dimension of the hidden state. ACT contains rich information on hallucination within a model and is used for detecting hallucinations. The paper proposes ACT-VIT architecture which takes ACT as input and hallucination classification as output. To leverage data of diverse LLMs, the paper introduces the adapter and pooling to align the spatial and feature dimensions across LLMs. \n\nExperiments are composed of In-domain (ID) and out-of-domain (OOD) setups. In ID, the performance of ACT is larger than the probing methods. Note that probing methods are strong baselines because the classifiers are trained in a supervised way. In OOD, ACT-VIT shows the dataset and LLM generalization; for the LLM generalization, the paper fixes VIT and trains the adaptor. The paper validates the proposed method.", "questions": "See above Weaknesses.", "limitations": "Yes, the paper includes Limitations in Sec. 6 and  Broader Impact in Appendix D,", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- The paper provides the analysis that the location of hallucination information is different depending on LLMs. These findings are connected to the generalization and motivates the design of ACT and ACT-VIT architecture for dynamic existence of hallucination and generalization.\n- ACT is large; thus, the paper considers the efficient design to use it. \n- Although probing classifiers show promising performance in hallucination, the performance of OOD is low. The paper validates that the proposed method is effective in OOD setup.\n\n### Weaknesses\n- Is the sequence length N fixed? Is it zero-padding to match the max sequence length?\n- What is the dataset size for pre-training?\n- Is the adaptor reusable if the hidden dimensions are aligned?", "was_revised": true, "final_justification": "The authors addressed my questions, such as the setup details of sequence lengths and dataset size, and the fine-tuning approach of the adaptor. I agree that the reusability of the adaptor is a future research direction, as mentioned in the authors' rebuttal. I have also read other responses. I think that the linear prob is a strong baseline with high performance for HD because it is a training approach. I will maintain my score.", "rebuttal": "Thank you for your positive feedback. We're pleased to see that you found our analysis valuable—especially our findings on how the location of hallucinated information varies across different LLMs, and the effectiveness of ACT-ViT in the off-domain setting. Below, we will address each of the points you raised.\n\n\n**Is the sequence length N fixed? Is it zero-padding to match the max sequence length?**\n\nYes, we follow the setup described in [1], where the maximum response length is fixed at 100 tokens. If a response is shorter than this length, we apply zero-padding to reach the maximum length.\n\n**What is the dataset size for pre-training?**\n\nEach LLM-dataset pair consists of 10,000 samples. We have a total of 15 such combinations. Depending on the experimental setup, different subsets of these 15 combinations are used. As a result, the total pre-training dataset size varies between 10,000 and 150,000 samples, depending on the experiments.\n\n**Is the adaptor reusable if the hidden dimensions are aligned?**\n\nYes, that’s a valid point—thanks for noting it. In principle, given prior knowledge that two LLMs have the same hidden dimension and that their hidden dimensions are aligned (e.g. a finetuned model), they could potentially share the same linear adapter. Experimenting with this aspect could be an interesting future direction.\n\n\n\n**References:**\n\n[1] Llms know more than they show: On the intrinsic representation of llm hallucinations. Orgad et al., ICLR 2024"}, {"review_id": "oDYgrNKzZS", "reviewer": "Reviewer_gx9D", "summary": "This paper introduces a novel vision transformer-inspired model for probing LLM activations for the task of hallucination detection. They process all the activations across layers and token positions as input, showing this enables cross dataset and LLM (with LLM-specific adapters) generalization, and strong performance at hallucination detection.", "questions": "- This is framed as hallucination detection but it seems like the task is accuracy prediction. Could you clarify your definitions?\n- Have you investigated other probing tasks?\n- Have you looked into interpreting the ACT-ViT itself? What layers/tokens are relevant and whether they agree with the single token/layer results?\n- Did you compare to other hallucination detection approaches: predictive entropy, semantic entropy, p(true)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Novel and well motivated technical contribution.\n- Fast and general purpose framework which can be used in practice with minimal overhead after pretraining.\n- Strong empirical results showing generalisation and improvement over other probing and logit based methods\n\nWeaknesses:\n\n- No comparison to other hallucination detection approaches: predictive entropy, semantic entropy, p(true)", "was_revised": true, "final_justification": "The authors' address my concerns so I maintain my score.", "rebuttal": "Thank you for your positive feedback. We're pleased that you found our technical contribution novel and well-motivated, and that you appreciated the practicality of our framework and the strength of our empirical results. Below, we address each of the points you raised.\n\n**\"No comparison to other hallucination detection approaches: predictive entropy, semantic entropy, p(true)\"**\n\nThanks for pointing this out. In our paper, we compared our approach with methods of low detection latency, and hence mostly focused on approaches based on output scores/probabilities and probing on hidden states.\nThe techniques the reviewer is referring to operate on multiple LLM generations or multiple prompts, thus introducing a non-negligible computational overhead that may prevent their real-time use. We note, though, that these may represent strong hallucination detection baselines, and we agree that comparing with them may improve the quality of our contribution. We will further emphasize it in our writing. \nWithin the limited time frame of the rebuttal period, we conducted a preliminary experiment on P(true) and Semantic Entropy (SE), in particular. In the table below, we report results for the Movies dataset and two LLMs: LlaMa-8B and Mistral-7B (Mis-7B), which we considered in our paper. Note that SE was evaluated with the Deberta entailment model considered in the original publication [4].\n\n| Method    | Mis-7B – Movies (AUC) |  LlaMa-8B – Movies (AUC) |\n|-------|------|-----|\n| P(True)  | 62.00 | 66.00 |\n| Semantic Entropy | 70.06 | 72.41|\n| **ACT-ViT** | **79.63** | **84.81** |\n\nWe note that in all cases, these approaches are significantly outperformed by ACT-ViT, which is not surprising since they are considered “gray-box” (don’t have access to internal artifacts, such as hidden states).\nTo provide a clearer, quantitative figure on the complexity incurred by these methods, we calculated the average runtime required by SE to output a prediction. This includes generating $10$ additional responses, and clustering them by calculating mutual entailments via the auxiliary Deberta model. On average, SE required around 6.2 seconds per sample on the Movies/LlaMa combination (std 2.5), and slightly less than 6 seconds on Movies/Mistral (5.9, std 1.7). Note that we made our best efforts to minimise the overhead caused by auxiliary generations by running them in parallel via batching. As a last comment, we note that the time required by the clustering phase is not negligible, as it amounts to ~ 2 seconds on LlaMa responses and ~ 1.35 second on Mistral’s.\nThese results concur to underscore the relevance of our approach, which not only performs better, but also much faster, with detection runtimes in the order of $10^{-5}$ seconds, as we noted in our paper (line 75).\n\n**\"This is framed as hallucination detection but it seems like the task is accuracy prediction. Could you clarify your definitions?\"**\n\nThe task we consider is essentially to classify whether the LLM’s response is correct or incorrect. We formulated this task as “hallucination detection”, consistently with previous papers e.g., [1,2,5,6], but we understand the source of confusion as this is very related to accuracy prediction. We will clarify this aspect in the next revision of the paper.\n\n**\"Have you investigated other probing tasks?\"**\n\nIn this paper, we focused specifically on hallucination detection, as we believe it is a sufficiently rich and impactful problem. This focus is also consistent with prior probing works, such as [1, 2], which concentrate only on this task. That said, applying our approach to other probing tasks—such as data contamination detection [3]—is a promising direction, and we consider it an avenue for future work.\n\n **\"Have you looked into interpreting the ACT-ViT itself? What layers/tokens are relevant and whether they agree with the single token/layer results?\"**\n\nThank you for the thoughtful question. While we haven't yet conducted a deep analysis of ACT-ViT’s interpretability, we agree that this is a valuable and promising direction. Understanding why ACT-ViT works could provide important insights into its effectiveness. Since our framework draws inspiration from principles in Vision, we plan to explore interpretability methods such as saliency maps and other techniques like those discussed in [7].\n\n\n **\"Did you compare to other hallucination detection approaches: predictive entropy, semantic entropy, p(true)?\"**\n\nWe kindly refer the reviewer to the response provided to the first comment.\n\n**References:**\n\n[1] The Internal State of an LLM Knows When It’s Lying. Azaria et al., EMNLP Findings 2023\n\n[2] Llms know more than they show: On the intrinsic representation of llm hallucinations. Orgad et al., ICLR 2024\n\n[3] Detecting Pretraining Data from Large Language Models. Shi et al. ICLR 2024\n\n[4] Detecting hallucinations in large language models using semantic entropy. Farquhar et al., Nature 2024\n\n[5] Language Models (Mostly) Know What They Know. Kadavath et al., Arxiv 2022  (Anthropic AI)\n\n[6] The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models. Slobodkin et al., EMNLP 2023\n\n[7] A Unified Approach to Interpreting Model Predictions. Lundberg et al.,  NeurIPS 2017"}, {"review_id": "Pipcd8a7Wh", "reviewer": "Reviewer_kwZz", "summary": "This paper presents an LLM hallucination detection approach.\nThe motivation is that previous probing layer methods are largely limited by their generalization capability to new tasks, new datasets, and new models.\nThe key idea of the proposed method is to leverage the activation of LLMs.\nAfter the initial extraction of several LLMs with respect to multiple queries, the authors then utilize a ViT model to capture some spatial relationship between responses and LLM layers. \nAcross 15 LLM-dataset pairs, ACT-ViT consistently outperforms prior white-box models and supports zero-shot generalization to new datasets.", "questions": "I will update my score if the authors can well address my concerns after rebuttal.", "limitations": "NA", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper presents an interesting ensemble method to detect LLM hallucination, which may be useful for future research.\n- The proposed method can outperform the previous probing layer baselines by a large margin.\n- Across 15 LLM-dataset pairs, ACT-ViT consistently outperforms prior white-box models and supports zero-shot generalization to new datasets. \n\nWeaknesses:\n- I don't quite grasp the usefulness of this task. To me, the hallucinations for different LLMs are quite subtle and fine-grained, and may range from token-level to sentence-level. And the hallucination even is different for the same two input queries repeated. In other words, hallucination is dynamic, while detection is static. \n- How did the authors get the labels for each response? Is it based on each token?\n- It seems the pooling layers are carefully selected. This will limit the generalization of the proposed method.\n- Are all the involved parameters trainable? If so, why use a ViT model?\n- Following the previous concern, I don't see the necessity of using a ViT model for the ensemble processing. \n\nMinor:\nLine 111, their internal", "was_revised": true, "final_justification": "Though I still hold concerns with the necessity of this task, I agree with other reviewers that this work has some merits.", "rebuttal": "Thank you for your thoughtful feedback. We're glad you found our method interesting and valuable for future research. We also appreciate your recognition that ACT-ViT consistently outperforms prior approaches. Below, we address each of the points you raised.\n\n**\"I don't quite grasp the usefulness of this task. To me, the hallucinations for different LLMs are quite subtle and fine-grained, and may range from token-level to sentence-level. And the hallucination even is different for the same two input queries repeated. In other words, hallucination is dynamic, while detection is static.\"**\n\nThank you for the opportunity to clarify. ACT-ViT does not attempt to produce a single classification score shared across multiple LLMs. While it is trained jointly on activation tensors from different LLMs, the input to our model and the prediction is made specifically, per each LLM's response.\nFurthermore, as you rightly pointed out, hallucinations are nuanced and may manifest differently across LLMs and granularities. Their dynamic and context-dependent nature actually motivates our approach. Rather than using heuristics or model specific, static, single position probes, ACT-ViT learns from full activation tensors from diverse LLMs and datasets. This way it can extract predictive features at the level of tokens, sentences, or otherwise, within each tensor and learn from a diverse set of hallucination signals. By outperforming the aforementioned baselines (e.g., see Table 1 in the paper), we validate that this joint training enables ACT-ViT to identify relevant patterns within each activation tensor, effectively learning to adapt its focus to the dynamic nature of hallucination.\n\n**\"How did the authors get the labels for each response? Is it based on each token?\"**\n\nWe followed the approach described in [1] to label the responses, performing the labeling at the sentence level rather than the token level. This involved matching sentences from the LLM-generated responses to those in the ground-truth answers using routines involving string matching.\n\n**\"It seems the pooling layers are carefully selected. This will limit the generalization of the proposed method.\"**\n\nWe would like to clarify our pooling strategy – the full pooling algorithm can be found in Appendix B, Algorithm 1. Our pooling technique selects the pooling kernel such that we end up with a fixed final size $(L_p, N_p)=(8, 100)$, which was fixed and kept constant across all experiments in the paper; except the ablation study in Figure 3, which specifically ablates this size.\nWhile the pooling does introduce some information loss, it proved effective in practice, as demonstrated by our results throughout the paper (e.g., see Table 1). Furthermore, we conducted an ablation study on different pooling sizes (see Figure 3), which shows a trade-off between performance and computational complexity: larger pooling dimensions $(L_p, N_p)$ reduce information loss and improve performance, but at the cost of increased training time. This flexibility allows our pooling strategy to be adapted based on the user’s computational constraints. Notably, as shown in Figure 3, all pooling variants continue to outperform the strongest probing baseline, demonstrating a degree of robustness across configurations.\n\n**\"Are all the involved parameters trainable? If so, why use a ViT model?\"**\n\nThank you for your question. Our ViT backbone (ViT-B), is not pre-trained on vision tasks: in fact, all parameters in our pipeline are trained from scratch on a corpus of hallucination detection datasets; an overview of our method is available in Figure 1 in the paper. After this procedure, and only in the off-domain experiments, ViT-B is kept frozen and only the linear adapters are trained.\nIn short, we do not use a pretrained ViT model, but rather adopt a ViT-like architecture we train from scratch for hallucination detection. We chose this architectural design pattern due to the structural analogy between Activation Tensors and images (two sequential dimensions, i.e., tokens and layers, arranged spatially in a 2D map). We elaborate on this analogy in our paper, in Section 4.1, lines 174–184. This motivates the use of a ViT-like architectural backbone, which proves to be effective in our experiments as it outperforms its MLP-based counterpart. In the next revision, we will make sure to highlight this rationale and aspects more clearly.\n\n**\"Following the previous concern, I don't see the necessity of using a ViT model for the ensemble processing.\"**\n\nWe believe the ViT backbone (ViT-B) plays an important role in our setup, and we would like to stress that it is not used for ensemble processing (e.g., combining outputs from multiple LLMs), but rather as a unified processing architecture on activation tensors. To clarify, the overall approach has two components: (1) we use pooling and linear adapters to bring activation tensors from different LLMs into a consistent shape and space, allowing joint training across models; and (2) this resulting tensor, structurally analogous to an image as explained in our previous responses, is then processed by ViT-B. As already mentioned, ViT-B is a natural and effective fit for this, as it allows to process activation tensors in a structured and spatially-aware way, beyond static, single position probing. We will make these rationales more explicit in the next revision.\n\n**\"Minor: Line 111, their internal\"**\n\nThanks for the note, we will fix this in the next version of the paper.\n\nWe appreciate your thoughtful feedback and hope our responses have addressed all of your points. If any issues remain unclear, we would be happy to provide further clarification. Otherwise, as you mentioned, we kindly ask you to consider updating your score based on our responses.\n\n**References:**\n\n[1] Llms know more than they show: On the intrinsic representation of llm hallucinations. Orgad et al., ICLR 2024"}]

Author rebuttal:

Meta-review: (a) Summary of Claims and Findings

This paper introduces ACT-ViT, a novel method for hallucination detection that addresses the poor generalization of prior probing classifiers. By treating an LLM's full activation tensor as an image-like structure processed by a ViT-based architecture, the model can be trained jointly across different LLMs. The key finding is that this approach significantly outperforms baselines and demonstrates strong zero-shot generalization to unseen models and datasets.

(b) Strengths

The paper's core strengths are its novel and well-motivated approach to the critical problem of LLM hallucination, backed by strong empirical results that validate its superior performance and generalization capabilities. The work is technically sound and clearly presented.

(c) Weaknesses

Initial weaknesses, primarily a limited comparison to non-probing baselines (e.g., Semantic Entropy) and the need for more detailed ablations, were comprehensively addressed during the author rebuttal period.

(d) Reasons for Acceptance

The primary reason for acceptance is the paper's elegant and impactful solution to a key challenge in LLM reliability. The ACT-ViT method is a technically sound and novel contribution that demonstrates impressive generalization—a significant advance for probing-based techniques. The thorough empirical validation solidifies this as a strong and valuable contribution to the field.

(e) Summary of Discussion and Rebuttal

The rebuttal period was highly productive. The authors provided extensive new experiments, including comparisons against requested baselines and a full suite of ablation studies. This successfully addressed all major technical concerns, converting reviewer feedback into a strong positive consensus. The paper was substantially improved by the review process and is now in excellent shape for publication.
Decision: Accept (poster)