GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
Authors: Seongheon Park
Sharon Li
Venue: NeurIPS 2025 poster ()
Keywords: large vision-language models
multi-modal large language models
object hallucination
Primary area: deep_learning
TL;DR: We propose GLSim, a training-free framework that combines global and local embedding similarity signals for accurate object hallucination detection in LVLMs, outperforming prior methods.

Abstract: Object hallucination in large vision-language models presents a significant challenge to their safe deployment in real-world applications. Recent works have proposed object-level hallucination scores to estimate the likelihood of object hallucination; however, these methods typically adopt either a global or local perspective in isolation, which may limit detection reliability. In this paper, we introduce GLSim, a novel training-free object hallucination detection framework that leverages complementary global and local embedding similarity signals between image and text modalities, enabling more accurate and reliable hallucination detection in diverse scenarios. We comprehensively benchmark existing object hallucination detection methods and demonstrate that GLSim achieves superior detection performance, outperforming competitive baselines by a significant margin.

GLSIM: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
Seongheon Park Sharon Li
Department of Computer Sciences University of Sesconsin-Madisee (seongheon_park, sharonli>@cs edu
Abstract
Object hallucination in large vision-languant models presents significant chal- lenge their safe deployment real-world applications. Recent works have proposed object-level hallucination scores to estimate the likelihood of object hallucination; however, these methods typically adopt either a global or local perspective in isolation, which may limit detection reliability. In this paper, we introduce GLSIM, novel training-free object hallucination detection framework that leverages complementary global and local embedding similarity signals be- tween image and text modalities, enabling more accurate and reliable hallucination detection in diverse scenarios. We comprehensively benchmark existing object hallucination detection methods and demonstrate that GLSIM achieves superior de- tection performance, outperforming competitive baselines by significant margin'.
Introduction
Large Vision- Lisiun-Language Models (LVLMs)) [1, 3, 4, 5, 6, have made striking advances in understanding real-world visual data, enabling systems that can describe images, answer visual questions, and follow multi-modal instructions with fluency and creativity [9, 10]. Yet beneath this surface of impressive capability lies critical vulnerability-objer hallucinations (OH)-whers the model generates plausible-sounding mentions objects that are not present the image [11]. An example is illustrated in Figure I, where the LVLM describes "dining table" in birthday party scene, even though the image contains no such object. These hallucinations can undermine user trust, and are particularly concerning in high- domains including medical imaging [1], autonomous navigation [12], and accessibility applications [13]. Detecting such hallucinations is thus essential for safe and reliable deployment of LVLMs, and has become an increasingly active area of rescarch [14].
Existing approaches to object hallucination detection often rely on external knowledge sources, such as human-annotated ground truth annotations [11, 15, 16, 17, 18, 19]. Others prompt or fine-tune external large language or vision-language models as judge to detect hallucinations (20, 21, 22, 23, 24, 25J. However, these approaches face practical limitations: ground-truth references are often unavailable in real-world scenarios, and external LLMMs are prone to hallucinating themselves, thereby limiting reliability. This highlights the need for lightweight, model-interine approach that can detect and self-evaluate hallucinations without supervision or auxiliary models.
In this paper, we propose an object-level hallucination scoring function that operates without relying on external sources, leveraging the embedding similarity between image and text modalities within the latent space of LVLMs. We introduce Global-Lock Similarity (GLSIM) score, method that unify two complementary perspectives: global similarity score, which captures how well an object semantically fits the overall scene, and a local grounding score, which checks whether any specific
"Code available
wtt cow.bb/e/oc///eeeeaniiggggg wisc/glsim
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generated description
Notation
The image features
child, atanding in oz birthday eake. The child vearing a [so, the cake is placed en Eiasing tablg and there ase ballacos 15
Image:
Laleni embeddlng
Unembedding matrix
Cosine similarity
Prompt:
Describe this image.
Objects o distag
"Lie"
Pre-trained LVLM
0.05
Global score Siglobal
Hallucintted
0.26
Top-K
Local score
0.92
Image
Prompt
0.83
GLSim score Sot.5w
(a) Extracting latent embeddings
(b) Object grounding
(c) Computing score
Figure I: Overall framework. (a) We detect object-level hallucinations by leveraging latent embed- ding similarity. (b) For each object, the most relevant image regions identified via unembedding from latent image representations.
(c) The final GLSIM score is computed as a weighted combination of local (Section 4.2) and global (Section 4.3) signals, capturing both scene-level plausibility and spatial alignment, enhancing object hallucination detection accuracy.
region in the image actually supports the object's presence. This fusion addresses key shortcoming of prior approaches that rely on one perspective in isolation [26, 27, 28. 29]. For instance, a global- only method may wrongly consider "dining table" plausible in a birthday party scene (Figure 1), simply because such contextual associations are common pretraining visually present. On the other hand, local-only approaches may struggle when hallucinated object visually similar to real objects the scene, as Figure 2, where a model hallucinates a "handbag" due to confusion with leather seat of motorcycle. By integrating both global and local signals, our method can ask not only "does this object belong contextually to the scene?" but also "is there concrete visual evidence for it?", resulting in more accurate, well-rounded, and interpretable hallucination detection across diverse scenarios.
even if no table
As illustrated in Figure 1, GLSIM works by evaluating each object mention along two axes. First, the global score measures the similarity between the object token's 's embedding and the overall scene embedding- captured by the final token of the multimodal instruction prompt (highlighted in yellow). Next, we compute a local similarity score that checks for spatial grounding. Specifically, we identify the top image patches most relevant to the object using an adapted Logit Lens technique [30], then assess whether these regions provide strong visual evidence for the object using the average similarity between the object token's 's embedding and the top-K image token embeddings (highlighted in green). By combining these two complementary signals, GL.SIM produces a holistic score that reflects both contextual fit and visual grounding-effeen
distinguishing real objects from hallucinations.
We extensively evaluate GLSIM across multiple benchmark datasets and LVLMs, including LLaVA- 1.5 [1]. MiniGPT-4 [3], and Shikra [31], demonstrating strong generalization and state-of-the-art performance detecting object hallucinations. On both MSCOCO and Objects365 datasets, GLSIM consistently outperforms the latest baselines, including Internal Confidence [28] and attention- grounding scores [27], achieving up a +12.7% improvement in AUROC, Ablation studies confirm the complementary roles of the global and local components: removing either degrades performance, while their combination yields the most reliable detection. Qualitative results further illustrate how GL.SIM accurately flags subtle hallucinations, making it a practical tool for real-world deployment.
Our key contributions are summarized as follows:
1. We propose GLSIM. a novel object hallucination detection method that combines global and local similarity scores between latent embeddings. To the best of our knowledge, this the first work to demonstrate their complementary effectiveness for the OH detection task.
2. We provide a comprehensive benchmarking of existing OH detection methods, addressing important gap that has been overlooked in prior work.
3. We demonstrate the superior performance of GLSIM through extensive experiments, conduct in-depth ablations to analyze the contributions of each component and design choice, and verify the generalizability of our method across various LVLMs and datascts.

---

2
Related Works
Object Hallucination Detection in LVLMs. Object hallucination (OH) refers to the phenomenon where LVLMs generate textual descriptions that include non-existent objects in the image- a crit- ical but underexplored problem in LVLMs with direct implications for reliable decision-making Such hallucinations can stem from factors including statistical biases in training data [32], strong language model prior [9], or visual information loss [33]. Recent studies have focused on evaluating
and detecting OH by leveraging ground-truth annotations [11, 15, 16, 17, 18, 19]. For instance, CHAIR [11] suggests utilizing the discrete ratio of objects presented in the answer relative a ground-truth object list to identify OH. Another line of work evaluates OH using external LLMs or LVLMs [20, 21, 22. 23, 24, 25]. For instance. GAIVE [22] leverages a stronger LVLM (e.g., GPT- 4 [34]) as a teacher to assess the responses of a student model, while HaLEM [35] fine-tunes an LLM (e.g., LLaMM [36]) to score LVLM generations. While effective, these methods are resource-intensive and often lack transparency.
Several recent works have proposed object-level hallucination scores that self-evaluate OH likelihood without requiring an external judge model or additional training. For instance, LURE [26] utilizes the negative log-likelihood (NLL) of the object token generation probability; Internal Confidence (IC) [28] computes the maximum probability of the object token across all image hidden states [30]; and Summed Visual Attention Ratio (SVAR) [27] leverages attention weights assigned to image tokens with respect to the object token. While promising, these methods primarily target hallucination mitigation and often fall short in detection performance: these methods typically leverage either global (e.g., NLL, SVAR) or localized (e.g., IC) signals in isolation and thus fail to capture the nuanced interplay between the overall semantic context and fine-grained visual grounding. Moreover, NLL often fails since LVLMs tend to favor linguistic fluency over factual accuracy [37]; IC does not fully capture contextual information from the generated text; and SVAR can be biased toward previously generated text tokens [38] and vulnerable to attention sink effects [39].
Different from prior works, we introduce the first object hallucination detection method that explicitly integrates both global and local uigfying localized attribution with holistic semantic align- ment between the image and generated text. We benchmark our approach against existing object-level hallucination detection methods across diverse settings to offer a comprchensive comparison in this space. Further related works are provided in Appendix C.2.
3 Problem Setup
Large Vision-Language Models for text generation typically consist of three main components: a vision encoder (e-g., CLIP [40]) which extracts visual features, multi-modal connector (e-g., MLP) that projects these visual features into the language space, and an autoregressive language model that generates text conditioned on the projected visual and prompt embeddings.
Given an input image, the vision encoder processes it into a set of patch-level visual embeddings, commonly referred to as visual tokens. These tokens are then projected into the language model's embedding space through the multi-modal connector, resulting in a sequence of N visual embeddings:
(V1,.... vN } E R Nxd where each e; corresponds to transformed visual token of dimension d. On the language side, the input text prompt (c.g.. "Describe this image in detail.") ") is tokenized
and embedded into sequence of language embeddings: tokens v and the textual embeddings prompt length. These two modalities the projected visual are concatenated and passed as the input sequence the language model. The language model then generates a sequence of output tokens: y (yi,..., yas ), where each y; E V is drawn from a vocabulary space and M is the output length.
(t1.....f tt} } R where L is the
Object hallucination detection. In this work, we focus on detecting object existence hallucination in LVLML where the model generates text that references objects not present in the image [15, 41, 14]. This represents the most fundamental and critical form of errors affecting model reliability. We provide the formal task definition below. Definition 3.1 (Object Hallucination Detector). Leix (v. t) denote the input to the LVLM, and y = (vi,..., ywu be the sequence of generated tokens from the model. From y, we extract set of object mentions o = {or,..., Ongtn, C O, where ni and n, denote the number of hallucinated and real objects, respectively. The task of object hallucination detection is to design scoring
3

---

(a)
(b)
Descripion Input Image
Object: "dining table'
Description Input Image
Object: "handbag"
Score
Score
young posxibly baby, standing in trant of birthday eake. The child is vearing :ie, the oske is placed
Inage
image Feacures
purple zotoreycle, which la parked on street, and the nan appeaxs to be wearing a leacher jacket, Dondbad can be soen close to the
dobsllocalGLSmm
dobal
Bintor bebig, and theze balioons the
Figure 2: Qualitative evidence. In the generated descriptions. hallucinated objects are highlighted in red. The localized image regions are shaded with the same color their corresponding objects. The gray line shows a threshold value т. an object's 's score lower than the threshold т, we consider hallucination. In (a), the local score successfully compensates for the failure of the global score, while (b), the global score offsets the limitations of the local score.
function 8: x X -> [0, 1), where s(o, x) quantifies the likelihood that object E is present in the input x E X. Here O and X denote the space of objects and input, respectively. Based on this score, we define the object hallucination detector:
x) - =f) [)(
G(o,x)=
G(o,
(I)
otherwise.
where E [0, 1] is decision threshold. Here, G(o, x) 1 indicates that object is real (i.e., occurs in the image), while G(o,x) = 0 indicates a hallucinated object.
4 Method
Overview. In this section, we propose an object-level hallucination scoring function that operates without relying on external sources, leveraging the embedding similarity between image and text modalities within the latent space of LVLMs. We introduce Global-Local Similarity (GLSIM) score, a method that leverages both global and local similarity measures, and discuss how these complementary signals contribute to effective object hallucination detection.
4.1 Motivation: Both Local and Global Signals Matter
Object hallucination in LVLMs often arises when models generate plausible- soundiing descriptions that are not visually grounded. But detecting such hallucinations is challenging: they can stem from subtle biases, background patterns, statistical co-occurrence in training data [15, 42, 26]. Critically, relying on a single perspective-either global similarity or a local region-level score-is often not enough to reliably catch them. In particular, global similarity quantifies how semantically related the object is to the image as whole. It captures holistic alignment between the object mention and the overall scene, and is useful for assessing whether the object "makes sense" in context. In contrast, local similarity measures how well the object is visually grounded in specific region. It focuses on fine-graines evidence aligned with spatial areas most relevant to the object, helping verify whether is actually present.
Qualitative evidence. Figure illustrates how each signal alone can be insufficient. panel (a), the LVL--generated description includes a "dining table", yet no table is present in the image. A global similarity score fails to flag this hallucination cake, party setting) frequently co-occurs with tables in training data, leading to a high false-positive signal. In contrast, a local score that focuses on the visual region associated with "dining table" correctly assigns low similarity, reflecting the absence of meaningful grounding in that region. In contrast, panel (b) shows failure case for local similarity. The model hallucinates a "handbag," and while the global similarity correctly captures that the handbag is not semantically compatible with the overall scene, the local score becomes unreliable-likely due to a visually similar object in the image ei.e., leather seat of the motorcycle).
because the overall scene (e.g., birthday

---

These examples underscore the inherent limitations of using either signal in isolation. Global similarity can be overly influenced by high-level contextual associations, leading to false positives when hallucinated objects are contextually plausible within the scene but not visually present. On the other hand, local similarity is sensitive to spatial precision, but can misfire when localization noisy or there are visually similar objects. As result, each signal captures only partial view of the grounding problem. To overcome this, we propose a unified approach, Global-Local Similarity (GLSIM), that leverages the complementary strengths of both perspectives and offers more accurate and reliable detection of object hallucinations across a diverse range of visual-textual contexts. the next subsections, we introduce the score definition in expllnexpliiinn how we design global and local similarity for each object mention, and how they are integrated into a single decision score for hallucination detection. More qualitative results are presented in Appendix B.1.
4.2 Object Grounding via Local Similarity
key component of our approach is the computation the local similarity score, which captures how well an object mention is visually grounded in a specific region of the image. Unlike global similarity, which reflects scene-level plausibility, the local score focuses on verifying the presence of the object at the spatial level. The main challenge lies in identifying the most relevant region for each object mention -ethoon-without relying on external annotations or bounding boxes.
Unsupervised object grounding, We leverage an unsupervised approach that leverages internal representations of the LVLM itself, to ground whether predicted object token is hallucinated or not. Given the LVLM input = (v, t), where {ri,.... ........ ,UN) are the visual tokens and t are the prompt embeddings, we extract the hidden representations (1x) E Rd of each visual token uy at decoder layer I. To project these representations into the vocabulary space, we can leverage Visual Logit Lens (VLL) as:
(L) (v())) WU.
where Wo E dRd is the unembedding layer matrix. Unlike the original Logit Lens [30], which operates solely in language models, our approach adapts it multimodal setting to attribute generated object mentions to relevant visual tokens. We apply a softmax and extract the predicted probability for the target object token or Nettmax(VLLii) [o]. probability quantifies how likely visual token v; is to predict the object word o, offering a model-internal signal of relevance between the image patch and object token. Importantly, we select the Top-K image patches with the highest probabilities as the localized regions corresponding to the object o:
[(o)= = TopK,. ((softmax (iooftmax(VLL(())) (VL (o)))
We visualize object grounding results in Section 5.3 and Appendix B.2.
Local similarity score. Based on the localized regions I(o). we compute average cosine similarity between each localized image embedding and object embedding:
Slocal (0, x)
1 K
Σ sim(h)(t)),hr hv, U.EZ(0)
where sim(:,-) denotes cosine similarity, and 1' is the decoder layer used represent the text embedding at the position of the object word. The score should be higher for real objects and relatively lower for hallucinated objects.
4.3 Scene-Level Grounding via Global Similarity
While the local similarity score focuses on spatially grounding an object in specific image regions, it alone may be mnsffiiient-eeseccii this, we introduce a global similarity score that measures scene-level semantic coherence between an object mention and the entire image. This can be useful for identifying out-of-context hallucinations (e.g., referencing a "handbag" in a motorcycle scene).
in cases where localization is ambiguous. To complement
Global similarity score. We compute the global similarity as the cosine similarity between the embedding of the object/text token and the embedding of the final token in the instruction prompt. The final instruction token often encodes condensed summary of the model's 'ss understanding of both
5

---

image and prompt context. By comparing the object token to this representation, the global score quantifies how well the object semantically aligns with the overall scene. This allows the model to down-weigh mentions that may be contextually implausible, even if they are locally aligned with some visual region.
Formally, given an object mention and LVLM input x = (v, t), let (o) E R be the object token representation at layer 1', and let (v, t) E R be the hidden representation of the last visual-text prompt token at layer l. The global similarity score is then defined as:
Sylobal (o, x) sim (hy(v, (), hy (o))
where sim denotes cosine similarity.
Global-Local Similarity (GLSIM) score. To fully leverage the complementary strengths of both grounding signals, we define the final hallucination detection score as a weighted combination of local and global similarity. Specifically, we define the GLSIM score
SGLSIM (o. x) w Sglobal x) + (1 w) (0,
(5)
where w E [0. 1] is a hyperparameter controlling the balance between local evidence and global context. This fused score captures both spatial alignment and scene-level plausibility, enabling more accurate detection of hallucinated objects. In practice, we find that a moderate value of w (e.g.. 0.6) yields consistently strong performance across diverse scenarios (see Section 5.3). Based on
the scoring function, the object hallucination detector is G(o,, x) = { (o, x) 2 TJ, where I indicates real object and indicates a hallucinated object.
5 Experiments
5.1 Setup
Datasets and models. We utilize the MSCOCO dataset [43], which is widely adopted as the primary evaluation benchmark in numerous LVLM object hallucination studies and contains 80 object classes. In addition, we employ the Objects365 dataset [44], which offers a more diverse set of images and a larger category set comprising 365 object classes, along with denser object annotations per image. For evaluation, we randomly sample 5,000 images each from the validation sets of MSCOCO and Objects365. We conduct experiments on three representative LVLMs: LLaVA- [1], MiniGPT-4 [3], and Shikra [31]. For LLaVA-1.55, we evaluate both 7B and 13B model variants to study scalability. Implementation details are provided in Appendix A. We evaluate on three additional [2], L.L. VA-N- [45], Cambrian-1-S8 [46], Qwe-2.5-VLL-B [47]. and Intern VL.3-BB [48] in Appendix D.1.
Evaluation. We formulate the object hallucination detection problem as an object-level binary classification task, where a positive sample is a real object and a negative sample is a hallucinated object. We extract objects from the generated descriptions and perform exact string matching against the ground-truth object classes of each image and their synonyms, following CHAIR [1]]. To evaluate OH detection performance, we report: (1) the area under the receiver operating characteristic curve (AUROC), and (2) the area under the precision-recall curve (AUPR), both of which are frrehhod-inddppnncee metrics widely used for binary classification tasks.
Baselines. We compare our approach against a comprehensive set of baselines, categorized as Serrooachhs-Negatiee
LigeLieel (NLL) [26] and En- tropy [49]; (2) Logit Lens probability-asse approach-IItern Confidence [28]; (3) Attention-based Visual Attention Ratio (SVAR) [27]; and (4) Embedding similarity-based Caaonacch-Cntexttur Lens [29]. To ensure fair comparison, we evaluate all baselines iden- tical test sets using the default experimental configurations provided their respective papers. As Contextual Lens was originally proposed for sentence-level hallucination detection, we adapt it for object-level hallucination detection. Further details of these baselines are discussed in Appendix C.
follows: (1) Token probabillty-baseed
Approach-Sunine
5.2 Main results
As shown Table 1, we compare our method, GI.SIM, with competitive object hallucination detection methods, including the latest ones published in 2025. GLSIM consistently outperforms existing

---

Dataset
Method
LLaVA-1.5-7B
LLaVA-1.5-13B
MiniGPT-
Shikra
AUROCT
AUPR
AUROC1
AUPR
AUROCT 1
AUPR
AUROCT
AUPR
NLL [26]
ICLR'24
63.7
84.9
63.1
86.1
39.4
81.2
60.4
82.1
Entropy [49]
ICLR'21
85.0
63.2
86.3
60.6
83.2
62.9
84.0
MSCOCO
Internal Conf. [28]
ICLR
72.9
89.3
71.0
90.0
75.7
93.0
69.1
88.5
SVAR
CVPR'25
74.7
91.2
75.2
92.9
83.6
95.9
70.7
89.1
Contextual Lens [29]
NACC1/25
75.4
90.7
78.7
92.8
84.9
96.2
69.5
87.6
GLSIM (Ours)
83,7
94,2
84.8
95,8
87.0*044
97.0
83.0
94,9
NLL [26]
ICLR'24
60.8
59.4
56.7
70.4
64.8
Entropy [49]
ICLR'21
63,3
60.9
59.1
60.4
$7.3
70.7
60.7
67.6
Objects365
Internal Conf. [28]
ICLR'25
67.4
68.5
72.5
SVAR [27]
CVPR'25
64,9
66.6
63.5
68.2
71.0
79.4
60.6
68.3
Contextual Lens [29]
NACC1:25
63.2
62.6
62.1
65.6
70.2
77.8
59.6
67.0
GLSIM (Ours)
72.6400
74.6*
70.4
74.0
74.8
82.4=0.7
69.7
75,9
Table I: Main results. Comparison with competitive object hallucination detection methods on different datasets. For our method, the mean and standard deviation are computed across three different random seeds. All values percentages, and the best results are shown in bold.
state-of-the approaches across different models and datasets by a significant margin. Specifically, on the MSCOCO dataset with AA-1-A-1.--75 GL.SIM outperforms SVAR by 9.0% AUROC, and achieves an 8.3% AUROC improvement over Contextual Lens, an embedding similarity- baseline. Unlike Contextual Lens, which relies the maximum cosine similarity between text embeddings and image embeddings, GLSIM integrates global and local signals, resulting in more robust detection performance. Notably, method also demonstrates strong performance on Shikra, achieving a 12.7% % improvement in AUROC on the MSCOCO dataset compared to SVAR. Given that Shikra is trained with focus region-level inputs and understanding, this result suggests that our method is effective in models with strong spatial alignment capabilities.
Comparison with Internal Confidence. Recently, the Internal Confidence (IC) method [28] was proposed to detect hallucinations using visual logit lens probabilities. Our approach differs from IC three key ways. First, IC directly uses the maximum probability from the visual logit lens across all image patches and layers, which can be overconfident- assionggnnnnntsigiinnnggggn assigning high scores to hallucinated objects (see Figure 3). In contrast, we compute the seman- tie similarity in representation space between the object token embedding and the Top-K visual tokens, yielding more reliable and semantically meaningful signal. For hallucinated objects, this leads to alignment with semanti- cally irrelevant regions, resulting lower similarity scores, thereby enabling more reliable object hallucination detec- tion. Second, IC considers only the most probable patch, while we aggregate over the Top- K most relevant patches. As shown in our ablation study (Section 5.3), using multi- ple visual regions improves performance by capturing spatially distributed evidence and reducing sensitivity local noise. Third, IC purely local in nature, whereas our framework also integrates global similarity score that captures object-scene coherence at the image level. Together, these advantages enable our method outperform by substantial margin of 10.8 AUROC.
Object: "adult"
Score
Ours
adul
Latent Embedding
Cosine Similarity
Figure 3: Internal Confidence (IC) can assign high confidence to incorrect re- gions for hallucinated objects. Our lo- cal score (Section 4.2) mitigates this by cross-modal embedding similarity.
5.3
Ablation Studies
In this section, provide various in-depth analysis of each component of our method. All exper- iments are conducted using LLaVA-115578 and Shikra on the MSCOCO dataset, and results are reported in terms of AUROC (%). Further ablation studies are provided in Appendix D.
Analysis of global and local scores. We systematically compare global and local scores on the MSCOCO dataset, as shown in Table 2. Finding 1: Embedding similarity is an effective scoring metric. Embedding similarity (ES)-based methods consistently outperform other scoring functions, with GLSim (Top-K) achieving a 22.6% improvement over NLL on Shikra. In contrast, token probability (TP)-based approaches are optimized for linguistic fluency rather than object existence accuracy; attention weight (AT)-based methods often fail to align with causal attributions [50]; and
7

---

Attention
Cosine Similarity
Logit Lens (Ours)
Attention
Cosine Similarity
Logit Lens (Ours)
(a) Object: "car"
(b) Object: "surfboard"
Figure 4: Object grounding results with LLaVA. Ground-truth bounding boxes are shown red.
Method
Metric
LLaVA
Shikra
NLL [26]
TP
63.7
60.4
Score
Grd. Method
LLa VA
Shikra
Entropy [49]
TP
64.0
62.9
Global
SVAR [27]
AT
74.7
70.7
Ogicbal
79.3
79.8
Sylobal (Eq. (4))
ES
79.3
78.9
Attention
66.3
65.0
Internal Conf. [26]
L..P
72.9
69.1
Slocal
Cosine Sim.
76.2
70.1
Logit Lens
78.8
76.8
Local
Contextual Lens [27]
ES
75.4
69.5
Slocal (Top-1)
ES
76.5
73.1
Attention
79.4
80.0
Stecal (Top-K)
ES
78.8
76.8
SGLSM
Cosine Sim.
80.7
80.9
Logit Lens
83.7
82.0
8GLSoM (Top-1)
ES
82.0
81.0
G&L. 1.
SGLSIM (Top-K)
ES
83.7
83.0
Table 2: Comparison of global and local scores.
Table 3: Object grounding methods.
Logit Lens probability (LLP) methods tend to exhibit overconfidence. By directly capturing the semantic alignment between image and text modalities, embedding similarity provides more reliable signal for OH detection. Finding 2: Object grounding improves OH detection. Among local methods, approach leverages grounded objects in the image and computes embedding similarity directly with those object representations, achieving a 7.7% improvement over Internal Confidence the Shikra model. This enables fine- aie-ggrieed aligmment, unlike Internal Confidence and Contextual Lens methods, which rely only on the maximum token probability or cosine similarity score. Finding 3: Combining global and local scores further improves performance. By combining global (8gtohal) and local (8tcat) similarity scores, we observe additional gains of 2.7% in Top-1 and 4.4% in Top-K K for the LLaVA model. This demonstrates that scoring function design in Equation (5) effectively integrates the complementary strengths of both global and local signals.
Comparison of object grounding methods. We explore several design choices for the patch selection for object grounding Section 4.2, with results summarized in Table Specifically, we vary the metric used for Top-K K (K 32) patch selection, comparing (1) attention weights, (2) cosine similarity, and (3) our method (visual logit lens). Our method outperforms attention weights by 12.5% and cosine similarity by 2.6% in local score evaluation. When combining global and local scores, our method achieves gains of 4.3% over attention weights and 3.0% over cosine similarity. We further observe that high attention weights tend to be assigned irrelevant regions [39]; cosine similarity better localizes object regions but still assigns spuriously high scores to background areas. In contrast, accurately highlights object regions, leading to more reliable patch selection for grounding.
visualize the Top-K patch scores for each metric in Figure 4. From the visualization,
Design choices for global and local scores. We ablate sev- eral key design choices for each scoring function in Table 4. For the global score (Sgloba),, we compare (1) similarity with the last image token embedding, (2) average similarity across all image tokens, and (3) similarity with the last instruction token. The last instruction token performs best, outperform- ing the average similarity by 8%, highlighting its strength in local score (Mteat), capturing scene-level semantics. weiigtte average
Score
Method
LLaVA
Shikra
Last token
65.9
56.2
Global
Average image
Last instruction token
79.3
79.8
Local
Weighted iverage Non-weighted average
75.8
73.0
76.8
GLSIM
Average & Eq (3)
Last inst. & Eq. (3)
83.7
79.2
77.0
$2.0
Table 4:
Design choices for global and local scoring functions.
we compare 1) Logit Lens of local similarities among top-K K patches and (2) non-weighted average as Equation (3), where the latter works slightly better. Finally, combining global and local scores improves performance for

---

84
LLavA Shikra
84
LLaVA
80
LLaVA Shikrr
AUROC (%)
Shikra
70
80
AUROC
80
AUROC
60
78
78
40
76
76
30
4
16
32
64
128
All
0.0
0.2
0.4
0.6
0.8
1.0
5
10
15
20 25 30
(a) Number of patches K
(b) Weighting parameter
(c) Text embedding layer /'
Figure 5: (a) Effect of the number of selected image patches K; (b) effect of the weighting parameter w in Equation (5); and (c) effect of the text embedding layer index I".
both variants of the global score. This confirms that the two signals are complementary and supports the design of our scoring function.
How do the selected number of patches affect the performance? We analyze the impact of varying the number of selected patches K in Equation (2) on object hallucination detection performance in Figure 5(a). Performance improves with increasing K up to K-32 for LLaVA and K-16 for Shikra, after which it degrades. This trend suggests that a small K may fail capture sufficient object information, while a large K introduces irrelevant regions, adding noise. Given that LLaVA processes 576 image tokens, the optimal K roughly corresponds 6% of total image tokens. This highlights the importance of choosing K relative to input resolution for effective OH detection.
How does the weighting parameter affect performance? In Figure 5(b), examine the effect of the weighting parameter w in Equation (5) on OH detection performance. Performance increases with up to 0.6, after which it declines. Smaller values of w place greater emphasis the local score, while larger values prioritize the global score. We find that moderate values consistently yield the best results across models, suggesting that global and local signals are complementarily informative-where the global score captures scene-level semantics, and the local score captures fine-grained, spatial-level semantics. These results support our design choice of combining both components through a balanced weighting scheme, effectively enhancing overall performance.
How does the text embedding layer index affect performance? We examine how the choice of text embedding layer I' influences overall performance when computing embedding similarity in Equation (3) and Equation (4). We fix the image embedding layer to the 32nd layer for LLaVA and the 30th layer for Shikra, as specified in Table 5. As shown in Figure 5(c), the best performance is achieved at the 31st layer for LLaVA and the 27th layer for Shikra. Performance improves with later layers, suggesting that semantic representations are progressively refined later layers. However, it slightly drops afterward, which supports the observation from [51] that the optimal layer for downstream tasks may not necessarily be the final layer. These findings indicate that later-intermediate layers are particularly effective for object hallucination detection. The complete performance matrix over all (1, (") layer pairs is provided in Appendix E.
6 Conclusion
In this paper, we propose GLSIM, novel training-free framework for object hallucination detection, which exploits the complementary strengths of global scene-level semantics and fine-grained spatial alignment by leveraging embedding similarity. Empirical results demonstrate that GLSIM achieves superior performance across diverse families of LVLMs and two representative datasets. Our in in-depth quantitative and qualitative ablations provide further insights into understanding the effectiveness of GLSIM. We hope work will inspire future research on OH detection from diverse perspectives.
Limitations and future work. Our analysis in this paper focuses on object existence hallucinations, as annotations and benchmarks for attribute and relation hallucinations are currently limited. Nonethe- less, would be interesting to investigate further the grounding ability of the Logit Lens technique for attributes and relationships quantify local similarity beyond object presence. Moreover, leveraging accurate OH detection from our method to guide model editing or prediction refinement presents a promising future direction for mitigating object hallucinations.

---

Acknowledgement
We gratefully acknowledge Changdae Oh and Hyeong Kyu Choi for their valuable comments on the draft. Seongheon Park and Sharon Li are supported in part by the AFOSR Young Investigator Program under award number FA9550-23-1-011 National Science Foundation under awards IIS-2237037 and IIS2331669 Alfred P. Sloan Fellowship, and Schmidt Sciences Foundation.
References
[i] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-viing assistant for biomedicine in one day. In NeurIPS, 2023.
[2] Wenliang Dai, Jumnan Li. Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstrucIBLIP: Towards purpose vision- saiionlanguage models with instruction tuning. In NeurlPS, 2023.
[3] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv: 10592, 2023.
[4] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su. Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisti tasks. In CVPR, 2024.
[5] Qinghao Yc, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal modal large language model with modality collaboration. In CVPR, 2024.
[6] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuayi Yang, Lei Zhao, Song XiXuan, et al. Cogylm: Visual expert for pretrained language models. In NeurlPS, 2024.
[7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han. Fei Huang,
Qwen technical report. arXiv preprint arXiv:2309. /6009, 2023.
[8] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseck-vl: towards real-world vision- language understanding. arXiv preprint arXh::2403.05522 05522, 2024.
[9] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In NeurlPS, 2023.
[10] Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang. Qiang Fu, and Ke Liu. A survey of multimodel large language models. Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, 2024.
[11] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burs,, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In EMNLP, 2018
[12] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In WACV, 2024.
[13] Bufang Yang, Lixing He, Kaiwei Liu, and Zhenyu Yan. Viassist: Adapting multi nolti-modal large language models for users with visual impairments. In 2024 IEEE International Workshop on Foundation Models for Cyber-Plysical Physical Systems & Internet of Things (FMSys), 2024.
[14] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong Hc, Zongho Han, 7heng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint 2eXi::24041199.0 2024.
[15] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. EMNLP, 2023.
[16] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint ArXiv:2306.11 13394, 2023.
10

---

[17] Junyang Wang. Yuhang Wang. Guohai Xu. Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu. Ming Yan, Ji Zhang, al. Amber: An llm-free multi-dimensional benchmark for milms hallucination 2023.
evaluation. arXiv preprint arXiv:231
[18] Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang. David Fouhey, and Joyce Chai. Multi-object hallucination in vision language models. In NeurlPS, 2024.
[19] Suzanne Petryk, David M Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E Gonzalez, and Trevor Darrell. Aloha: A new measure for hallucination in captioning models. In NACCL,, 2024.
[20] Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du Faithscore: Fine-grained evaluations of hallucina- tions in large vision-language models. In EMNLP Findings, 2024.
[21] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li. Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. In ACL Findings, 2024.
[22] Fuxiao Liu, Kevin Lin,
I Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In ICLR, 2024.
[23] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR, 2024.
[24] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In AAAI, 2024.
[25] Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. allm-as-a-judge: Assessing multimodal amm-as-a-judg with vision-languag benchmark. In /CML, 2024.
[26] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng. Chelsea Finn. Mohit Bansal, and Huaxiu Yao. Analyzing and miligating object hallucination large language models. ICLR, 2024.
[27] Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-languan models: Interpreting, detecting and mitigating object hallucinations via attention lens. In CVPR, 2025.
[28] Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision- language representations to mitigate hallucinations. In ICLR, 2025.
[29] Anirudh Phukan, Harshit Kumar Morj, Apoorv Saxena, Koustava Goswami, al. Beyond logit lens; Contextual embeddings for robust hallucination detection & grounding in vims. NACCL, 2025.
[30] nostalgebraist. Interpreting gpt: The logit lens. https:/ / /www. lesswrong com/ com/posts/ tit-the-looit-eenn 2020. LessWrong.
[31] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang. Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
[32] Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. In ICLR, 2024.
[33] Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In CVPR, 2024.
[34] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almcida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2303::::30008777 2023.
[35] Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu. Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint 2.3: 15/26, 2023.
[36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 230. 2023.

---

[37] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 2019.
[38] Shi Liu, Kecheng Zheng. and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in Ivims. In ECCV, 2024.
[39] Seil Kang, Jinyeong Kim, Junhycok Kim, and Scong Jac Hwang. See what you told: Visual attention sink in large multimodal models. In ICLR, 2025.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In ICML, 2021.
[41] Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Rethinking and controlling object existence hallucinations in
Li, Tan Yan, and Xiangjun Fan, large vision-language models for detailed caption. https///penrevieeweefffrumm http:/fooummmmddddd 9ihi QuQQQ 2023.
[42] Xuan Gong. Tianshi Ming. Xinpeng Wang. and Zhihua Wei. Damro: Dive into the attention mechanism of Ivlm to reduce object hallucination. arXiv preprint arXiv:2410005114, 2024.
[43] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[44] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
[45] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
[46] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In NeurlPS, 2024.
[47] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijic Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[48] Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye. Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
[49] Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In ICLR, 2021.
[50] Sarthak Jain and Byron Wallace. Attention is not explanation. NACCL, 2019.
[51] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz Ziv. Layer by layer: Uncovering hidden representations in language models. In JCMI, 2025.
[52] A Paszke. Pytorch: An imperative style, tegh-performance deep learning library. In NeurlPS, 2019.
[53] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang. Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2025.
[54] Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribution detection and selective generation for conditional language models. In ICLR, 2023.
[55] Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. In TMLR, 2022.
[56] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In EMNLP, 2023.
[57] Amas Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. In EMNLP Findings, 2023.
[58] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In ICLR, 2023.
12

---

[59] Xuefeng Du, Chaowei Xiao, and Sharon Li. Haloscope: Harnessing unlabeled Ilm generations for hallucination detection. In NewrIPS, 2024.
[60] Seongheon Park, Xuefeng Du, Min-Hsuan Ych, Haoho Wang, and Yixuan Li, Steer Um latents for hallucination detection. In ICML, 2025.
[61] Qing Li, Jiahui Geng, Chenyang Lyu, Derui Zhu, Maxim Panov, and Fakhri Karray. Reference-free hallucination detection for large vision-language models. In EMNLP Findings, 2024.
[62] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. In ICLR, 2023.
[63] Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. V1-uncertainty: Detecting hallucination in large vision- language model via uncertainty estimation. arXiv preprint arXiv:2411. 2024.
[64] Min-Hsuan Yeh, Max Kamachee, Scongheon Park, and Yixuan Li. Halluentity: Benchmarking and understanding entity-level hallucination detection. In TMLR, 2025.
[65] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint 21..:2240..2127.. 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction discuss in detail the studied problem and the contributions of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
Limitation is discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclose the experimental details in Section 5.1 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e-g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

reproducible, they Answer: [Yes]
Justification: We use datasets that are publicly available. Detailed instructions for repro- ducing our results are provided in Section 5.1 and Appendix A. An anonymous GitHub repository containing our code is linked in the abstract for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. wttpccccpppcccccccccccccccccccct ggtccigg/////////iiieeooi/iii Soodeeebmisssiiiieooea ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Detailed instructions for training and test are provided in Section 5.1 and Ap- pendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: All reported results are averaged over three different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g.mu negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details on compute resources in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttopp:://euus http///nurrrp.ppeeee wctpppblcc/.thiccccceeeeeeeeeeee
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that our work adheres to its guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader societal impacts in Appendix F.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work introduces method for detecting hallucinations in LVLMs. We believe it does not present a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite relevant works for the resource we use for the experiments in Sec- tion 5.1.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs as part of our core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: : / ht////uurpsscccccco caarrrips.ccc/ooeeeeennee/////// /LLM) for what should or should not be described.
20

---

Appendix
Contents
Implementation Details
Additional Qualitative Results
22
B.1 Qualitative Results
22
B.2 Object Grounding
23
C Related Works
23
C.I Baselines
23
C,2 Extended Literature Review
24
Further Ablation Studies
25
D.I Results for Additional Models
25
D.2 Attribute and Relational Hallucinations
................................
25
D.3 Visualization of Score Distributions
26
D.4 Comparison with POPE
26
D.5 Comparison with External Model-Based Methods
27
D.6 Multi-token Objects
27
D.7 Distance Metric
27
Layer-wise Performance Matrix
28
Broader Impacts
30
A Implementation Details
We implement our method using greedy decoding with maximum generated token length of 512. The layer indices (1, l'). the number of selected patches K, and the weighting parameter w used for computing the final score selected based on separate validation set, as detailed in Table 5, For multi-token objects, we use the first token to compute the scores and consider the first occurrence of each object for hallucination detection. The total number of generated objects is shown in Table 6. For all experiments, we report the average over three different random seeds. All experiments are conducted using Python 3.11.11 and PyTorch 2.6.0 [52], single NVIDIA A6000 GPU with 48GB of memory.
Hyperparameters
Model
MSCOCO
Objects365
Model
Layer indices
Real
Hallu,
Real
Hallu.
LLaVA-1.5-7b
(32, 31)
32
0.6
LLaVA-L.5-hh
14,910
4,121
14,357
9,850
LLaVA-1.5-13b
(40, 38)
32
0.6
LLaVA-1.5-13b
15,372
3,687
15,086
9,672
MiniGPT-4
(32, 30)
0.5
MiniGPT-4
11.642
2,282
11,603
7.222
Shikra
(30, 27)
16
0.6
Shikra
15.724
4.727
15,063
10,350
Table 5: Hyperparameters.
Table 6: Number of generated objects.
21

---

B Additional Qualitative Results
B.1
Qualitative Results
(a)
(b)
"people"
Object: "oook"
Imputimage
Imput ImputImage Image
Description
The image testurez yellow and purple train traveling down the
Description
The inage teatures group oz children gathered around a dining table. enjoying neal acqutiaz.
tzacks.Ther Thare are geverel Seople in the image, sons of chen standing near the cratn, pesetbiy wateing for it to stop or diserbark.
Globa LocalotSim
Gobal GLSm
there are mil-ppe bowla the table, 2005 viztble
the center of the table.
Imput Image
Object: "remote"
Imput Imagge
Object: "olock"
Descriptio
iaage person they
Descoipliin
Trage
their hand, possitiy playing gare.
foregroind
Innput Image
Object: 'chair"
Iipptt Image
Object: "sink"
Descriptio
bed in center the brown blanket,
Desccippiinn
black shire and tie. looking the canera, possiblly ng for pictuze. in the backgsound, pink viziaie.
le oovered with s160 present in the pealtioned
Globe LocaloL3m
Gobal CLSm
Figure 6: Additional qualitative evidence. the generated descriptions, hallucinated objects are highlighted in red. The localized image regions are shaded with the same color as their corresponding objects. The gray line shows threshold value т. an object's score is lower than threshold r, consider it a hallucination. In (a), the local score successfully compensates for the failure of the global score, while in (b), the global score offsets the limitations the local score.
22

---

B.2
Object Grounding
Object: "mouse"
Object: "bull"
Object: "vase"
Object: "sandwich"
Object: "remote"
Object: "donuts"
Figure 7: Object grounding results for real objects. We visualize the Top-K Logit Lens probabili- ties at the 32nd layer of LLaVA-115-7B. Ground-truth bounding boxes are shown in red.
Object: "cake"
Object: "people"
Object: "car"
SALISBURY
LANDS LANE
C CRUIKSHANK
ROSS
Object: "cup"
Object: "chair"
Object: "ball"
Figure 8: Object grounding results for hallucinated objects. We visualize the Top-K Logit Lens probabilities at the 32nd layer of LLaVA-1.5-7B.
C Related Works
C.1 Baselines
Negative Log-likelhhood. Zhou et al. [26] represents the probability autoregressive decoding for each object token as p(o yıj, v), where j denotes the positional index of object For each object o, the corresponding hallucination score is defined as:
Sall =-log pog Y v).
(6)
To align with the definition in Equation (1), we use s' = Sall-
23

---

Entropy. We further investigate the object-level hallucination score by estimating the entropy [49] of the token probability distribution at position j:
Σ
Eply
ply Yes. logply ve..
Pentropy
BEV
To align with the definition in Equation (1), we use sentropp - Sentmpy
Internal Confidence. Jiang et al. [28] apply the logit lens to image representations, enabling the analysis of how visual features are transformed into textual predictions. To quantify the model's confidence for object hallucination detection, the internal confidence score is computed as the maximum softmax probability of the object word o across all image representations and layers. Following the notations introduced in Section 4.2, the hallucination score is defined as:
SIC
max (L+)(x:))o): iE[L] (E[N]
(8)
where L denotes the total number of layers and denotes the total number of image patches.
Summed Visual Attention Ratio (SVAR). The Visual Attention Ratio (VAR) quantifies the interaction of a generated token with visual information by summing its attention weights assigned to image tokens in a specific attention head h and layer f:
VAR (O)
&
Σ A (o. vo.u.)
where ((c.)) (o, 0;) represents the attention weight from object token to image token t; h-th head (-th layer. Building on this, Jiang et al. [27] define the Summed Visual Attention Ratio (SVAR), which measures the overall visual attention by averaging VAR scores across all heads and summing over a range of layers. Specifically, for an object token o within layers &s to L1s. SVAR is computed as:
18 H
SSVAR =
H VAR (AR)
Σ
(o),
(10)
8 Am
where H denotes the total number of attention heads.
Contextual Lens. To detect sentence-level hallucination, Phukan er al. [29] compute the maximum cosine similarity between the average embedding of the generated description at specific layer lT and each image embedding layer I,.
1 m
Sentence-level Score
max sim kE[N] M j=1
Σ her (yj), hi, (v:)).
(11)
To compute the object-level hallucination score, we modify the original score with:
SCL- max sim(hu (o), ho,, (w:)).
(12)
[E[N]
C.2 Extended Literature Review
Sentence-level hallucination detection in LLMs and LVLMS aims classify an entire generation as either hallucinated or correct, providing a coarse graiine assessment of factuality [53]. A plethora
of work addresses sentence-level hallucination detection large language models (LLMs)) by design- ing uncertainty scoring functions, such as utilizing token generation probabilities [54], prompting LLMs to quantify their confidence [55], and evaluating consistency across multiple responses [56]. Specifically, internal state-based methods leverage latent model embeddings [57]. employing tech-
niques such as contrast-consistent search [58], identifying hallucination--elan subspaces [59], or reshaping the latent space for hallucination detection [60].
24

---

Recently, reference-free sentence-level hallucination detection for large vision- models (LVLMs) has attracted research attention. Li et al. [61] first compare uncertainty quantification methods from LLMs for application to LVLMs. Inspired by [62], VI.-Uncertainty [63] estimates uncertainty by measuring prediction variance across semantically equivalent but perturbed prompts.
In contrast to these works, we propose the hallucination scoring function for object-level hallucination detection in LVLMs, which provides fine-grained assessment by localizing hallucinations within generations rather than classifying entire outputs [64]. Our method leverages latent embeddings from both visual and textual modalities and explores intrinsic metrics tailored LVLMs.
D
Further Ablation Studies
D.1 Results for Additional Models
Method
InstrucBBII
LLaVA-NeXT
Cambrian-1
Qwen2.5-VL
Intern
NLL [26]
ICLR'24
65.1
50.1
59.1
$5.7
Entropy [49]
ICLR'21
65.6
57.5
50.2
59.1
$5.5
Internal Conf. [28]
ICLR'25
81.9
77.8
65.4
60.3
63.3
SVAR [27]
CVPR'25
78.4
76.9
60.5
70.8
68.8
Contextual Lens (29]
NACCL'25
83.0
70.1
63.4
65.1
65.2
GLSIM (Ours)
85.0
81.4
79.7
76.1
73.2
Table 7: Performance on additional models evaluated on MSCOCO.
We further evaluate our approach on five additional advanced large vision-language mod- els InstructBLIP [2], LLaVA-NeXT-7B [45], Cambrian- [46], Qwen2.5-VL-77B [47], and InternVL3-8B [48] using the MSCOCO dataset. Our method consistently surpasses baseline ap- proaches, yielding AUROC improvements of 2.0% on InstructBLIP, 4.2% on LLaVA-NeXT, 14.3% on Cambrian-1, 5.3% on Qwen2.5-VL, and 4.4% on Intern These results highlight the robustness
of GLSIM across diverse architectures and model scales.
D.2 Attribute and Relational Hallucinations
Attribute Hallucination
Relational Hallucination
Method
ILLVA-I.5-7B
1LAVA-1.5-13B
Qwen2,5-V1.-7B
LLaVA-1.5-7B
LLaVA-1,5-13B
Qwen2.5-VL.-77
NLL
58.62
60.50
$6.89
57.06
$7.35
54,90
Entropy
52.21
55.32
55.84
55.72
56.27
55,03
Internal Confidence
74.24
73.67
70.06
69.38
68.94
62.09
SVAR
67.03
68.62
71.09
61.20
65.88 83
63.01
Contextual Lens
74.02
75.48
71.98
66.46
69.85
64.88
GLSI (Ours)
77.19
78.07
74.09
70.03
73.64
68.95
Table 8: Comparison of different methods on attribute and relational ha salluuinnsi
Given the lack of comprehensive benchmarks for token-level detection of attribute and relational hallucinations in open-ended generation, our study primarily focuses on object existence hallucina- tions. Nonetheless, addressing these more complex forms remains important open challenge for real-world deployment. Attribute hallucination refers to assigning incorrect properties to objects (e.g., "a red car" when the car is blue), whereas relational hallucination arises when relationships between objects are misstated (c.g., "a cat sitting on a table" when is under the table).
To explore the applicability of GLSIM in these settings, we conducted an extension study. Specifically, generated captions for each LVLM using 500 randomly selected images from the MSCOCO validation set and employed GPT-4o [65] to produce pseudo ground-truti annotations for both attribute and relational hallucinations. We then computed levenllevel GLSIM scores and aggregated them by averaging across attribute- -bbbetttee-oettttttteeee spans (e.g., "a red car") and object-relation spans (e.g.. "a cat sitting on table").. These aggregated scores served as unsupervised estimates of hallucination likelihood. Despite its simplicity and lack of task-specific modifications, GLSIM demonstrated meaningful detection capabilities and consistently outperformed baseline methods across both hallucination types.
25

---

D.3 Visualization of Score Distributions
NLL
Ours
Real
Real
Hallucinated
Hallucinated
Frequency
Frequency
Score
Score
Figure 9: Score distribution for negative log-likelihood [26] vs. our method.
We provide score distribution for the negative log- log-likelihood (NLL) [26] and GLSIM (ours) in Figure Our approach exhibits a more distinct separation between the real and hallucinated data distributions. This enhanced separation highlights the effectiveness of our global and local scoring designs, as well as their combination strategy, contributing to more reliable detection performance compared to existing method.
D.4
Comparison with POPE
Model
Method
Time (s)
ACC ?
PR (True) f
PR (False) t
Recall t
F1 t
POPE [15]
8.1
79.5
80.3
70.8
96.7
87.8
LLVVA-155-BB
Ours
81.8
83.1
71.8
96.6
89.3
POPE [15]
80.9
81.2
75.0
98.6
89.1
LLaVA-1.5-13B
Ours
1.8
83.6
86.4
75.2
95.0
90.5
Table 9: Comparison with POPE on MSCOCO.
To compare with prompting bassdd methods such as POPE [15], we extract objects from the generated captions produced using the prompt "Describe this image in detail." We then convert these objects into a of Yes-or-No short-form questions. We prompt the LVLM with the following template:
Input prompt for POPE evaluation
Prompt:
Q: there [object] in the image?
A:
Responses containing "Yes" are labeled as (real), and all others as (hallucinated). For evaluation, each question is labeled using the ground-truth object annotations from the MSCOCO dataset, following the same procedure as our main pipeline. We select the decision threshold r that maximizes the Fl score. We report accuracy (ACC), precision (PR) for real (true) and hallucinated (false) objects, recall, and FI score with % in Table 9. To compare computational efficiency, we also report the average inference time (Time) required to detect object hallucinations per image. For LLaVA-1.5-7B. our method achieves 2.8% improvement in the precision of real objects and a substantial 1.0% improvement in the precision of hallucinated objects compared to POPE. From a computational perspective, our method requires only single forward pass for generation, whereas prompting- methods like POPE require (1 + C) forward passes-one for generating the description and for the number of object-level verification prompts. Notably, our method reduces inference time by 6.5
26

---

seconds per generated description compared to POPE, demonstrating substantial efficiency gains. These results demonstrate the superior effectiveness and computational efficiency of our method in detecting object hallucinations, even when compared to prompting-based approach, particularly in accurately filtering hallucinated content while maintaining precision on real objects.
D.5
Comparison with External Model-Based Methods
Method
Time (6)
Accuracy
Precision (Real) f
Precision (Hallue.) 1
Recall
Fl
External-base
78.6
78.9
70.8
87.6
GLSIM (Internal)
1.6
81.8
83.1
71.8
96.6
89.3
Table 10: Comparison with external model-based method.
Recent approaches have explored hallucination detection using external knowledge sources, such as large language models (L..Ms) and large vision-language models (I.V.Ms). These methods prompt an external model with triplet input consisting of the image, instruction, and the generated caption. Such approaches are inherently limited to post-generation evaluation, as hallucinations can only be assessed after the entire caption has been produced. Morcover, they often require multiple forward passes-one to generate the caption from the base model and additional passes for external evaluation-leadingg to substantial computational overhead.
In contrast, GLSIM operates during the token decoding phase of the base LVLM, enabling real-time hallucination detection at the token level. This property makes GLSIM particularly suitable for interactive or streaming applications where immediate feedback is essential. Importantly, it requires only a single forward pass and does not rely on any external models, thereby avoiding additional uncertainty introduced by potentially hallucination-proon external evaluators.
Table 10 presents comparison the MSCOCO dataset using LLaVA-1.5-7B as the base model, and LLaVA-1.5-133 as the external evaluator. GLSIM achieves a substantial inference efficiency advantage, requiring only 1.6 seconds per image compared 9.3 seconds for the external model- based approach, corresponding to an 82.8% speedup. Despite this efficiency gain, GLSIM also delivers competitive or superior reliability, outperforming the external method across most evaluation metrics. These findings highlight that GLSIM offers a practical and efficient alternative for real- world deployment, combining speed, reliability, and independence from external models in fully self-evaluating, unsupervised, training-free free manner.
D.6 Multi-token Objects
Token
LLaVA-155-7B
LLaVA-1.5-13B
First
83.7
84.8
Last
83.3
84.0
Average
83.4
84.2
Table 11: Comparison of token selection strategies for multi-token objects.
To compute the visual logit lens probability for object grounding and the embedding similarity for hallucination detection, we default to using the first token of multi-token objects. To evaluate the impact of this design choice, we conduct an ablation study comparing three strategies: (1) using the first token, (2) using the last token, and (3) taking the average across all tokens. Results on the MSCOCO dataset in Table 11 show that the first-token strategy is most effective, since the first token often captures the core semantic meaning of the object.
D.7 Distance Metric
We investigate the impact of the choice of distance metric for computing embedding similarity on overall performance on the MSCOCO dataset, as shown in Table 12. Specifically, we compare cosine similarity and L22 distance (i.e., Euclidean distance) as the underlying metric our GLSIM score. On the 1-LaVA-155-77 model, L.2 distance yields slightly better performance, improving AUROC by 0.3%. In contrast, on the Shikra model, cosine similarity outperforms 1.2 distance by 1.7%. These results suggest that the effectiveness of distance metric may depend on the model's training
27

---

Metrie
LLaVA
Shikra
1.2
80.2
77.2
Glohal
Cosine
79.3
78.9
79.9
75.6
Local
Cosine
78.8
76.8
84.0
81.3
G &
Cosine
83.7
83.0
Table 12: Ablation on the impact distance metric.
strategy and architecture. Nevertheless, both metrics consistently outperform the baselines Table 1, demonstrating the robustness of our method across different metric designs.
E
Layer-wise Performance Matrix
We provide the full performance (AUROC) matrix across all combinations of image and text bedding layers (I, l') on the MSCOCO dataset, illustrating how the composition of layers influences hallucination detection performance.
(LaVA-1.5-70
0.8
0.7
Text Embedding Layer
10 2121222222222 2122222222 21 28TH 3 12 10
0.4
$ 9011 12:554 15 20 21 Image Embedding Layer
Figure 10: Performance matrix of LLaVA-1.5-7B.
28

---

LLaVA-1.5-13B
0.8
0.7
Text Embedding Layer /
0.6
0.5
0.4
0.3
Image Embedding Layer
Figure 11: Performance matrix of 1.5VA-1.--13B.
MiniGPT-4
0.8
Text Embedding Layer
3231 3024 21 313655 222222 32 17 11
0.7
0.6
0.5
Image Embedding Layer
Figure 12: Performance matrix of MiniGPT-4,
29

---

Shikra
0.8
Text Embedding Layer
32 12 20 222 72 21 22324 37
0.6
0.5
8. 9 14 21
Image Embedding Layer
Figure 13: Performance matrix of Shikra.
Broader Impacts
Ensuring the reliability of LVLMs is critical as they are increasingly deployed in high-stakes do- mains such as autonomous navigation, medical diagnosis, and accessibility applications. This work addresses critical challenge of object hallucination detection, which identifies objects mentioned in generated outputs that are not present in the input image. We propose practical, training-free method that combines global and local signals from pre-trained LVLMs to enhance hallucination detection. Our research not only advances the technical frontier in this area, but also contributes to the development of trustworthy AI systems, fostering confidence in the deployment LVLMs in safety-critical applications.
30


Reviews:
[{"review_id": "AGgcw5cWZM", "reviewer": "Reviewer_5sQ6", "summary": "This paper presents a training-free method for detecting object hallucinations in large vision-language models (LVLMs). The authors propose a new scoring mechanism called GLSIM (Global-Local Similarity). This method assesses whether an object mentioned in a model-generated caption is truly in the image by evaluating: (1) Global similarity, which captures how well the object semantically aligns with the overall scene representation; (2) Local similarity, which measures whether the object is visually grounded in any specific image region using internal representations from the model. The GLSIM score combines both signals into a single metric. The authors evaluate GLSIM across multiple LVLMs and datasets. The results show that GLSIM outperforms recent baselines.", "questions": "1. Can GLSIM be extended to detect other types of hallucinations? (I am not asking for experiments showing that GLSIM can handle these cases, but it would be useful to hear your thoughts on how the framework might be adapted to detect these more complex hallucinations.)\n2. Can GLSIM handle high-resolution images? The performance of local similarity in GLSIM depends on the accurate localization of objects within the image. This is especially critical in high-resolution images, where objects may occupy only a small portion.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The proposed GLSIM is a training-free hallucination detection method, making it practical for scenarios without annotations.\n2. The authors provide extensive experiments across multiple LVLMs and compare with recent strong baselines.\n3. The paper includes in-depth ablation studies that analyze the impact of global and local similarity and different design choices for them.\n\n**Weaknesses**\n1. GLSIM focuses on object-level hallucinations, but real-world hallucinations often involve relationships or attributes that are incorrect (e.g., incorrect spatial relationships or erroneous object attributes). GLSIM might miss these more complex hallucinations.", "was_revised": true, "final_justification": "The authors' responses have addressed my previous concerns, demonstrating the generalizability of the proposed method to other types of hallucination as well as to high-resolution images. Therefore, I will maintain my original positive scores.", "rebuttal": "We appreciate the reviewer’s comments and suggestions. We are encouraged by the recognition of our method as practical, along with the acknowledgment of our comprehensive experiments and thorough ablation studies. \n\nWe also thank the reviewer for the interesting and insightful questions, which we are happy to address below:\n\n---\n> A1. How can we extend GLSIM to detect other types of hallucinations?\n\nGreat point, and thank you for inviting our thoughts! We agree that attribute-level and relational hallucinations are important challenges for real-world applications. As noted in our Limitations and Future Work section, our current study focuses on object existence hallucinations _because publicly available token-level annotations and benchmarks for attribute and relational hallucinations in open-ended generation settings remain limited_.\n\nThat said, we believe GLSIM can be naturally extended to address these more complex types of hallucination, which are typically defined as follows:\n\n- Attribute Hallucination: Incorrect properties of objects are described (e.g., “a red car” when the car is blue).\n- Relational Hallucination: The relationship between objects is incorrectly stated (e.g., “a cat sitting on a table” when it is under the table).\n\nTo this end, we explored whether the current GLSIM score, _without any task-specific modification_, can capture attribute and relational hallucinations. We generated captions with each LVLM for 500 randomly selected images from the MSCOCO validation set. We then used a powerful multi-modal model, `GPT-4o`, to produce pseudo ground-truth annotations for both attribute and relational hallucinations. We then computed token-level GLSIM scores and aggregated them by averaging across attribute–object (e.g., “a red car”) and object–relation (e.g., “a cat sitting on a table”) spans. These aggregated scores served as unsupervised estimates of hallucination likelihood. While this is a simple implementation, **GLSIM demonstrated meaningful detection capabilities without requiring task-specific modifications, and consistently outperformed baseline approaches**.\n\nBeyond methodology, one of the key challenges in this direction is the lack of comprehensive benchmarks for attribute and relational hallucinations at the token or span level. Given the open-ended nature of caption generation, constructing such annotations is non-trivial, especially for nuanced or fine-grained visual concepts. Developing such benchmarks would be a crucial step toward enabling fair and systematic evaluation in this area.\n\nWe view these as promising future directions and believe GLSIM can serve as a promising baseline for exploring more complex forms of hallucination detection.\n\n- Attribute Hallucination\n\n| Method  | LLaVA-1.5-7B | LLaVA-1.5-13B | Qwen2.5-VL-7B\n|-|-|-|-|\n| NLL            | 58.62 | 60.50 | 56.89|\n| Entropy        | 52.21 | 55.32 | 55.84|\n| Internal Confidence | 74.24 | 73.67 |70.06 |\n| SVAR           | 67.03 | 68.62 | 71.09|\n| Contextual Lens| 74.02 |75.48 | 71.98|\n|**GLSIM (Ours)**    | **77.19** | **78.07** |**74.09**|\n\n- Relational Hallucination\n\n| Method  | LLaVA-1.5-7B | LLaVA-1.5-13B | Qwen2.5-VL-7B\n|-|-|-|-|\n| NLL            | 57.06 | 57.35 | 54.90|\n| Entropy        | 55.72 | 56.27 | 55.03|\n| Internal Confidence | 69.38  | 68.94 |62.09 |\n| SVAR           | 61.20 | 65.83| 63.01|\n| Contextual Lens| 66.46 | 69.85 | 64.88|\n|**GLSIM (Ours)**    | **70.03** | **73.64** |**68.95**|\n\n---\n> A2. Can GLSIM handle high-resolution images? \n\nGood point! **Yes, GLSIM is compatible with high-resolution inputs**. We acknowledge that processing high-resolution inputs is critical for reliable real-world deployment. GLSIM remains effective in such scenarios because its local similarity component operates on patch-level visual tokens generated by the vision encoder.\n\nIn Section 5.2, we show that GLSIM outperforms other methods on the Objects365 dataset, which contains extremely high-resolution images—up to 8412×5608, significantly larger than the 640×480 resolution typical of MSCOCO.\n\nTo further demonstrate GLSIM’s robustness on high-resolution inputs, we conducted additional experiments on 500 randomly sampled images from the top 20% highest-resolution images in the Objects365 dataset. We evaluated our method using two advanced LVLMs: `LLaVA-1.5-7B` and `LLaVA-NeXT`, as well as `Qwen2.5-VL-7B` [1], a state‑of‑the‑art model that processes images at their native resolution, dynamically converting higher‑resolution inputs into thousands of visual tokens. As input resolution increases (e.g., from 224×224 to 1024×1024), `Qwen2.5-VL` adjusts its number of visual tokens accordingly, providing finer spatial granularity and enabling more precise object localization for GLSIM's local similarity component.\n\nThese results demonstrate that **GLSIM consistently outperforms all baseline methods, confirming its effectiveness on high-resolution images**.\n\n\n| Method  | LLaVA-1.5-7B| LLaVA-NeXT | Qwen2.5-VL-7B |\n|-|-|-|-|\n| NLL            | 60.58 | 56.57 | 54.30|\n| Entropy        | 60.91 | 56.17 | 54.28|\n| Internal Confidence | 65.73 | 70.04 | 67.09|\n| SVAR           | 64.21 | 69.90 | 68.32|\n| Contextual Lens| 68.37 | 66.93 | 70.01 |\n|**GLSIM (Ours)**    | **71.57** | **72.18** | **73.04**\n\n\n[1] Bai et al., \"Qwen2.5-VL Technical Report,\" arxiv:2502.13923"}, {"review_id": "slfFSic4SR", "reviewer": "Reviewer_mtgR", "summary": "This paper focuses on the issue of object hallucination in LVLMs and proposes an unsupervised, external-model-free detection method called GLSIM. GLSIM combines global semantic similarity and local visual anchoring similarity to improve the accuracy and robustness of hallucination detection. The authors conduct systematic evaluations on several mainstream LVLMs (e.g., LLaVA-1.5, MiniGPT-4, Shikra) and datasets (e.g., MSCOCO, Objects365), demonstrating significantly better performance compared to existing methods.", "questions": "I would like to know how the authors determine or fix the decision threshold. From the case studies provided in the paper, many of the GLSIM scores appear to be close to the threshold, which suggests that the method may be prone to misclassification around the boundary. Are there any strategies or mechanisms to mitigate such borderline errors?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Compared to many methods that rely on annotated data or additional LLMs, GLSIM is more practical, as it requires no external supervision or judge models, making it easier to deploy.\n\n2. The experiments in this paper are comprehensive, covering mainstream open-source models, and as shown in Appendix E, the proposed method demonstrates a certain degree of generalizability.\n\nWeaknesses:\n1. One concern I have about the proposed method is that it involves too many tunable parameters, such as the threshold for classifying an object as hallucinated and the choice of layers used for score computation.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thoughtful comments and valuable feedback. We are encouraged by the recognition of the practicality and generalizability of our approach, as well as the comprehensiveness of our experiments. \n\nBelow, we address your key concerns.\n\n---\n> A1. Detection threshold and the choice of layers\n\nThank you for this thoughtful observation!\n\nIn our evaluation, we use a threshold-independent metric, AUROC (Area Under the Receiver Operating Characteristic curve), which is standard in binary classification for comparing model ranking performance over all possible decision thresholds.\n\nHowever, for deployment purposes, as you rightly noted, we do need to choose a specific detection threshold $\\tau$ to convert continuous hallucination likelihood scores into binary decisions (hallucination vs. non-hallucination). In practice, this threshold can be set using a small labeled validation set (e.g., 100 samples), depending on downstream priorities (such as higher precision vs. higher recall). A common and effective strategy is to select the threshold that maximizes the F1 score on the validation set, which we also use for the case study.\n\nCompared to fixed rule-based heuristics, our continuous scoring approach provides greater **flexibility**, making it well-suited for real-world scenarios where hallucination severity and uncertainty may vary across applications.\n\nFor the choice of layers $(l,l')$, as you helpfully pointed out, we refer to our analysis in Appendix E. Our layer-wise performance analysis shows that selecting **middle-to-late layers (typically after 70% depth)** for both image and text embeddings **yields consistently strong and stable performance across different LVLMs**. These layers tend to encode richer semantic information, and we observe low variance across models, which minimizes sensitivity and reduces the practical burden of hyperparameter tuning.\n\n---\n> A2. How to mitigate borderline errors?\n\nThat’s an excellent point raised. We acknowledge that borderline errors are a valid concern in real-world deployment.\n\nTo mitigate such borderline misclassifications, a promising direction is to train a lightweight meta-classifier using pseudo-labels derived from GLSIM itself. Specifically, we can select a threshold on a held-out validation set that maximizes F1 score, which allows us to generate high-quality pseudo-labels despite lacking ground-truth object annotations. These labels then supervise a linear classifier whose input features include the GLSIM score as well as auxiliary uncertainty metrics like negative log-likelihood and entropy. This meta-classifier can learn an **adaptive decision boundary**, replacing the inflexible numeric threshold and allowing the system to output calibrated probabilities. \n\nThis strategy has proven useful in safety-critical applications where conservative behavior is preferred over false positives or negatives. In our context, such an adaptive scoring mechanism allows GLSIM to retain high precision and recall for confident cases, while flagging ambiguous ones for downstream handling (e.g., human-in-the-loop review or secondary verification). By supporting low-confidence abstention, the meta-classifier improves robustness near the threshold and makes GLSIM more practical for real-world deployment. We will include this discussion in the revised version to clarify how GLSIM can be made more resilient to borderline uncertainty -- thanks again for your insightful comment!"}, {"review_id": "SQTGVjg6Vv", "reviewer": "Reviewer_S1xd", "summary": "This paper proposes a method for detecting object hallucinations in large vision-language models (LVLMs) by combining global and local similarity scores. The local score captures the alignment between object mentions and specific image regions using embedding similarity, while the global score measures the semantic coherence of the object within the overall scene context. By integrating these complementary signals, the proposed approach enables more accurate and robust hallucination detection. Extensive experiments on multiple LVLMs and datasets demonstrate consistent improvements over prior methods.", "questions": "Line 286: It is unclear what the reported 0.3–11% AUROC improvement is measured against. Clarifying the specific baseline used for comparison would improve clarity. \n\nI would consider raising the score if the identified weaknesses are properly addressed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n[1] The method is grounded in a well-motivated analysis of object hallucination issues in LVLMs and is supported by consistent performance gains across models and datasets.\n\n[2] The paper provides a detailed comparison with Internal Confidence (IC), highlighting the limitations of probability-based scores and demonstrating consistent improvements through semantic similarity-based scoring (Section 5.2).\n\n[3] The authors conduct thorough ablation studies on key design choices, including object grounding strategies and scoring variants, offering clear empirical justification for their final architecture (Section 5.3). \n\nWeaknesses:\n\n[1] The paper lacks a clear description of how target objects are selected for evaluation—whether all generated nouns are considered or restricted to predefined object categories (e.g., COCO 80 categories), which may impact the consistency of the evaluation.\n\n[2] While the method is evaluated on multiple representative LVLMs, it remains unclear whether the proposed approach maintains state-of-the-art performance on stronger models such as InstructBLIP or Qwen-VL.", "was_revised": true, "final_justification": "The authors have well addressed my concerns. I am satisfied with the result and will maintain my score.", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and constructive feedback. We are encouraged by the recognition of our motivation, performance improvements, and comprehensive ablation studies. \n\nWe are also grateful for the insightful questions and are happy to clarify the points raised as follows:\n\n---\n> A1. How are target objects selected for evaluation?\n\nThank you for raising this important point. As noted in **Lines 226–227** of the main paper, we follow a verified and widely adopted evaluation protocol from prior works [1], where object mentions are matched against the ground-truth annotations provided in the MSCOCO and Objects365 datasets. This ensures consistency and comparability across different models and detection methods.\n\nSpecifically, we extract objects from the generated captions and compare them with ground-truth object classes using exact string matching with synonym expansion. **This standardized evaluation is applied uniformly across all tested LVLMs and baselines in our experiments.**\n\nWhile our main results are based on a benchmark-grounded setup to ensure rigorous and fair evaluation, we note that GLSIM can be naturally extended to real-world scenarios with unrestricted vocabulary settings. In such cases, lightweight object extraction tools such as noun chunking or external taggers like `spaCy` [2] can be used to identify candidate noun or object phrases from generated text in real time. These phrases can then be processed by GLSIM’s mechanism, allowing for consistent and automated hallucination detection even without a fixed set of predefined object categories.\n\nWe will revise the main paper to provide a more detailed clarification.\n\n---\n> A2. Performance on advanced models\n\nGreat point! To address this, we conducted additional experiments on two recent and advanced LVLMs: `Qwen2.5-VL-7B` [3] and `InternVL3-8B` [4]. For further evaluations on other strong models, including `Cambrian-1`, `LLaVA-NeXT`, and `InstructBLIP`, please refer to **Appendix D.1**.\n\nWe evaluate object hallucination detection performance on 2,000 randomly sampled images from the MSCOCO validation set and report the average AUROC (%) over three different random seeds. These results demonstrate that **GLSIM consistently outperforms all baseline methods, further validating its effectiveness and generalizability on state-of-the-art models**. Specifically, GLSIM surpasses the previous top-performing method, SVAR, by **+5.36%** on `Qwen2.5-VL-7B` and **+4.41%** on `InternVL3-8B`.\n\nWe will include these results in detail in the revised version of the paper.\n\n\n| Method   | InstructBLIP | Qwen2.5-VL-7B | InternVL3-8B | \n|-|-|-|-|\n| NLL            | 65.11 |59.13 | 55.70 |\n| Entropy        |65.63 |59.12 | 55.57 |\n| Internal Confidence | 81.89 |60.35 | 63.36 |\n| SVAR           | 78.35|70.82 | 68.83 |\n| Contextual Lens| 82.96 |65.16 | 65.26 |\n|**GLSIM (Ours)**   | **85.01** | **76.18** | **73.24** |\n\n---\n> A3. Clarification for Line 286\n\nThank you for your careful reading and for pointing out this typo and ambiguity. In Line 286, we intended to report the minimum and maximum AUROC improvements observed when comparing our local method ($s_{\\text{local}}$) to prior local baselines (Internal Confidence, Contextual Lens) to highlight the effectiveness of our proposed object grounding approach for object hallucination detection.\n\nSpecifically:\n\n- The minimum gain of **+1.1%** is observed when comparing $s_{\\text{local}}$ (Top-1) to Contextual Lens on the `LLaVA` model.\n- The maximum gain of **+7.7%** is observed when comparing $s_{\\text{local}}$ (Top-K) to Internal Confidence on the `Shikra` model.\n\nWe acknowledge this was a typo and will revise the main paper to clarify the specific baselines used for comparison and improve overall clarity. \n\nThank you again for catching this detail. We appreciate your close attention.\n\n---\n\n[1] Rohrbach et al., \"Object Hallucination in Image Captioning,\" EMNLP 2018\n\n[2] Honnibal et al., \"spaCy: Industrial-strength Natural Language Processing in Python\"\n\n[3] Bai et al., \"Qwen2.5-VL Technical Report,\" arxiv:2502.13923\n\n[4] Zhu et al., \"InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models,\" arxiv:2504.10479"}, {"review_id": "oCb7sIaXVw", "reviewer": "Reviewer_ZsgW", "summary": "This paper presents GLSIM, a new training-free framework for detecting object hallucinations in LVLMs. The method combines global similarity scores, which assess how well an object fits the overall scene, with local similarity scores that verify if specific image regions support the object's presence. The authors argue that this dual perspective improves hallucination detection accuracy. Experiments are conducted across multiple benchmark datasets and LVLMs, showing superior performance compared to existing methods.", "questions": "1. The authors claim that their method is more reliable than approaches relying on external knowledge sources. Could they provide a more detailed comparison with these external knowledge-based methods, particularly in terms of reliability and practicality?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed GLSIM framework is training-free, making it lightweight and potentially easier to adopt compared to methods that require additional training or fine-tuning.\n\nWeaknesses:\n1. The paper focuses solely on object hallucinations, while other types of hallucinations (e.g., attribute or relational hallucinations) are not addressed. This narrow focus reduces the applicability of the proposed solution.\n\n2. The combination of global and local similarity signals, while intuitive, does not represent a significant methodological advancement. \n\n3. The experiments rely on older models like LLaVA and Shikra. More recent and advanced models such as Qwen2.5 VL and InternVL2.5 have not been evaluated, which limits the relevance and impact of the findings.", "was_revised": true, "final_justification": "The author has addressed most of my concern, I'm happy to raise my socre.", "rebuttal": "We appreciate the reviewer for taking the time to provide detailed and constructive feedback. We are grateful for the recognition of our motivation, the performance of our method, and the training-free design.  \n\nWe also thank the reviewer for the insightful questions and are happy to clarify the points raised as follows:\n\n---\n> A1. Attribute and relational hallucinations\n\nWe appreciate you raising this important point. We agree that attribute-level and relational hallucinations are critical challenges for real-world applications. As mentioned in our Limitations and Future Work section, our current study focuses on object existence hallucinations because _publicly available token-level annotations and benchmarks for attribute and relational hallucinations in open-ended generation settings remain limited_.\n\nThat said, we explored whether GLSIM can handle these more complex hallucination types. To this end, we generated captions with each LVLM for 500 randomly selected images from the MSCOCO validation set. We then used a powerful multi-modal model `GPT-4o`, to produce pseudo ground-truth annotations for two new categories:\n\n- Attribute Hallucination: Incorrect properties of objects are described (e.g., “a red car” when the car is blue).\n- Relational Hallucination: The relationship between objects is incorrectly stated (e.g., “a cat sitting on a table” when it is under the table).\n\nWe then computed token-level GLSIM scores and aggregated them by averaging across attribute–object (e.g., “a red car”) and object–relation (e.g., “a cat sitting on a table”) spans. These aggregated scores served as unsupervised estimates of the hallucination likelihood.  **Without any task-specific modification, GLSIM consistently outperformed baseline methods.** We hope GLSIM can serve as a promising baseline for detecting more complex forms of hallucination.\n\n- Attribute Hallucination\n\n| Method | LLaVA-1.5-7B | LLaVA-1.5-13B | Qwen2.5-VL-7B\n|-|-|-|-|\n| NLL            | 58.62 | 60.50 | 56.89|\n| Entropy        | 52.21 | 55.32 | 55.84|\n| Internal Confidence | 74.24 | 73.67 |70.06 |\n| SVAR           | 67.03 | 68.62 | 71.09|\n| Contextual Lens| 74.02 | 75.48 | 71.98|\n|**GLSIM (Ours)**    | **77.19** | **78.07** |**74.09**|\n\n- Relational Hallucination\n\n| Method  | LLaVA-1.5-7B | LLaVA-1.5-13B | Qwen2.5-VL-7B\n|-|-|-|-|\n| NLL            | 57.06 | 57.35 | 54.90|\n| Entropy        | 55.72 | 56.27 | 55.03|\n| Internal Confidence | 69.38  | 68.94 |62.09 |\n| SVAR           | 61.20 | 65.83| 63.01|\n| Contextual Lens| 66.46 |69.85 | 64.88|\n|**GLSIM (Ours)**    | **70.03** | **73.64** |**68.95**|\n \n---\n> A2. Methodological advancement\n\nWe understand how your concern arises. While the idea of combining global and local signals may seem intuitive in hindsight, what distinguishes GLSIM is not only the formulation of these two signals but the _methodological rigor_ with which we analyze and instantiate them. Below, we elaborate on the methodological significance of our work:\n\n1. First, our motivation is original and offers a new lens through which to understand hallucination in LVLMs. Through detailed case studies in Section 4.1, we demonstrate, for the first time, that neither global nor local similarity alone is sufficient for reliable hallucination detection. This insight itself is a valuable contribution, as it challenges the prevailing reliance on single-perspective signals and motivates a more structured diagnostic approach. **The impact of our work, therefore, goes beyond a specific method—it points the field toward a promising direction for hallucination detection in multimodal models and opens up room for more future work**.\n\n\n2. Second, what sets our method apart is the **rigorous investigation of how each component, global and local similarity, should be designed and integrated optimally**. We do not treat the combination as a black-box fusion; instead, **we design, ablate, and validate every aspect of the formulation---this is both a new and non-trivial effort**. For example, in Section 5.3 (Lines 293–315), we carefully explore the design of both local and global scores by evaluating different unsupervised grounding methods, token positions, and aggregation strategies. Lines 316–330 examine how to best integrate the global and local scores. We further ablate distance metrics (Appendix D.5) and analyze all layer pair combinations to identify meaningful patterns for the scoring function (Appendix E). This is also recognized by other reviewers:\n\n    > \"_The paper includes in-depth ablation studies that analyze the impact of global and local similarity and different design choices for them_\".  Reviewer 5sQ6 \n    \n    > \"_The authors conduct thorough ablation studies on key design choices, including object grounding strategies and scoring variants, offering clear empirical justification for their final architecture (Section 5.3)._\" Reviewer S1xd\n\n3. Lastly, we validate our design through extensive quantitative (Section 5.3, Appendix D, and Appendix E) and qualitative (Lines 147-167, Lines 299-302, and Appendix B) ablations, showing that the proposed global and local metrics capture interpretable signals critical for robust object hallucination detection. These results reinforce the value of our approach, and we hope they address the reviewer’s concerns regarding methodological significance.\n\n\n---\n> A3. Evaluation with advanced models\n\n\nThank you for pointing this out. In response, we have conducted additional experiments using two recent and advanced LVLMs: `Qwen2.5-VL-7B` [1] and `InternVL3-8B` [2]. For additional evaluations on models such as `InstructBLIP`, `Cambrian-1`, and `LLaVA-NeXT`, please refer to **Appendix D.1**.\n\nThese results demonstrate that **GLSIM consistently outperforms all baseline methods, further validating its effectiveness and generalizability on state-of-the-art models**. Specifically, GLSIM improves over the previous top-performing method, SVAR, by **+5.36%** on `Qwen2.5-VL-7B` and **+4.41%** on `InternVL3-8B` in AUROC.\n\nWe will include these results in detail in the revised version of the paper. \n\n| Method  | Qwen2.5-VL-7B | InternVL3-8B | \n|-|-|-|\n| NLL            | 59.13 | 55.70 |\n| Entropy        | 59.12 | 55.57 |\n| Internal Confidence | 60.35 | 63.36 |\n| SVAR           | 70.82 | 68.83 |\n| Contextual Lens| 65.16 | 65.26 |\n|**GLSIM (Ours)**    | **76.18** | **73.24** |\n\n\n---\n> A4. Comparison with external model-based methods\n\n\nWe appreciate the reviewer’s request for a more detailed comparison. \n\nAs reviewers 5sQ6 and mtgR noted, our method offers practical advantages over approaches that rely on external knowledge sources, such as LLMs and LVLMs. These methods typically rely on prompting an external LLM or LVLM with a triplet input consisting of the image, instruction, and the generated caption. This approach is inherently restricted to **post-generation evaluation**, as hallucinations **can only be assessed after the entire caption has been produced by the base model**. In contrast, **GLSIM operates during the token decoding phase, enabling real-time hallucination detection at the token level**. This makes GLSIM particularly well-suited for interactive or streaming applications, where **immediate feedback** is crucial. Moreover, external LVLM-based approaches often require **multiple forward passes**—one to generate the caption from the base model under evaluation, and at least one or more to evaluate it using external LVLMs. GLSIM requires only **a single forward pass**, with no additional queries or models needed, making it much more efficient at inference time.\n\nRegarding reliability, external LVLMs may themselves be prone to hallucination. Our method avoids this dependency by operating entirely within the base LVLM, **without introducing additional uncertainty from external evaluators**. Below, we provide a quantitative comparison on the MSCOCO dataset using `LLaVA-1.5-7B` as the base model. The results show that GLSIM is significantly more efficient, requiring only 1.6 seconds per image for inference, compared to 9.3 seconds for the `External-based` method, which uses the more powerful `LLaVA-1.5-13B` model. This results in an **efficiency gain** of **+82.8%**. Despite this significant inference efficiency advantage, **GLSIM still performs competitively on reliability metrics**. Compared to `External-based`, GLSIM shows better performance across almost all evaluated metrics.\n\nThese findings support our claim that GLSIM is **practical and reliable** for real-world use. It achieves this without relying on any external models in a fully unsupervised, training-free manner, **using only internal signals from the base model itself**.\n\nWe will update the main paper to make this comparison clearer.\n\n| Method  | Time (s) &#8595; | Accuracy &#8593; | Precision (Real) &#8593; | Precision (Hallucination) &#8593; | Recall  &#8593; | F1 &#8593; |\n|-|-|-|- |-|-|-|\n| External-based      | 9.3 | 78.6 | 78.9 | 70.8 | 98.5 | 87.6 |\n|**GLSIM (Internal-based)**      | 1.6 | 81.8 | 83.1 | 71.8| 96.6 | 89.3|\n\n---\n\n[1] Bai et al., \"Qwen2.5-VL Technical Report,\" arxiv:2502.13923\n\n[2] Zhu et al., \"InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models,\" arxiv:2504.10479"}]

Author rebuttal:

Meta-review: The paper describes hallucination mitigation in VLM and investigates  a training-free object hallucination detection framework that uses uses global and local similarity signals between images and text. Overall positive feedback from reviewers, but some concerns regarding too many tuning parameters are raised.
Decision: Accept (poster)