Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
Authors: Weixing Wang
Zifeng Ding
Jindong Gu
RUI CAO
Christoph Meinel
Gerard de Melo
Haojin Yang
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Language Models
Hallucination
Primary area: deep_learning

Abstract: Large Vision-Language Models (LVLMs) with discrete image tokenizers unify multimodal representations by encoding visual inputs into a finite set of tokens. Despite their effectiveness, we find that these models still hallucinate non-existent objects. We hypothesize that one reason is due to visual priors induced during training: when certain image tokens frequently co-occur in the same spatial regions and represent shared objects, they become strongly associated with the verbalizations of those objects. As a result, the model may hallucinate by evoking visually absent tokens that often co-occur with present ones. To test this assumption, we construct a co-occurrence graph of image tokens using a segmentation dataset and employ a Graph Neural Network (GNN) with contrastive learning followed by a clustering method to group tokens that frequently co-occur in similar visual contexts. We find that hallucinations predominantly correspond to clusters whose tokens dominate the input, and more specifically, that the visually absent tokens in those clusters show much higher correlation with hallucinated objects compared to tokens present in the image. Based on this observation, we propose a hallucination mitigation method that suppresses the influence of visually absent tokens by modifying latent image embeddings during generation. Experiments show our method reduces hallucinations while preserving expressivity.

Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
Weixing Wang 1; Zifeng Ding 2 Jindong Gu 3 Rui Cao Gerard Melo', Christoph Meinel', Haojin Yang! Hasso Plattner Institute, University of Potsdam, University of Cambridge University of Oxford, *German University of Digital Science weixing wangOhpi.cd de, zd3200cam ac. uk, j jindong www.ttooooo..oo.oo com
Abstract
Large Vision-Language Models (LVLMs) with discrete image tokenizers unify multimodal representations by encoding visual inputs into finite set of tokens. Despite their effectiveness, find that these models still hallucinate non-existent objects. We hypothesize that this may be due to visual priors induced during train- ing: When certain image tokens frequently co-occur in the same spatial regions and represent shared objects, they become strongly associated with the verbaliza- tions of those objects. As result, the model may hallucinate by evoking visually absent tokens that often co-occur with present ones. To test this assumption, we construct a co-occurrence graph of image tokens using segmentation dataset and employ a Graph Neural Network (GNN) with contrastive learning followed by a clustering method to group tokens that frequently co-occur in similar visual contexts. We find that hallucinations predominantly correspond to clusters whose tokens dominate the input, and more specifically, that the visually absent tokens those clusters show much higher correlation with hallucinated objects compared to tokens present in the image. Based on this observation, we propose a halluci- nation mitigation method that suppresses the influence of visually absent tokens by modifying latent image embeddings during generation. Experiments show our method reduces hallucinations while preserving expressivity. Code is available https//ggihubb.com/wwixxgggg wwtpee/ngwW/CCCCCC/Dee////maii
Introduction
Large Vision-Language Models (LVLMs) have achieved remarkable progress in understanding and generating multimodal content. However, they are still prone to hallucination, often generating descriptions of non-existen objects. To address this issue, recent efforts have explored decoding time interventions such Contrastive Decoding 2 as well as techniques that leverage latent
feature representations to detect and mitigate hallucinations [ 6
promising direction in advancing LVLMs is the integration of discrete image tokenizers g 10 [3). which quantize continuous image features into sequences of discrete tokens using finite codebook. These tokens are structurally analogous text tokens, enabling unified token-based
processing across modalities within single transformer architecture [14]. Despite the potential of this approach, we find that LVLMs built with discrete image tokenizers remain susceptible to hallucinations, i.e., generating false or misleading content based on the input image and prompt, such
39th Conference on Neural Information Processing Systems (NeurIPS 2025).
Equual contribution
t Corresponding author

---

as referencing nonexistent objects or producing incoherent responses unrelated to the visual content. Furthermore, our experiments reveal that directly applying existing hallucination mitigation methods to these models may not always be effective, underscoring the need for more tailored investigation within this emerging paradigm.
We address this problem by analyzing the role of visual priors introduced by image token co- occurrence. Prior work [15 [6] has shown that object co-occurrence can create language priors in LVLMs, where models hallucinate objects not present in the current visual or textual input but frequently co-occurring with seen objects in the training text. We extend this insight the visual modality by investigating how discrete image tokens, much like text tokens, may encode similar co-occurrence patterns. Since image tokenizers are trained on large-scale image datasets, their learned embeddings inevitably capture co-occurrence relationships among visual patterns, implicitly introducing visual priors. We hypothesize that these learned associations among image tokens can drive hallucinations in LVLMs.
To test our hypothesis, we first introduce Condext-Guided Clustering (CGC) capture the visual priors encoded in discrete image tokens. CGC constructs a co-occurrence graph using a corpus of segmented images, where each node represents unique token from the image token vocabulary. The edges in this graph encode contextually-awan co-occurrence patterns, with edge weights determined by two factors: the spatial proximity of image tokens within the quantized image representation and their semantic association, based on whether they are used to represent the same segmented object. Graph Neural Network (GNN) is trained on this graph using a contrastive learning objective, producing graph-based token embeddings that reflect these semantic and spatial co-occurrence patterns. These embeddings are then used to cluster image tokens using K-means [17 to form groups of tokens that frequently co-occur in localized visual contexts or correspond to related visual concepts. This process yields a structured representation of the visual priors embedded in the image token vocabulary. Using the learned clusters, we analyze how hallucinations in LVLMs are related to token co-occurrence patterns. Our key finding is that hallucinated objects are often highly related to image tokens that are absent from the current visual input but belong to the top few dominant clusters (those most frequently represented in the image). These absent tokens co-occur frequently with the present ones, and their strong association may lead the LVLM to mistakenly reference them, resulting in hallucinations.
Motivated by this observation, we propose Visual Token Decontamination (VTD), latent-space intervention that reduces the influence of the visually absent tokens identified above during au- toregressive decoding. VTD projects these tokens into the model's latent space and subtracts their contributions from intermediate hidden representations, thereby mitigating their hallucinative effect.
The contribution of our work is summarized as follows.
We identify and quantify a novel source of hallucination in LVLMs with discrete image tokenizers, showing that co-occurrence patterns among image tokens introduce visual priors that can mislead the model. To our knowledge, we are the first to uncover this phenomenon.
We propose CGC and VTD as a two-step framework that captures visual priors and mitigates hallucinations by identifying co-occurrence-drive token patterns and suppressing their influence through targeted latent space editing.
Extensive experiments demonstrate that our method can effectively reduce LVLM hallucina- tions while maintaining strong efficiency.
2
Related Work
LVLMs with Discrete Image Tokenizer. Recent advancements in LVLMs have increasingly adopted discrete image tokenization a strategy to unify multiple modalities within a shared representation space. Chameleon (7) was among the first to integrate this paradigm, enabling the model to interpret and generate images and text in arbitrary sequences. This paradigm has since evolved, with Emu3 extending the approach by enlarging the image vocabulary and incorporating video as an additional modality combine continuous and discrete image encoders for greater representational flexibility [ 13). Show- also advances this line of work by integrating discrete image tokens into unified framework that supports autoregressive text generation alongside iterative masked token prediction for images ([].
[8]. Janus and Janus-Pro further introduced hybrid architectures that

---

While these works demonstrate strong multimodal capabilities, they also reveal new challenge: LVLMs with discrete image tokenizers remain susceptible hallucimations, How effectively mitigate hallucinations in this new class of models remains underexplored question.
Hallucination Mitigation Methods for LVLMs. As LVLMs continue to advance, significant re- search has been devoted to understanding and mitigating their tendency to hallucinate, particularly in the form object hallucination generating descriptions of objects that are not present in the visual imput [18 19 [5]. A major line of work attributes hallucination to biases in the language modeling component. For example, OPERA A [3] identifies problematic decoding patterns and applies penalty and retrospection strategies, while LURE [5] explicitly models object co-occurrence, uncertainty, and positional factors through revision- algorithms. Though effective, these methods rely on iterative identify-and-revve procedures and require multiple decoding passes, leading to high compu- lational overhead. Another group of methods focuses on hallucination arising from the association between visual and textual modalities. Contrastive decoding (CD) techniques (20 3 4 mitigate hallu-
cinations by perturbing the visual input and contrasting the resulting token logits. VCD [ hallucinations via input noise and contrasts them against normal outputs, while SID improves efficiency and effectiveness by adaptively focusing on part of visual input. PROJECTAWAY [6] offers post-hoc alternative by associating hallucinations with visual embeddings and subtracting possible hallucinative text tokens from intermediate LVLM layer output. These approaches have shown strong results on earlier models like LLaVA 21,, but their effectiveness on newer LVLMS with discrete image tokenizers remains unproven. In our experiments, CD>bbaeed methods often perform worse in models with unified visual and text token spaces, likely due to incompatibility with discrete token representations. Compared to these directions, visual prior-driven hallucination mitigation remains largely underexplored. OHD 22 investigates hallucinations caused by CLIP encoder biases, but requires full model retraining with dense supervision, which is computationally expensive. In contrast, our approach focuses specifically on the visual priors embedded in discrete image tokenizers. We introduce lightweight GNN-base framework to capture image token co-occurennce patterns and identify vallucination-pron tokens before generation, enabling efficient and effective mitigation without retraining the LVLM.
amplifies
bench
Layer
Transformer Layer
Ar((ra)
C₂ n Q
Trenstormer Layer
Ar(ar)
quantization
E
LM
The image shows person sitting on the ground with their knees drawn up and their hands resting on their knees.
Please desorbe this image"
Clustering
Vieual Token Decontsmination
Figure 1: Overview of our proposed method, which consists of two key components: Context-Guided Clustering (CGC) and Visual Token Decontamination (VTD). CGC organizes image tokens into clusters according their co-occurrence patterns. VTD identifies potentially hallucinative tokens based on visual context and suppresses their influence during autoregressive generation. During inference, only VTD is applied while CGC is performed only once for every model offline.
3 Method
propose a two-step approach mitigate hallucinations LVLMMs with discrete image tokenizers. First. Context-Guided Clustering (CGC) captures visual priors by constructing co-occurrence

---

graph of image tokens and clustering them via graph-baser embeddings (Sec. 3.1 We show that hallucinations often correlate closely with the visually absent image tokens in the first few dominant clusters. (Sec. (.2). To address this, Visual Token Decontamination (VTD) reduces their influence by modifying latent representations during autoregressive decoding (Sec. 3.3). See Figure I for an overview.
Paradigm of LVLMs with Discrete Image Tokenizer. We begin by defining the paradigm of LVLMs with discrete image tokenizers follows. An input image is first encoded into sequence of discrete image tokens v = {u;} S V using an image tokenizer, typically based on a Vector Quantized- Variational AutoEncoder (VQ- VAE) [3] VQGAN (24], where the embedding of each image token corresponds to an entry in learned codebook of size V Similarly, the text input is tokenized into sequl
sequence of text tokens 1. = {44} S T, where IT] represents the size of the text vocabulary. For visual understanding task, the final input to the model is usually given by the concatenated image and {u, During inference, this token sequence is processed by stack of T. transformer layers [25] Given 2;, the i-th input token, we denote its output representation from the 1-th transformer layer as g (a:)) E IRd, where is LVLM's hidden dimensionality. The probability distribution over the next token y computed P(y|z) softmax(g"(x-1) Wout), where x_1 the last token in the input and Wout [ is the output projection matrix.
text token sequences
3.1 Model Visual Priors via Context-Guided Clustering
Capturing Visual Priors via Clustering. In LVLM with a discrete image tokenizer, each image token is mapped to an entry ID from a learned codebook of finite size. To maintain high compression efficiency (24), the codebook is typically limited in size (e.g., 8,192 entries in Chameleon (7). As a result, each image token is reused to represent a wide range of visual concepts. Moreover, certain groups of image tokens frequently appear together across different images or image regions to represent common objects or scenes, depending on the visual con- tent. These recurring patterns of co-occurrence among tokens form what refer as visual priors, which quantify underlining distribution bias from the image data that is used for training. To capture visual priors, we cluster image tokens based on their co-occurrence patterns. We per- form clustering based on codebook embeddings rather than directly on image token embeddings, motivated by three key observations. First, as discussed earlier, each image token in an LVLM directly mapped from a unique codebook en- try ID. This means that clustering through code- book entries (each entry a codebook is an embedding) is equivalent tokens. For simplicity, we henceforth refer to the co-occurrence and clustering of image to- kens, with the understanding that these oper- ations potentially performed via their cor- responding codebook entries. Second, visual priors emerge when certain groups of image to- kens frequently co-occur across different images visual regions represent common objects or scenes. Clustering codebook entries that tend co-occur in such contexts allows us expose these priors. This enables us to use the image tokens present in given input to identify other tokens absent from the image but belonging to the same clusters- that frequently co-occur with the observed ones and may influence the model's behavior due to learned visual priors.
Step1: Graph Construction
Step2: Leaning
Co eccurrence Graph
Leunsrrat Lryy
clustering image
C.
Ce
Codebook
Step3: Clustering
Figure 2: Illustration of the CGC pipeline. Code- book entries are first used to help construct a co- occurrence graph over image tokens, followed by a learning process to obtain graph-based embed- dings, which are then used to cluster image tokens.
Third, codebook entries typically have much lower dimensionality than the image token embeddings used within an LVLM,, making clustering over them more efficient and lightweight than clustering directly via image token embeddings. A straightforward approach might involve applying standard

---

clustering algorithms such as K-means [77 directly to the raw codebook embeddings. However, such methods primarily rely on vector similarity in the embedding space and do not account for how tokens are actually used together in visual inputs. As result, they often fail to reveal meaningful visual priors and can produce misleading clusters that obscure important co-oceurrence structures. This limitation calls for a dedicated method specifically designed to cluster tokens based on their contextual co-occurrence, in order to accurately capture visual priors.
Context-Guide Guided Clustering. To accurately capture visual priors, we propose CGC, illustrated in Figure 2 We begin by collecting images from COCO 2017 Panoptic validation set [26], segmentation dataset suitable for identifying object-level regions. 2 Using the model's discrete image tokenizer, we extract tokenized image representations for each image. Based on these representations, CGC constructs an undirected graph G (N, E), where each node E N corresponds to an image token, and each edge e E & connects a pair of tokens that frequently co-occur across the dataset. CGC first computes the co-occurrence strength between each pair of image tokens based on two complementary contexts: (1) spatial proximity, where tokens that appear within the same local region of the quantized image representation (e.g., within a 3 x 3 grid cell in our implementation) lead to greater strength, and (2) semantic coherence, where tokens located within the same object segmentation mask are given greater strength. These two signals together capture both fine- ane-grained spatial context and broader semantic relationships. To retain only the most informative connections, CGC constructs the graph G by linking the top 10% of token pairs based on co-occurrence strength, with edge weights 8ij given by their normalized values.
After constructing 9, we employ GNN to learn node representations. We initialize node representa- tions by projecting codebook embeddings Z E R V Xxdestookk to latent dimension: H))) ZW proj+
where W proj E IR doootta X xdoxs, Inspired by Brody et al. [27. we define the computation of the l'tth GNN layer as an attentional aggregation function
( (E)ENo(on))(..)) Wonng-")
h (()
b and by (1-1) denote the node representation of node n; and n, from the 1-tth and (I' - 1)-th layer, respectively. Nei(n.) is the neighborhood of node n4. Wil") E Rddoxxdow is weight matrix and a.)) is an activation function. The attentional coefficient 0i incorporating their edge weight stj. Due to space constraints, we provide the detailed computation in Appendix C.2. After L' layers of GNN aggregation, we train node representations on a contrastive
(L') between n, and ny is computed by
learning objective inspired by InfoNCE 28
Eeiec 81 exp(sim(h, (h; (4) ,,
Coontrast
log
(2)
emm Rm. EN exp(sim(hl)) (hm h ///))
sim(-) denotes cosine similarity and is temperature parameter. For each batch of nodes, connected pairs are treated as positives and all others as negatives. The contrastive objective encourages positive pairs be pulled closer, i.e., more tightly clustered, while pushing apart negative pairs. By weighting positive pairs according to their co-occurrence strength, the model learns to reflect visual priors in clustering, drawing together tokens that frequently co-occur in similar visual contexts. To improve clustering quality, we add positive pair similarity loss that enforces minimum similarity between connected nodes, scaled proportionally to their edge weight 8:j. Our core intuition that while Leomtrat uses co-occurrence weights for relative optimization, Lyps enforces absolute similarity thresholds that ensure adequate separation between positive and negative pairs regardless of batch composition. We provide additional analysis in Appendix C.3 regarding the importance of this extra loss term
Cpps Eesye 8ij max 8ig -iimm))))) (a(h)) 10.5 (())))
(E)
Σe, 1B3 s.j > sim(h (17) (1))))
*COCO 2017 Panoptic provides lineggrinedd segmentation with 133 diverse segment labels (i.c., objects). The segment labels covers both "things" (countable objects like people, cars) and "stuff" (uncountable regions like sky, grass), which includes the objects considered in LVLM hallucination detection benchmarks. Capturing visual priors from this dataset will not introduce loss of generality in context.
5

---

3 is a scaling factor, and the complete training objective thus becomes L = Leontrast + Lpps. With trained node representations, we then apply K-means to the nodes g to achieve codebook token clustering. We present further details of CGC in Appendix
3.2 Visual Priors Induce Hallucinations
We assume that the visual priors captured in Sec. 3.1 may induce hallucination. To validate this, we conduct the following analyses. We first sample 1,000 images from the AMBER benchmark (2) and tokenize them using the discrete image tokenizer from Janus-Pro-7B (Results on other models are presented in Appendix (C.)). For each tokenized image, we record the set of image tokens present and determine their corresponding clusters using CGC. For each image, we assign its image tokens to their respective clusters and count the frequency of each cluster. The clusters with the highest token counts are defined as the dominant clusters, representing the most prevalent visual elements in the image. From the most dominant cluster, we divide the tokens into groups: C1, consisting of tokens that appear in the image (present tokens), and C2. consisting of those that do not (absent tokens). The remaining tokens in the image that do not belong the most dominant cluster form third group, denoted as C3. We use Janus-Pro-77 to generate a detailed description for each image and identify hallucinated objects using the ground-truth annotations provided by AMBER.
To quantify the association between a token group and an object, we count how often tokens from that group appear within the object's segmentation masks. The segmentation masks are obtained from the COCO 2017 Panoptic dataset, which we also use for CGC. To evaluate how strongly each group is linked to hallucinated content, we introduce a metric: HitRate@K K Eoby [(bbj E (op-K(C))/HHbbb)) H. obj' denotes a hallucinated object, [(obj') ] the number of all hallucinated objects, and top-K(C') is the set of top-K objects most frequently associated with the token group C. 1()) denotes the indicator function. This metric reflects the proportion hallucinated objects that fall within the most strongly associated object categories for given token group.
9
HitRate 0%6)
HilBate (S6)
C2- CGC
Kmeans
10
10
10
Cluster Ranking
Figure 3: Left: Hallucination HitRate for K from 1-10. Middle: identified by our CGC method versus naive K-means. Right: HitRate@5 of C2 tokens from the ten most dominant clusters.
Comparison between C2 tokens
We plot HitRate for K to 10 for the three token groups C1, C2, and C3 in Figure 3 (left). Among them, C2 Coktos that do not appear in the image but belong to the most dominant clus- ter ter-consistently achieves significantly higher HitRate values (5-10% higher) than both Ci (present tokens from the dominant cluster) and C3 (tokens from other clusters). This indicates that absent to- kens from dominant visual clusters assess the effectiveness of CGC, we compare the HitRate @ K for C2 against a baseline using solely K-means for clustering. As shown in Figure (middle), CGC consistently outperforms K-means, indicating its superiority in capturing the link between visual priors and hallucination. While these results highlight the impact of the most dominant cluster, it remains unclear whether less dominant clusters also contribute significantly to hallucination. To explore this, we plot the HitRate @5 5 for the top 10 dominant clusters in Figure 3 (right). We observe sharp decline in HitRate as cluster dominance decreases, suggesting that hallucinations are primarily associated with the most dominant clusters. In summary, our analysis shows that hallucinations are closely linked to visual priors that can evoke absent tokens from dominant clusters and CGC provides an effective means of uncovering these associations. We provide the same analysis on other LVLMS in Appendix C.4 to further confirm our conclusion.
more strongly associated with hallucinated objects. To further

---

3.3 Mitgate Hallucination via Visual Token Decontamination
Motivated by findings, we propose VTD, hallucination mitigation method designed to address hallucinations induced by visual priors in LVLMs with discrete image tokenizers. Given tokenized input image, VTD first identifies the dominant clusters and then locates tokens from these clusters that are absent from the image. For each such token Vaas. VTD performs a targeted modification of the intermediate layer output during autoregressive generation. Inspired by PROJECTAWAY [6. VTD achieves decontamination by subtracting projection of the embedding (Tabs) from the hidden representations of all image tokens present in the input:
g (v:) = (vx) -y. (vx) (V0) (cCas)))
(4)
where i2 denotes an image token. The layer index and the magnitude coefficient y serve as hyperparameters controlling the editing position and intensity of decontamination. For each LVLM, we employ different editing layer and magnitude coefficient. We also select different numbers of dominant clusters for hallucination mitigation. See Appendix B.1 for detailed discussion selecting suitable values for these hyperparameters.
Experiments
In Section 4.2 we begin by evaluating CGC+VTD against recent hallucination mitigation baselines across three benchmarks assess its effectiveness. We then explore its compatibility by combining it with other methods and analyzing the resulting performance. In Section 4.3) we assess the efficiency of method through runtime and memory comparisons with existing baselines, and perform ablation studies to validate our design choices. Due to space constraints, additional experiments presented in the Appendix, including a comparative study the Polling-based Object Probing Evaluation [19 benchmark (Appendix D.3p and several representative cases as qualitative results (Appendix D.3
4.1 Experimental Settings
Benchmarks and Evaluation Metrics. We evaluate on three popular benchmarks: (1) AM- BER (29), which assesses object existence, attribute, and relation hallucination across both generative and discriminative tasks; (2) Object HalBench (30], which uses large language model (LsMs-assisted evaluation 300 carefully designed visual understanding questions; and (3) MME [1I. widely adopted benchmark for evaluating LVLMs' general perception and cognition abilities across 14 subtasks. We use the first two datasets evaluate hallucination mitigation performance, and the last to assess whether the methods affect the model's general capabilities. Additional dataset details are provided in Appendix A We employ the evaluation metrics coupled with benchmarks and the metric details are presented in Appendix A.1.1 A.2.1 and A.3.1
LVLMs & Hallucination Mitigation Baselines. We evaluate our method on three state-of-the-ar LVLMS that utilize discrete image tokenizers: Chameleon-7B (7). Emu3-13B [8] and Janus-Pro- 7B [13] For baselines, we include both naive and dedicated hallucination mitigation methods. We begin with Nucleus Sampling (top-/' with P = 0.9). a standard decoding strategy not designed to reduce hallucinations, but used here to illustrate the default hallucination level of LVLMs with discrete image tokenizers. In addition, we evaluate several recent hallucination mitigation methods, including contrastive decoding approaches VCD and SID 2) the latent editing method PROJECTAWAY (6), and OPERA [5], which focuses on biases in language modeling. Finally, we compare against an additional baseline, Cluster-Base which performs clustering using only K-means without CGC. For VCD, SID and OPERA, we use their default hyperparameters. For PROJECTAWAY and our method, we conduct hyperparameter search on an AMBER subset to identify optimal configurations. We let Cluster-Bat use the same configuration as method. See Appendix B for implementation details.
7

---

Table I: Experimental results (average of three runs with error bar in parenthesis) on benchmarks. Best/second best results are marked bold/underlined Metries with 1/4 means the higher/lower the better. Disc. means discrimination task.
Model
Method
AMBER Generative Task
AMBER Disc. Task
Object HalBench
MME
CHAIR
Covert
Hall
Cogl
Acc.
CHAIR-s]
CHAIR4]
Nucleus Sampling
19.34
49.43p.m
6.78
43.8f1g270)
16.15
26.011.00
873.04
Janus-Prs Prou-PPo-BB Chameleon-77
VCD
SID
20.74
52.61 19.9
51.36gg cm
11.41.99
a2.00gza
6.G7 a.50
7:48:1480
43,68
44.10(2.x2)
44.32 49.50
49.53
06.79
26.18
26,66
975.60
965.13
PROJECTAWAY
$5.69/2541 14.71
01,38
5,86
46,69 54)
43.
(6.07(100
947 9030
OPERA
a5.6l
53...1 kg
58.6 18.6..10
3.2 199
15.14
19.5
39
25.26 90
928 53
Claster Base
18,65
68.95
63.94 o0g
( p.52
47, 47.34
43. 00
26.42
505.00 505.6
CGC-VTD
03.8
52. 75
p..49
300
46,83
(0.02
38.5 36.58
23.69
969, p530)
Nucleus Sampling
4.56
58.3 58.31 600
23, 33:
84.24 p.18
(90))
2,00
19. 9.393
10.0%
VCD
a.47
64.02
09.64g
3.
85.50
87.97
19. 00.00/1.0
9. 9.68 on
1279 20
SID
6.36g
02 250
11.1
2 (.29
85.4 2..4
68.32
18.53
10.95
1335.87 (5..7)
PROJECTAWAY
4.54
59.26 30
g0.0
1.05
83.73 oz))
13.00 2,0
6.73
1790.89 (x33 x.31)
OPERA
4.32
69.33
15.8
1.0p.31
$5.10
12.30 4)
6,93
1777. 11:
Claster-Bas Base
6.00
57 f1 mm
22.82
1.20
84.73
87.28-2.00
17 67 (9.9))
1735. (.3.01)
CGC.VTD
59.04gu mg)
17.50..43
0.96 9.06)
87.58 10)
90.43 (912
5.81 ts 350
181393 (427
Nucicus Sampling
65.85 (0.32;.43)
60. as:
1.5
70.93
84.78
23. (7.377
13.85
1636.64,033
Kinact- 1318
VCD
66.45 (0.52
(1.43
00.09
5 (.20
63.96(1.41) (8.95
13.81
1520, 18
SID
66.78 70.78 m0
64.45
5.88
20.30
15.34
23.33
12.75
PROJECTAWAY
63 63.45 9.57
67.12
5.90
g0.8%
82. (2.16:2433
10.07
11.00 om
1625 1025.07 19)
OPERA
66.9 66.91, 8.93
1 a.20
$1.75
56.65
18. (..73
11.21
1611.51,50
Clasier CGC-VTD
10. (0.12
65, 2:
60.64
1.77
20.20
(5.00
12.96
1578 62g
9.01
68.85
16.8%
4.25 (.20)
82.86
87.73
17.25 (2.25
(0.33
1635.71 /6.5.71
Table 2: Results of combining our method with baselines (model name with +). Numbers in parentheses indicate performance difference. Green denotes improvement and Red indicates drop, relative to baseline results in Table We provide combination details in Appendix B.2
Metric
VCD+
SID+
PROJECTAWAY+
OPERA+
CHAIR-s J 45.10 (+0.78)
42.90 (-3.84)
39.02 (-4.64)
(-6.06)
CHAIR-i 31.15 (+4.49)
(-1.8))
22.83 (-3.65)
(2.70
4.2
Experimental Results
We present our experimental results in Table D and highlight several key findings. (1) Without dedicated hallucination mitigationn, as in the case of Nucleus Sampling, LVLMs with discrete image tokenizers exhibit a high tendency to hallucinate across all benchmarks. (2) CGC+VTD achieves the best performance in 22 out of 27 metrics (top-2 in 25), demonstrating strong hallucination reduction while preserving generation quality. On the AMBER benchmark, our method significantly reduces hallucinations in both generative and discriminative tasks-for example, Chameleon, it improves CHAIR by 5.46 points and accuracy by 2.98 points compared to Nucleus Sampling. On Object HalBench, it achieves the best performance metrics; for Janus- Manus-Pro-7Pe Pro CHAIR-S drops by 8.9 points (from 19.33 to 10.43) and CHAIR-i by 4.87 points (from 10.68 5.81). Additionally, our method maintains or enhances general perception capabilities, outperforming all baselines on MME with Janus-Pro-7B and that hallucination reduction does not come the expense of model's perception and cognition abilities. (3) Cluster-Bass shows no consistent improvement, reinforcing the necessity of CGC for capturing meaningful visual priors. Unlike CGC, K-means clusters tokens purely based on embedding similarity, which does not reflect co-occurrence patterns, and thus fails to model visual priors effectively. (4) CD methods, VCD and SID, tend underperform across tasks-for example, they consistently yield higher CHAIR scores than even Nucleus Sampling on generative tasks for all evaluated LVLMs indiicatini their limitations when applied to models with discrete image tokenizers. We provide a more detailed analysis on this point in Appendix D.4
Main Results.
both sentence-level and object-level
Combination with Existing Methods. While our method shows strong performance in reducing hallucinations across multiple benchmarks, it specifically targets visual priors introduced by discrete image tokenization. Given that hallucinations in LVLMs can arise from various sources, we investigate whether our approach can be effectively combined with existing methods that address other causes. To this end, we combine our method with several baselines, each with a tailored strategy. The implementation details can be found in Appendix B.2 We report the results for Chameleon-7B on Object HalBench in Table[2 Overall, our approach enhances the performance of prior methods in most cases. We highlight several observations. (I) OPERA achieves substantial gains when combined
8

---

with our method, even outperforming CGC+VTD alone (as reported in Table This is likely because the two methods target different modalities- ours mitigates hallucinations via the visual pathway, while OPERA focuses on the language modality. As a result, hallucinations can be addressed from both perspectives without mutual interference. (2) We also observe improvements when combining with SID and PROJECTAWAY. both of which aim to reduce hallucination based on vision-and-text association. This suggests that addressing visual priors via our method complements their strategies, leading to further reductions in hallucination. (3) In contrast, combining not yield additional benefits. We conjecture that this is because VCD applies contrastive decoding uniformly across all image tokens by corrupting large portions of the input image with noise, without considering each token't 's relevance to generation. This broad perturbation may interfere with the targeted latent space suppression in CGC+VTD, resulting in unstable or conflicting effects.
method with VCD does
4.3 Further Analysis
Table 3: CGC+VTD achieves the best efficiency on Time and Memory while maintaining a reasonable generation length.
Metric
VCD
SID
PROJECTAWAY
OPERA
CGC+VTD
Time (x)
359
245
470
941
188
Memory (MB)
17,228
15,119
14,482
24,767
14,478
#oken
282
269
185
173
180
Efficiency Analysis. To gain more comprehensive understanding of our method relative to baselines, we conduct an efficiency analysis. Table [ compares computational efficiency across methods based on average generation time and GPU memory usage (evaluated on AMBER generative and Object HalBench tasks with Chameleon-7B) While OPERA A and PROJECTAWAY demonstrate strong performance, our observations show that they are significantly less efficient. Our method is the fastest-5x faster than OPERA and 2.5x faster than PROJECTAWAY and consumes the least GPU memory. We also find that CD-based methods tend to generate longer responses, which may increase object coverage but also lead to more hallucinations, as reflected by higher CHAIR and Cover scores in Table T Overall, our method delivers top performance with minimal computational overhead, making it both efficient and effective. We provide analysis on other LVLMs in Appendix D.2
Ablation Study. To validate the contribution of different design choices, we conduct a series of ablation studies. Specifically, we test three variants: removing semantic coherence in co-occurrence computation (w.o. Semantic), removing spatial proximity (w.o. Spatial). and removing the loss used to enforee similarity among positive pairs (w.o. Epps). As shown in Table incorporating both spatial and semantic signals consistently yields the best performance across all models, highlighting the importance of capturing both spatial and semantic context for modeling visual priors. Additionally, the positive pair similarity loss proves crucial for enhancing the clustering of co-occurring tokens, leading to more effective hallucination mitigation.
Table 4: Ablation study results.
Chameleon-7B
Panus-Pro-BB
Em133-13
Variant
CHAIR-s
CHAIR-i 1
CHAIR-s
CHAIR-i ]
CHAIR-s
CHAIR-i
W.o. Semantic
42.15
23.84
14.38
7.91
19.73
10.62
w.o. Spotial
43.67
14.95
8.95
19.01
11.13
w.o. Eyps
41.74
25.15
13.33
8.03
18.47
11.01
CGC+VTD
38.58
23,69
10.43
5.81
17.25
9.24
5
Conclusion
In this work, we identify visual priors as previously underexplored source of hallucination in LVLMs with discrete image tokenizers, and propose a two-step mitigation approach. Context- Guided Clustering captures visual priors by modeling co-occurrence patterns among image tokens, while Visual Token Decontamination suppresses potentially hallucinative tokens during generation. Compared to recent state-of-the aht hallucination mitigation methods, our approach outperforms them

---

across multiple benchmarks on three widely used sescrete-token based LVLMs. Furthermore, our method is computationally efficient and complementary to existing techniques, enabling additional gains when combined.
6 Limitations & Future Directions
Despite the promising results, observe slight drop in coverage scores (Cover 1 in Table D with our proposed method, suggesting a trade-off between reducing hallucinations and maintaining comprehensive object recognition. This indicates that future work should aim to balance hallucination mitigation with visual coverage preservation. Another limitation is that our method is specifically designed for LVLMs with discrete image tokenizers and is not directly applicable to earlier models such as LLaVA 221, which rely on continuous visual features. Extending our approach to bridge this gap and generalize across different visual encoding paradigms presents an important direction for future research.
7 Acknowledgement
Zifeng Ding receives funding from the European Research Council (ERC) under the European Union's Horizon 2020 Research and Innovation programme grant AVeriTeC (Grant agreement No. 865958).
The project which this report is based was funded by the Federal Ministry of Research, Tech- nology and Space under the funding code "Kl-Servicezentrum Berlin- Beetin--ranenbbrg"" 161S22092. Responsibility for the content of this publication remains with the author.
References
[1] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vasion-language models through visual contrastive decoding. arXiv preprint 0rXiv:2311.16921 2023.
[2] Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decoding: Alleviating hallucinations for large vision- Saiiun-a models, 2024.
[3] Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. Code: Contrasting self-generated description to combat hallucination in large multi-modal models, 2024.
[4] Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Yc, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding, 2024.
[5] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-alless
2024.
[6] Jiang Nick, Kachinthaya Anish, Petryk Suzie, and Gandelsman Yossi. vision-language representations to mitigate hallucinations. arXiv preprint arXiv:2410.02762. 2025.
Interpreting and editing
[7] ChameleonTeam, Chameleon: Mixed-modas early-fusion foundation models, 2024.
[8] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang. Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need, 2024.
[9] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024.
10

---

[10] Jian Jia, Jingtong Gao, Ben Xue, Junhao Wang, Qingpeng Cai, Quan Chen, Xiangyu Zhao, Peng Jiang, and Kun Gai. From principles applications: A comprehensive survey of discrete tokenizers in generation, comprehension, recommendation, and information retrieval. arXiv preprint arXiv:2502. 12448, 2025.
[11] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin. Yuchao Gu. Zhijie Chen. Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation, 2024.
[12] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang. Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge 7hang, Linyang L.,, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal IIm with discrete sequence modeling, 2024.
[13] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025.
[14] Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, et al. Next token prediction towards multimodal intelligence: A comprehensive survey. arXiv preprint arXiv:2412.1800 2024.
[15] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang. Zhun Deng. Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models, 2024.
[16] Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-cnn, 2020.
[17] J. MacQueen. Some methods for classification and analysis of multivariate observations. 1967.
[18] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning, 2019.
[19] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023.
[20] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization, 2023.
[21] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024.
[22] Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. Investigating and mitigating object hallucinations in pretrained vision-language (clip) models, 2024.
[23] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018
[24] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2021.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2023.
[26] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
[27] Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks?, 2022.
[28] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.

---

[29] Junyang Wang. Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang. Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang. Amber: An llm-free multi-dimensional benchmark for milms hallucination evaluation, 2024.
[30] Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness, 2024.
[31] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, 2024.
[32] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng. Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grat tne-grained correctional human feedback, 2024.
[33] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308.0633 2024.
[34] Teng Hu, Jiangning Zhang, Ran Yi, Jieyu Weng, Yabiao Wang, Xianfang Zeng, Zhucun Xue, and Lizhuang Ma. Improving autoregressive visual generation with cluster-oriented token prediction, 2025.
[35] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the ofrtteeart natural language processing, 2020.
[36] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hack- able transformer modelling library. htttp:://gittubb.cmoffaccbbbbll 2022.
[37] Sudong Wang, Yunjian Zhang, Yao Zhu, Jianing Li, Zizhe Wang. Yanwei Liu, and Xiangyang Ji. Towards understanding how knowledge evolves in large vision-language models, 2025.
[38] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023.
[39] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[40] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. 2019.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Sec. 3 for the methods regarding visual prior modeling, hallucination detection and mitigation. Sec. 4 for experiments about hallucination mitigation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations in Sec. 6. We also provides efficiency analysis in Sec. 4.3, comparing our proposed method with previous methods.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification:
We provide systematic statistical analysis in Sec. 3.2 to prove our assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide details of methods in Sec. 3 and experimental results in Sec. 4. We provide details of benchamrk datasets used in our work in Appendix A. We provide implementation details in Appendix B, including hyperparameter tuning and experimental configurations of all considered methods our work. We provide more details of Context766 Ceaittrring in Appendix C. We also report error bar of experiments in our main result table (Table 1).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
14

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The benchmark datasets used in this work are already open-sourced. We provide the code implementation of our work as Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
geweddeii///////e/eeeeeeessssse/ CodeSubmiee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide corresponding information in Appendix A, B and C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the main results with enror bars (standard deviation) calculated from runs with different random seeds in Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide Table 3, 14 and 15 providing statistics of efficiency. We also discuss computational resource consumption in Section 4.3, Appendix C.2.3 and D.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics pt http:///wwurrr...copoopp.ccccaaa
Answer: [Yes]
Justification: We fully confirm the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work targets on the ability of Large Vision Language Models, which is not subject to one specific application.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
As decoding method for LVLM, our work does not have such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All contents are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All code will be released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We haven't conducted crowdsourcing.
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are only used assist writing our work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ //pppr////eurp....m http///ripssccc/oofeeeeeeeeeee cc/Confferecces/20555 for what should or should be described.
A Benchmark Details
A.1 AMBER
AMBER consists of 1,004 high-quality images collected from MS-COCO 2014 test [26 and Unsplash The dataset includes 337 distinct object classes (covering 14 major object categories: Nature, Architecture, Street View, Vehicles, People, Animals, Furniture, Electronic Devices, Tools, Stationery & Toys, Kitchen Utensils & Cutlery, Fruits & Vegetables, Food, and Sports), representing a 4x increase compared to existing benchmarks like COCO (80 objects). It also contains 350 attribute classes for evaluating hallucination beyond obejet level. The benchmark contains two main task types with total of 15,220 samples:
Generative task: 1,004 samples using prompt: "Describe this image."
Discriminative task: 14,216 samples across three subtypes:
Existence: 4,924 samples using prompt: "Is there (object) in this image?"
Attribute: 7,628 samples using prompt: "Is the (object) (state) in this image?"
Relation: 1,664 samples using prompt: "Is there direct contact between [objectl) and [object2)?"
A.1.1 Evaluation Metrics
Generative Task Metrics:
CHAIR (Challenging Hallucinations Assessment for Image Captioning):
CHAIR(R) len R O Aorj)
len (
measures the frequency of hallucinatory objects in responses, where Rob is the set of final objects extracted from the response and Aobi is the annotated objects list.
Cover (Coverage):
len( R2 0 Aoby)))
Cover(R) R)
(en(Aob))
quantifies the proportion of annotated objects mentioned in the response.
'nttss//unsplash.com/
19

---

Hal (Hallucination Rate):
1
if CHCAIIR(E)) +
Hal R)
otherwise
represents the proportion of responses containing hallucinations.
Cog (Cognitive Similarity):
Len(R's n Hobi)
Cog( R)
len( (en(
evaluates whether hallucinations similar to human cognition, where Hobj is set of pre-defined hallucinatory target objects.
Discrininative Task Metrics:
Accuracy: percentage of correct binary predictions across all questions.
Precision: proportion of true hallucinations among predicted hallucinations.
Recall: proportion of actual hallucinations that were correctly identified.
Fl Score: harmonic mean of precision and recall.
The overall AMBER Score is calculated as:
AMBER Score 51 CHAIR + F1).
A.2
Object HalBench
Object HalBench [32 is widely adopted for evaluating object hallucination in image descriptions. The benchmark leverages diverse prompts for detailed image description evaluation, with 4 instructions adapted from M-HalDetect [33] and 4 generated by GPT-4. The full set of prompts shown in 5 The evaluation uses 300 randomly sampled images from the COCO validation set.
A.2.1
Evaluation Metrics
Instead of using exact-match detection in previous work, the evaluation in Object HalBench leverages ChatGPT to extract mentioned objects from generated descriptions (Instructions shown in 6), providing better precision and recall. The evaluation process generates descriptions for benchmark images and uses these ChatGPT-extracted objects to calculate final scores. The metrics focus on two ballucination rates:
Response-level hallucination rate CHAIR-s:
Number of responses with object hallucinations
divided by the number of responses that introduce COCO objects
Instance-level hallucination rate CHAIR-i: number of falsely mentioned COCO objects divided by total number of mentioned COCO objects
A.3
MME
The Multimodal Large Language Model Evaluation Benchmark (MME)
31
represents pioneering
effort to comprehensively assess the capabilities LVLMs. This benehmark evaluates models across two fundamental dimensions: perception and cognition. The perception dimension measures model's s ability to recognize specific objects, including their existence, count, position, and color, as well as perform fine-graines recognition tasks. The cognition dimension evaluates model's capacity to integrate perception information with language model knowledge to generate more complex answers.
MME encompasses 14 distinct subtasks distributed across these two dimensions. The perception category includes 10 subtasks: four coarse-grained tasks (existence, count, position, color), five fine-grained recognition tasks (movie poster, celebrity, scene, landmark, artwork), and OCR (text recognition in images). The cognition category comprises four subtasks: commonsense reasoning,
20

---

Table 5: List of prompts used in Object HalBench.
Prompts
Provide a thorough description of the given image.
What is this photo about? Please answer great detail.
Provide a thorough description of the given picture.
Explain the narrative or story that the image seems to convey, detailing each part that contributes
to it.
Please provide detailed description of the image. Describe the visual elements, colors, shapes, textures, and any objects or people present along with the overall mood or atmosphere portrayed in the image.
Please provide detailed description of the image, including its visual elements, such as colors, shapes, textures, objects, and people.
Provide an intricate description of the image, capturing its visual elements, including colors, shapes, textures, objects, and any people present.
Compose detailed account of the image, encompassing its visual characteristics, like colors, shapes, textures, objects, and any human subjects, by paying careful attention to the specifics.
Table 6: GPT-4o-mini instruction for object extraction.
GPT-4o-mini Instruction
You are an expert in image objects extraction according to question answer pair. We asked an examiner to answer question about picture.
[Start of Question] (question) [End of Question]
[Start of Examiner's Answer) [answer) [End of Examiner's Answer]
Assume that the answer is correct, please identify all visible objects that are directly shown in the image. Please following the instructions in below:
1. You should only mention objects that are explicitly mentioned in the examiner's 's answer.
2. You should only extract the object names without the attributes of the objects.
3. You should not include the properties of the object, like the color, material, etc. as part of the
object name in your result.
4. Make your answer precise. Present the results in a JSON list format: ["object 1", 1".... "object
nn].
5. You should return empty JSON list (() if no visible objects can be found.
numerical calculation, text translation, and code reasoning. This comprehensive approach ensures thorough evaluation of LVLMs' capabilities across various visual understanding and reasoning scenarios.
MME includes 30 images with 60 innrructionanssweer pairs for each coarse-grained perception task, 147 to 200 images for fine-grained recognition tasks (147 for movie posters, 170 for celebrities, and 200 each for scenes, landmarks, and artworks), 20 images with 40 instruction-answer pairs for OCR, 70 images with 140 instruction-answer pairs for commonsense reasoning, and 20 images with 40 instruction-answer pairs each for numerical calculation, text translation, and code reasoning. All instruction-answe pairs are manually constructed to avoid data leakage, even when images from public datasets are utilized.
A.3.1 Evaluation Metrics
MME employs two evaluation metries: ccuracy and accuracy+. Accuracy measures the percentage of questions answered correctly, while accuracy+ is stricter metric requiring that both questions for given image are answered correctly, better reflecting comprehensive understanding. The score for each subtask is calculated the sum of accuracy and accuracy+, with maximum possible score of 200 per subtask. Consequently, the full perception score 2000 (10 subtasks), and the full cognition score is 800 (4 subtasks).
21

---

Table summmarzes the statistics of the benchmarks we consider in our work.
Table Statistics of benchmarks used in our work.
Benchmark
Images
Total Samples
Answer Type
LLM Evaluation
AMBER
1,004
15,220
Generative & Discriminative
No
Object HalBench
300
2,400
Generative Discriminative
Yes
MME
1,127
1457
No
B Implementation Details
For constructing the co-occurrence graph, we utilize the COCO 2017 validation set [26]. dividing each image representation into 3 x 3 grid cells. When computing semantic coherence co-occurrence statistics, we employ panoptic segmentation masks while deliberately excluding ubiquitous segments such sky-other-merred. To minimize noise in co-occurrence patterns, we retain only the top 10% of total connections in the final graph. Our GNN architecture comprises two layers and utilizes neighborhood sampling with a batch size of 2048. Detailed analysis of GNN training, e.g., training convergence and performance metrics, is presented in Appendix C.2 To ensure balanced-sized clus- ters, modify the K-means algorithm to produce more uniform cluster sizes following the previous work [34], which has been demonstrated to be beneficial for autoregressive image token processing. We discuss how to tune other other performance related hyperparameters in Appendix B. For baseline methods (SID, VCD, and OPERA), we maintain their default hyperparameter configurations. For PROJECTAWAY, due its model-specific hyperparameter sensitivity, we conduct comprehen- sive hyperparameter search on a representative subset of AMBER to identify optimal configurations for each model. For Chameleon-7B. we use (1' , ... (20, 18, 3.0). For Janus-Pro-7B. we use (1",1",, "4,00 a) = (25, 23, 1.5). For Emu3-13B, we use (1' 12 a (18, 24, 1.5) We use a single
NVIDIA H100 (80G) GPU for all experiments. We modify the Transformers package K5 adapt
all baseline methods to the discrete image token-based LVLM architectures. xformers 36 is used
for efficient attention calculation and to save memory.
B.1 Determine Optimal Hyperparameters
We tune four hyperparameters that impact hallucination mitigation performance: (1) cluster size, (2) number of selected most dominant clusters for hallucination mitigation, (3) editing layer I, and (4) editing magnitude coefficient 7. We employ a two-stage grid search approach. the first stage, we fix the editing parameters at intermediate values (1 20, 7 0.5) while exploring cluster configurations. We evaluate cluster size within (5, 10. 15, 20) and number of selected dominant clusters within 2,, 3, 4, 5} on subset of the AMBER benchmark. After identifying the optimal clustering configuration for each model, we proceed to the second stage, where we fix the searched cluster byperparameters and conduct searches over editing layers I {1, .... 30) and magnitude coefficient y E (0.1, 0.2, 0., Performance evaluated using the CHAIR metric on generative tasks.
Table summarizes the optimal hyperparameter configurations identified for each model. While the specific values vary across architectures, we observe that moderate cluster size (10) consistently leads to good performance, with the number of selected dominant clusters varying based on model size and architecture. For editing parameters, iitermedditteto-ddee layers (21 to 27) with moderate editing magnitude coefficient (0.2 to 0.6) yield optimal results.
To evaluate the robustness of our method, we plot the performance of LVLMs under different editing hyperparameter combinations on the AMBER generative task. As shown in Figure 4 many settings yield substantial improvements over the baseline, demonstrating that CGC+VTD is robust to hyperparameter choices. This is especially valuable for practical use, as it suggests the method can be effectively applied to new models without extensive tuning. Figure [ further demonstrates how the performance varies with different hyperparameter choices along each layer. The results show layers (avoiding the last 2) yields optimal results. This aligns with LVLM knowledge evolution processes where early layers aggregate information while deeper layers form conceptual representations [37 38).
that editing intermediate-to-
22

---

Table 8: Optimal hyperparameter configurations for each model.
Model
Cluster Size
Selected Dominant Clusters
Editing Layer (1)
Editing Magnitude Coefficient (t)
Chameleon-P7
10
25
0.5
Janus-Pro-7B
10
Emu3-13B
10
21
25
24
10
13-
12
122
Figure 4: The three heatmaps show CHAIR score difference between our method with the Nucleus Sampling baseline. Blue means improvement and Red means deterioration. It is shown that our method benefits from wide range parameter combinations.
Editing analysis
Cluster analysis
ltyeers basiling
10
20 king layert))
35
25
10 15
20
20 Editing layer|0)
1.5
25
Editing layer(()
Figure 5: We show how editing weight y, cluster size, number of GNN layers and editing layer affect our method. We report CHIAR score for generative tasks and F1 for discriminative tasks using Janus-Pro-7B. Overall, our method is relatively robust to hyperparameters.
B.2
Combination with Existing Methods
While some methods be orthogonally combined with approach, others require special considerations to ensure effective integration.
Since OPERA focuses on biases in language modeling by applying penalty and retro- spection strategies, operates on fundamentally different aspect of the hallucination problem than method. When combined, VTD first addresses the visual priors by suppressing visually absent tokens from dominant clusters, while OPERA subsequently handles potential language modeling biases. This orthogonal combination yields substantial improvements beyond what either method can achieve alone.
OPERA:
For these approaches, we apply VTD only on the normal logits branch because otherwise the effect of promoting hallucinative logits through either noise or specific attentional mechanism might be compromised. Contrastive decoding relies on comparing original and perturbed inputs, and applying VTD to both branches could dilute the contrastive signal. Our experiments show that this selective application helps maintain the intended contrastive mechanism while still addressing visual prior-driven hallucinations.
Contrastive Decoding Methods (VCD and SID):
This method uses a two two-pass inference approach- the first pass calculates internal confidence to identify hallucinative text tokens, while second pass incorporates this information to suppress hallucinations. We apply VTD in both passes to maintain consistency in the model's internal representations. This ensures that the internal confidence measurements remain valid and that the subsequent projection operates properly decontaminated representations.
PROJECTAWAY:
23

---

More details in Context-Guiicdd-husteiing
C.1 Algorithm
We present the detailed algorithm 1 to construct the co-occurrence graph described in Sec. 3.1 Note that the notation used in this algorithm is solely for illustrating the graph construction process and may conflict with the formal notation in the main body of the paper. intended for clarity and should not be interpreted consistent with the main text.
Algorithm Context- Guidded Co-occurrence Graph Construction
Require: Codebook entries Z E R/V x denaseost image dataset D with segmentation masks Ensure: Graph G (N, with edge weights S Initialize co-occurrence matrix M E I V with zeros Initialize node set V = (1.2,...,IV) [V}} for each image I D with segmentation mask S do Encode and quantize to obtain codebook indices [21, 22, 2m Determine spatial dimensions h, such that m h w for index = 1 to m do
(ri, c:) 2D position of i in the h grid seg; + segment ID at position (ri, for index i m do
(ry, + 2D position of j in the x grid segj 6 segment ID at position (r); if (r;, ci) and (rj, Cj) are within 3x 3 grid cells or seg: = segj then
M 2 2j
M[2 2j] +1
M 222 2 t M 22 24 +1
end if end for
end for
end for
Normalize to the range [0.
Nonnections count of non-zero entries in M
+- [0.1 X N (Determine number edges to keep)
E top-k entries in M with corresponding node pairs
+ weights edges in from M
return G = (N, with edge weights S
C.2
GNN Training
C.2.1
Attentional Coefficient
The attentional coefficient (() in Eq. is computed by incorporating the edge weight Sij. i.c..
normalized co-occurrence strength, between n, and ny through:
exp (a)') a (.(" (('-))
w!
C1' + W
(((')s)) 81j
(5)
meNa((n) exp a(()) W
(V-1)
w (1) (1--1) 1)
+ w!l") 8am
((")) w and w-(") are three weight matrices with size dow xdoxx (1) E Rddow is a vector
parameter.
C.2.2 Configurations for GNN Training
Neighborhood Sampling For each node in a batch, we sample neighbors across 2 layers with sampling sizes [48, 16], meaning we sample up to 48 neighbors in the first layer and 16 neighbors in the second layer. This approach allows us to capture both immediate and extended neighborhood information while maintaining computational efficiency.

---

Learning Rate and Optimization We employ the AdamW W optimizer with an initial learning rate of 4 -000 and cosine annealing scheduler with warmup. The warmup period consists of 10% of the total training steps, during which the learning rate increases linearly from 0 to the maximum value. This helps stabilize early training stages, especially for large codebooks.
Regularization Techniques To prevent overfitting and ensure stable training, we implement multiple regularization strategies: (1) dropout with rate 0.1 on node features, (2) edge dropout with rate 0.1 applied randomly during training, (3) gradient clipping with maximum norm 0.5, and (4) weight decay of 0.01 for L2 regularization.
Dynamic Temperature Adjustment We implement dynamic temperature adjustment based on similarity statisties. The temperature starts 0.02 and is adjusted during training:
(0.00. x 0.95)
if A < 0.3
T++1
min(0.15, x 1.1) if Asi > 0.5
(6)
Tt
otherwise
where
sim
Spos
Sneg is the difference between average positive and negative similarities.
Table summarizes all training configurations, and Table 10 shows the model-specific architectural dimensions.
Table 9: Training hyperparameters for CGC GNN.
Category
Parameter
Value
Learning rate Scheduler
4 10-3
Cosine with warmup
Optimization
Warmup steps Optimizer
10% of total steps AdamW
Weight decay
0.01
Batch size
2,048 nodes
Total epochs
100
Training
Early stopping patience
10
Max gradient norm
0.5
Number of GNN layers
2
Number of attention heads
Architecture
Dropout rate Edge dropout rate
0.1
0.1
Sampling
Number of neighhors
[48, 16]
Table 10: Model-specific dimensional configurations.
Model
Initial Dim
Hidden Dim
Output Dim
Codebook Size
Chameleon-78
256
128
256
8.192
Janus-Pro-7B
8
128
32
16,384
Emu3-13B
64
32
32,768
The choice of dimensions reflects the different tokenization strategies used by each model. Chameleon uses larger initial dimension due to its VQGAN-based tokenization, while Janus and Emu3 employ more compact representations. The hidden dimension is chosen to balance expressivity with computational efficiency, while the output dimension is tailored to each model's 's downstream task requirements.
C.2.3 Computational Efficiency
Table II summarizes the computational consumption for GNN training.
25

---

Table 11: Computational efficiency metrics
Metric
Time/Memory
Graph construction time
~3 hours(for 1,000 images)
GNN training time
~0 minutes (HI00 GPU)
Memory requirement
16GB GPU memory
Inference time per image
<Ims
Parameters (hidden=32)
855K
FLOPS per forward pass
15M
C.3 Importance of the absolut similarity constraint
Our core intuition is that while Leomtrast co-oceurrence weights for relative optimization, Lyps en- forces absolute similarity thresholds that ensure adequate separation between positive and neg- ative pairs regardless of batch composition. Concretely, the gradient of Leomtraa w.r.t. sim(h;, hy) is positively correlated to Sij while negatively correlated to the number negative sample pairs. So the pressure improve the similarity between node and node j towards a high Stz could be diminished by a large number of negative pairs within the batch. Cpps addresses this by setting explicit minimum similarity targets (B 8ty) that must be satisfied independently of other pairs in the batch. Table T2 details the effect of Cpps on GNN representation learning using Janus-Pre Pre---ro-BBB 7B on the COCO 2017 validation set.
Table 12: Results of positive sample similarity and negative sample similarity. We use the averaged consine similarity scores among each partition.
Variant
avg. positive similarity
avg. negative similarity
avg. similarity gap
wfo. Cyps
0.56
0.35
0.21
W. Lyp
0.79
0.46
0.33
Without Cyps, the model achieves only 0.56 average positive similarity with a similarity gap of 0.21 between positive and negative pairs. With Lape. positive similarity increases substantially to 0.79, while the similarity gap expands to 0.33 by 57%. This demonstrates that Laps successfully enforces the absolute similarity constraints, ensuring that co-occurrence relationships preserved with sufficient fidelity for downstream clustering and hallucination mitigation. Furthermore, the ablation study in Table 4 consistently shows performance degradation when Lpps is removed, confirming that Cpps is crucial for our method.
C.4 HitRate @ K: Further Analysis Hallucination Detection
Here we present additional analysis on HitRate K with different image token groups Figure 6 shows the results on Chameleon-7B, and Figure 7 shows the results on Emu3-13B. We find that the observations from Sec. (322 still hold for the other two models. This confirms that hallucinations are closely linked to visual priors that can evoke absent tokens from dominant clusters, and CGC provides an effective means of uncovering these associations.
C2 CGC
12.5
15
15
K-means
Hiftate 186
Hitbate
2.0
HitRate
30
5.0
2.5
9
16
Cluster Ranking
Figure 6: Left: Hallucination HitRate for K from 1-10. Middle: Comparison between C2 tokens identified by our CGC method versus naive K-means. Right: HitRate@5 of C.2 tokens from the ten most dominant clusters. The analysis is based on Chameleon-7B.
26

---

C
20
C K-means
HitRatee 196)
Hitkate (99)
HitRater (963
20
2.0
10
Ranking
Hallucination HitRate for K from 1-10. Middle: Comparison between C2 tokens identified by our CGC method versus naive K-means. Right: HitRate@5 of C2 tokens from the ten most dominant clusters. The analysis is based on Emu3-13B.
Figure 7: Left:
D Additional results
D.1 Combination with Existing Methods:
Further Results
Table 1 and Table 14 provide combined methods' results Object HalBench with Canus-Pro-7B and Emu3-13B. We have the same observations as we have discussed about Table
Table 13: Manus-Pro-7B results when combining CGC+VTD with baselines.
Metric
VCD+
SID+
PROJECTAWAY=
OPERA+
CHAIR-s 4 19.67(+0.01)
18.33(-2.34)
12.03(-1.06)
11.94(-0.39)
CHAIR-i 11(+1.32)
10.65(-1.28)
7.56(-0.83)
6.01(-0.72)
Table 14: Emu3-13B results when combining CGC+VTD with baselines.
Metric
VCD+
SID+
PROJECTAWAY+
OPERA+
CHAIR-s L
21.03(-0.3)
24.98(+2.65)
1966(-0.02)
17.21(-1.61)
CHAIR-i
13.99(+0.12)
14.31(+1.56)
12.42(+1.33)
10.23(-0.88)
D.2 Efficiency Analysis: Further Results
Table 15 and Table 16 compare the efficiency of different hallucination mitigation methods Janus- Pro-7B and Emu3-13B, evaluated on AMBER generative and Object HalBench tasks. The results show that method consistently achieve the best performance.
Table 15: Efficiency analysis Janus-Pro Pro-7B
Metric
VCD
SID
PROJECTAWAY OPERA
CGC+VTD
Time (s)
488
387
521
853
166
Memory (MB) J
18,736
17,583
15,880
29,462
15780
Token
414
395
193
175
169
Table 16: Efficiency analysis on Emu3-13B. To perform OPERA, might need offload some computation to CPU.
Metric
VCD
SID
PROJECTAWAY
OPERA
CGC+VTD
Time (8) 4
797
770
935
472
Memory (MB) 1
20,583
19,358
17.593
17,489
# Token
285
268
198
186
190
27

---

D.3 Qualitative Results
We present three representative cases in Figure to illustrate our qualitative results. These examples show that CGC+VTD enables LVLMs with discrete image tokenizers to generate coherent and expressive image descriptions while demonstrating greater robustness to hallucinations.
Nucleus Sampling The image thows young child taking a both. The child eitting a batitu filed with scary water, The child is holding a yollow spout and pouring water over their hands. There's botto of purple soap or shampoo on the edge of the buthtue, and snred by peoeagll frading the staty water..
OPERA
tup There child taking bath both sile with
The child hoiding yellow from which try
fowing
bottl bath
edge the
flosiin the
bockground
SID: young thild taking coth i battnub filed with stapy water. The chil is holding a yellow oup from which water pouring info the water Them are bubeles in the water. the background them purple bottee hanging on the wall which appears to be shampoo or candiiooneeo the image depiage a scaew of bath time. The child is string up in the buthtue iedd with buttiles and theme are othew trye indainn wiloo stufled buttedly and grewn toy Boosing in the lut.... VCD: image
young taking purple bottle
filedd
soep the
The hokingg are childss
face. hair color? Blond The child taking a bath bathhub that filldd with water and seap bubbles,
and pouring waler
and
Ground outh objects:
purple bottle of
on the well. Additoonally there small boy resembiing ling lomon is floating the watar...
PROJECTAWAY
chiiel tiking bath baitlib The child menenee the water, which wolee floing There purple botl seheepoo the wage of the bettrut and some
shows
from
bett shampon, Iyy.
swen flowing wateer
chac, bette, woter
Lackground
modie.
will o1p
Ours: The image ahooe a young deid laking a bech The cheld adinn in belfitud: with butcbes and saup auths The cleid in haslding
yo cup from which water ls fowing The background includes wh iied walls and purple bottl placed on the edge of the bothib.
Nucleus Sampling:The The the photograph weitee van pertally visble en the right side of the imagee.
black border white text front with background, sky visible
canopy
yellow leaves.
STOP
OPERA The image is photograph ol stop sign in front of free with red and yellow leaves. The stop sign red with white border and white text. The sign is anached to a pole that attached to a tree by a metal bracket. The the has red and yellow leaves and front a white house with a black front door.
sign front posts. The sign has legibl text leeves are predominently
imaga a picture
stop
pole that attached white taxt and black ovarloy scattered throughout.
with some green
Ground truth
objects:
while border and the word I"STOPY wr iten on it it mounted on metal pole that angles up and loward the right. The pole
VCD: The image shows a stop sign street corner behind free with red and yellow leaves. The stop sign red with
sky, free sky'.
stoter
sign taken PROJECTIVN TAAAAYY is an mage of a stop sion with a picture a bird superimposed on it. The stop slon has red bac herogru wh white border around the picture the bird. The bird is the center the sign and is facing left. is black and white with a red beak and yellow eyes.
the branchee extend
behind the which right upward. This
shadow the for side the The trunk the tree picture of
the left side of the image,
"Gurs: The image ai stop sign displayed the center. "he yellow leaves behind the sign. The leaves have autumnal hues, with shades of yellow and brown, indicating the season likely foll.
with white lort. There green lealy tree with red
Nuciaus Sampling: This is an imagi a pizza on paper tray on a table. The pizza placed on the thin crust pizza with red sauce and cheese. There is also a small bawl cheese on the table next the pizza.
the tray.
OhERA image shows dese- up view a siice pizza on a cardboard tray The pizza on thin crust and is tooped with tamato sauce, choese, and a green herb. The pizza is sittnng on a beige doth napkin. There a smail brown ceramic vase on the sable nead to the pizza. The vase has small amount of water in The tible made of light brown wood. The background a beige carpet.
sib The image
close-pp. overheed view of Neapolian-slyle pizza on butling board
doomaaiiiis like"
material, with focal pizza partislly obsoured nearty coffee The
deep red-orange a thin, base tomato white and
and
byyer
herbs, skely plantro parsle. On top the herbs is Ined with doormat material, upon
Vou, mex_nen,so 212
yellow
ciroular slices. The background consists
Ground
objocts:
piasappie, ham onions. The plate
VCD:
is doss up of slice
gray woven
paper plate The pizza is trick cruet with while olives.
tokkoupuliee
PROJECTANAY The Image a close up view of freshly made pizza on cardboard box. The pizza is on a thin crust. tapped with various Ingra arraadentts Bik chaese and rad sauce, with a sprinkle of horts. Thare are also red sauce and choose on the pizza in the background. The pizza is placad on bolge cioth, and thore is dark brown and while stripd nug undemeath. in the backgroure and there is wooden bowi and smell pol of red sauce. The background is beige will with white sectured su surface The image in a cleeeup view of freehl made pazza on cardhoard hox.
plzza libk:
toconiz.hoosoo
nackin
pincapote
Ours: image shows close view a siice of pizza carcboard tray. The pizza on thin crust topped tomato sauce, cheese, and green herb. The pizza is sitting on a belge dloth napiin. There small brown ceramio cup on the table next to the pizza. The cable made of lightt brown wood.
Figure 8: Cases qualitative results. Comparison of applying different types of hallucination mitiga- tion methods on Chameleon-7B: Red words are hallucinated objects and green words correspond to correctly mentioned objects. Note that CD-based methods (SID and VCD) tend to continue generating until reaching the maximum output length.
28

---

Average Change
223%
220
50
Length Change (86
40
35%
30
24%
20
7%
10
Chameleon
Janus-Pro
Emu3
LLAVA-v1.
Figure 9: Change of answering length when applying VCD.
D.4 Discussion on the Failure of Contrastive Decoding-Based Methods for LVLMs with Discrete Image Tokenizers
To examine the limitations of contrastive decoding (CD) methods on LVLMs with discrete image tokenizers, analyze the behavior of VCD M applied to both discrete-token models and LLaVA- v1.6 [39], which uses continuous visual representations. Experiments on the AMBER generative task reveal clear pattern: As shown in Figure 9 VCD significantly increases response length in discrete- token models-by as much as 223% for pruus-rro-uuggesting instability or overgeneration this setting. While longer outputs may improve coverage by mentioning more ground truth objects, they also increase the likelihood of hallucinations. This trade-off is evident in our main results (Table I where CD-based methods applied to discrete tokenizer-based LVLMs consistently underperform on gains in coverage. We further illustrate
the CHAIR metric-indicating more ballucinations-despe this issue through case studies in Figure 8 For instance, SID 2 and VCD often extend responses unnecessarily, even when the initial answer is sufficient. In the first example, SID redundantly restates the scene ("The image displays... ") after the model has already completed coherent description.
D.5 Evaluation on POPE
POPE (Polling-basel Object Probing Evaluation) [19 is benchmark designed to systematically evaluate object hallucination in LVLMs by formulating the task as binary classification problem with yes/no questions about objects in images. We perform POPE evaluation on both MSCOCO [26) and GQA 40 datasets under three settings: Random, Popular, and Adversarial. The evaluation metrics are accuracy and FI score, which have been presented in Appendix(A.I.]]
The results in Table 17 demonstrate that our proposed CGC+VTD method achieves the best per- formance on 31 out of 36 metrics across different models and datasets and OPERA emerges as the second-best method in most scenarios Notably, the strong performance of CGC+VTD the GQA dataset demonstrates excellent generalization capabilities, considering that CGC was trained exclu- sively on the COCO images (including MSCOCO). This cross-dataset transferability suggests that our method captures fundamental aspects of visual priors rather than merely exploiting dataset-specific patterns.
29

---

Table 17: Evaluation results on POPE. Best/secood best results are marked bold/underlined.
Random
Popular
Adversarial
Dataset
Model
Method
Accuracy
F1 Score
Accuracy
Score
Accuracy
FI Score
Nucleus Sampling
50.6
60.0
55.00
62.4
52.7
Chameleon-27
VCD
62.2
51.7
62.3
50.9
61.4
51.4
61.0
50.9
60.7
52.2
62.1
OPERA
52.6
65.8
54.2
66.1
50.2
65.7
Cluster-Base
50.6
60.9
54.9
62.2
52.5
61.1
CGC+VTD
52.7
63.9
56.5
68.0
53.0
66.9
Nucleus Sampling
89.3
88.4
87.5
86.7
84.7
Janus- Manus-Pro-BB
VCD
70,9
63.5
68.6
61.1
MSCOCO
SID
68.9
68.0
57.7
66.3
$5.8
OPERA
88.9
87.8
87.2
85.4
85.0
Cluster-Buse
88.2
87.0
86.6
85.5
84.8
83.9
CGC+VTD
89.8
89.2
88.2
86.9
85.9
86.2
Nucleus Sampling
87.4
86.0
85,5
84.1
83,9
82.7
Emu3-13H
VCD
87.0
85.6
84.5
83.2
83.3
82.2
SID
87.5
86.1
85.0
83.7
83.0
81.9
OPERA
87.9
85.3
86.4
85.4
84.2
83.8
Cluster-Buse
87.3
85.8
85.4
83.9
83.8
82.5
CGC+VTD
88.3
87.0
87.1
86.1
84,9
83.8
Nucleus Sampling
53.2
62.5
57.6
64.8
63.9
Chameleon-7B
VCD
54.3
64.6
54.5
64.7
53.6
63.8
SID
54.1
63.5
53.7
63.2
54.9
64.5
OPERA
55.2
68.2
56.9
68.6
52.9
68.1
Cluster-Base
53.3
63.4
57.4
64.6
55.1
63.7
CGC+VTD
55.4
66.4
59.1
70.4
55.7
69.3
Nucleus Sampling
91.7
89,9
89.2
87.8
87.3
Janus-Pro-PP Pross-rro-BB
VCD
74.0
66.7
73.6
66.2
71.3
63.9
GQA
SID
71.6
62.1
70.7
60.6
69.0
58.7
OPERA
92.2
91.4
90.3
89.7
87.9
87.5
Cluster-Base
90.8
89.7
89.2
88.1
87.4
86.5
CGC+VTD
92.3
91.7
90.7
89.4
88.5
88.8
Nucleus Sampling
90.0
88.7
88.1
86.9
86.5
85.4
Emu3-13B
VCD
89.6
88.3
87.1
86.0
85.9
84.9
SID
90.1
88.8
87.6
86.5
85.6
84.6
OPERA
90.5
88.0
89.0
88.1
87.8
86.4
Cluster-Bass
89.9
88.5
88.0
86.7
86.4
85.2
CGC+VTD
90.9
89.7
89.7
88.8
87.5
86.5
30


Reviews:
[{"review_id": "abhSt00j7l", "reviewer": "Reviewer_eq2o", "summary": "This paper proposes a novel method, named CGC, to analyze the impact of relationships - including co-occurrence and semantic coherence - between discrete visual words on the hallucination ratio of LVLM models. Based on the analysis results, the authors further propose a solution, named VTD, which reduces the hallucination ratio during decoding by subtracting feature components that may cause hallucinations. Extensive experiments are conducted to demonstrate the effectiveness of the proposed methods.", "questions": "1. In Eq. 2 and Eq. 3, I am curious about the intuition behind why the contrastive loss alone is not strong enough, necessitating the introduction of an additional loss to constrain positive similarities. Could the authors elaborate on this point?\n\n2. In Figure 3, group C1 refers to tokens from the dominant cluster(s) that appear in the image. How do these tokens cause hallucinations if they are indeed present in the image?\n\n3. Section 3.2 should be described more clearly. How does the COCO segmentation mask match the AMBER annotations? Can we ensure that all objects contained in AMBER images fall within the predefined COCO segmentation classes?\n\n4. In Eq. 4, since $\\hat{g}^{(l)} (v_{hal})$ has been normalized using L2-Norm according to Line 255, can I understand that the denominator in Eq. 4 will always be 1?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. There is currently a lack of sufficient discussion about the causes of hallucinations in VLMs, and this paper makes a valuable contribution to addressing this gap.\n2. The authors conduct extensive experiments to demonstrate the effectiveness of their proposed methods.\n\n### Weaknesses\n1. The significance of the proposed method is a bit limited, as most VLMs now use continuous visual tokens to achieve better perception and cognition performance. As mentioned in the limitations section, the method can only be applied to VLMs using discrete tokens, which are mostly designed for unified understanding and generation models.\n2. Some points could be further clarified; please refer to the “Questions” section for details.\n3. The overall writing, organization, and illustrations could be improved to enhance clarity and readability.", "was_revised": true, "final_justification": "I still find the proposed framework overly complex, and I believe its methodological significance remains limited. Therefore, I will maintain my original score.", "rebuttal": "We appreciate your positive feedback on our **motivation and solid experiments**. We hope our answers can address your concerns and questions.\n\n---\n\nW1: The significance of the proposed method is a bit limited...\n\nA1: Recent advances in LVLMs have increasingly focused on unifying understanding and generation within single frameworks [1,2,3], with discrete image tokenizers emerging as a particularly promising approach [4,5,6]. The recent success of BAGEL [4], which outperforms Qwen2.5-VL on 6 out of 7 image understanding benchmarks, demonstrates the potential of this unified architecture paradigm.\n\nWe acknowledge that most previous hallucination mitigation research on LVLMs is based on continuous LVLMs; however, as LVLMs with discrete image tokenizers are gaining more popularity and have shown extraordinary performance, we believe it is very important to systematically study hallucination mitigation specifically for discrete tokenizer-based LVLMs. To our knowledge, we are the first to identify and quantify hallucination sources arising from discrete image token co-occurrence patterns, providing a novel perspective for understanding visual hallucinations in unified multimodal architectures. Our proposed CGC+VTD framework directly addresses these visual priors through targeted latent space intervention, achieving superior performance while maintaining computational efficiency. \n\n    [1] Zhang, Xinjie, et al. \"Unified multimodal understanding and generation models: Advances, challenges, and opportunities.\" arXiv preprint arXiv:2505.02567 (2025).\n    [2] Li, Zijie, et al. \"Dual diffusion for unified image generation and understanding.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025. \n    [3] Qu, Liao, et al. \"Tokenflow: Unified image tokenizer for multimodal understanding and generation.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025. \n    [4] Deng, Chaorui, et al. \"Emerging properties in unified multimodal pretraining.\" arXiv preprint arXiv:2505.14683 (2025). \n    [5] Lin, Haokun, et al. \"Toklip: Marry visual tokens to clip for multimodal comprehension and generation.\" arXiv preprint arXiv:2505.05422 (2025).\n    [6] Xie, Jinheng, et al. \"Show-o2: Improved Native Unified Multimodal Models.\" arXiv preprint arXiv:2506.15564 (2025).\n\n---\n\nW3: The overall writing, organization, and illustrations could be improved to enhance clarity and readability.\n\nA2: Thanks for the suggestion. We will improve the clarity and readability in the revision.\n\n---\n\nQ1: In Eq. 2 and Eq. 3, I am curious about the intuition behind why the contrastive loss alone is not strong enough...\n\nA3: Our core intuition is that while  $\\mathcal{L}\\_{\\text{contrast}}$ uses co-occurrence weights for relative optimization, **$\\mathcal{L}\\_{\\text{pps}}$ enforces absolute similarity thresholds that ensure adequate separation between positive and negative pairs** regardless of batch composition. Concretely, the gradient of $\\mathcal{L}\\_{\\text{contrast}}$ w.r.t. $\\text{sim}(\\mathbf{h}\\_i, \\mathbf{h}\\_j)$ is positively correlated to $s\\_{ij}$ while negatively correlated to the number of negative sample pairs. So the pressure to improve the similarity between node i and node j towards a high $s\\_{ij}$ could be diminished by a large number of negative pairs within the batch. $\\mathcal{L}\\_{\\text{pps}}$ addresses this by setting explicit minimum similarity targets ($\\beta \\cdot s_{ij}$) that must be satisfied independently of other pairs in the batch. To further address your concern, we conducted additional analysis on the similarity patterns during GNN training using Janus-Pro-7B on the COCO 2017 validation set. \n\n| **Variant** | **avg. positive similarity** | **avg. negative similarity** | **avg. similarity gap** |\n|-------------|------------------------------|------------------------------|--------------------------|\n| w/o. $\\mathcal{L}_\\text{pps}$ | 0.56 | 0.35 | 0.21 |\n| w. $\\mathcal{L}_\\text{pps}$ | 0.79 | 0.46 | 0.33 |\n\nWithout $\\mathcal{L}\\_{\\text{pps}}$, the model achieves only 0.56 average positive similarity with a similarity gap of 0.21 between positive and negative pairs. With $\\mathcal{L}\\_{\\text{pps}}$, positive similarity increases substantially to 0.79, while the similarity gap expands to 0.33 by 57%. This demonstrates that $\\mathcal{L}\\_{\\text{pps}}$ successfully enforces the absolute similarity constraints, ensuring that co-occurrence relationships are preserved with sufficient fidelity for downstream clustering and hallucination mitigation. Furthermore, the ablation study in Table 4 consistently shows performance degradation when $\\mathcal{L}\\_{\\text{pps}}$ is removed, confirming that $\\mathcal{L}\\_{\\text{pps}}$ is crucial for our method. \n\n---\nQ2: In Figure 3, group C1 refers to tokens from the dominant cluster(s) that appear in the image. How do these tokens cause hallucinations if they are indeed present in the image?\n\nA4: **C1 tokens do not directly cause hallucinations, but they can be associated with hallucinated objects due to the limited vocabulary size of discrete tokenizers**. Since discrete tokenizers use a fixed vocabulary to represent diverse visual content, each token must encode multiple visual concepts and objects. Consequently, any given token—including those present in the image (C1)—will be associated with various objects, encompassing both those that genuinely appear in the current image and those that do not. Our analysis demonstrates that while C1 tokens may indeed correlate with some hallucinated objects, the absent C2 tokens from the same dominant cluster show significantly stronger correlations with hallucinated content. This finding reveals that hallucinations primarily arise from the model's inappropriate activation of absent tokens that frequently co-occurred with present ones during training, rather than from the present tokens themselves. Therefore, C1 tokens are not the direct cause of hallucinations—they legitimately represent visual content in the image. The hallucinations stem from the learned co-occurrence priors that cause the model to inappropriately reference their absent cluster partners (C2 tokens). This is why our Visual Token Decontamination method specifically targets C2 tokens for suppression while preserving the contributions of present C1 tokens.\n\n---\nQ3: Section 3.2 should be described more clearly....\n\nA5: For the analysis in Section 3.2, we only consider the AMBER objects that are covered by COCO segmentation classes. We do this analysis as follows. We first construct a token-object association dictionary for each image token using the COCO panoptic dataset, where we compute the frequency of this token with which it is used to represent objects from different segmentation classes. A token is considered to represent an object if it lies within the segmented area corresponding to that object. The association dictionaries of all image tokens are then aggregated within token groups (C1, C2, C3) to summarize how frequently tokens in each group are used to represent different segmentation classes, based on which we construct Top-K (C) sets ranked by association frequency. To compute HitRate@K, we extract hallucinated objects identified by the AMBER and use string matching against COCO segmentation classes to determine whether each hallucinated object appears within the Top-K(C) set.\n\nAlthough this approach may overlook the hallucinations linked to AMBER objects not covered by COCO classes, it still ensures methodological consistency and does not compromise our core findings. The dramatic HitRate difference between C2 and other groups remains statistically significant, validating our hypothesis about visual priors.\nTo further address your concern, we conduct another HitRate analysis using 500 images from the COCO dataset with Janus-Pro-7B. We can observe that C2 has a much higher HitRate compared with using AMBER, and the huge gap between C2 and other groups (C1 and C3) remains.\n\n| **Token group** | HitRate@2 | HitRate@4 | HitRate@6 | HitRate@8 | HitRate@10 |\n|-----------------|-----------|-----------|-----------|-----------|------------|\n| C2              | 0.18      | 0.26      | 0.33      | 0.33      | 0.37       |\n| C1              | 0.08      | 0.11      | 0.11      | 0.14      | 0.16       |\n| C3              | 0.06      | 0.09      | 0.11      | 0.16      | 0.20       |\n\nMoreover, it i is important to emphasize that visual priors extend beyond object-level associations—they also capture spatial relationships where tokens frequently appear in proximity within images. Our CGC method deliberately incorporates both spatial co-occurrence (tokens appearing within the same local regions) and semantic co-occurrence (tokens representing the same segmented objects) to comprehensively model these visual priors at the token level.\n\nThe HitRate analysis serves as an illustrative proxy to demonstrate the effectiveness of our clustering approach. Our method fundamentally operates on token-level co-occurrence patterns rather than being limited to object-level associations. This token-centric approach enables our method to capture richer visual relationships that go beyond simple object categories.\nThe strong performance of our method across different datasets also validates its generalizability. As shown in Table 1, our approach trained on COCO data effectively mitigates hallucinations on AMBER, which has a different data distribution. This cross-dataset effectiveness demonstrates that our method captures fundamental visual co-occurrence patterns rather than dataset-specific artifacts, making it broadly applicable to diverse evaluation scenarios.\n\n---\n\nQ4: In Eq. 4, since $g^{(l)}(v_\\text{hal})$ has been normalized using L2-Norm according to Line 255, can I understand that the denominator in Eq. 4 will always be 1?\n\nA6: Yes, your understanding is correct. We will simplify this equation in the revision."}, {"review_id": "VMtvwYNbJR", "reviewer": "Reviewer_ecat", "summary": "This paper addresses the hallucination issue in large vision-language models (LVLMs) with discrete image tokenizers, proposing a two-stage framework: Context-Guided Clustering (CGC) and Visual Token Decontamination (VTD). CGC captures visual priors by constructing a co-occurrence graph of image tokens and leveraging graph neural networks, while VTD mitigates hallucinations via latent space editing during generation. Experiments on benchmarks like AMBER and Object HalBench demonstrate the method’s effectiveness, but the comparison settings and baseline models have notable limitations.", "questions": "1. Baseline relevance: Please verify how the proposed method compares to the SOTA model (e.g. Qwen2.5-VL) for comprehension tasks.\n2. Benchmark scope: Are existing benchmarks sufficient to evaluate the baseline model's hallucination in generation tasks? Please verify on a generation-centric dataset.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. Novel problem framing: Identifies visual priors from token co-occurrence patterns as a key source of hallucination in discrete-token LVLMs, filling a research gap.  \n2. Integrated methodology: CGC combines spatial and semantic contexts to model token relationships, and VTD provides a targeted latent editing strategy, forming a cohesive solution.  \n3. Comprehensive evaluation: Experiments compare multiple baselines across multiple benchmarks, showing consistent hallucination reduction without sacrificing model expressiveness.  \n\nWeaknesses:  \n1. Inadequate baseline fairness: Compared methods (e.g., VCD, SID, OPERA) are primarily designed for understanding-focused LVLMs, not tailored to discrete tokenizer architectures. This may lead to biased comparisons, as contrastive decoding methods, for example, may conflict with discrete token representations.  \n2. Suboptimal baseline models: Models that unify understanding and generation, such as Chameleon, Emu3, and Janus-Pro, use discrete tokenizers and perform worse than current SOTA LVLMs (e.g., Qwen2.5-VL, Intern3-VL) in visual language understanding tasks. The benchmarks only focus on understanding tasks and lack evaluation on generation tasks.", "was_revised": true, "final_justification": "The authors address my concerns on discrete tokenizer-based LVLMs.", "rebuttal": "Thanks for your valuable feedback. We appreciate that the reviewer found that our paper proposed a **novel problem and a cohesive methodology**. We address your concerns below.\n\n---\nW1: Inadequate baseline fairness: Compared methods (e.g., VCD, SID, OPERA) are primarily designed for understanding-focused LVLMs, not tailored to discrete tokenizer architectures. This may lead to biased comparisons, as contrastive decoding methods, for example, may conflict with discrete token representations.\n\nA1: To the best of our knowledge, we are the first to investigate hallucination detection and mitigation in LVLMs using discrete image tokenizers. Existing state-of-the-art methods were all developed for LVLMs with continuous image encoders, and our experiments have shown them to be ineffective when applied to models with discrete tokenizers. We believe our experimental setup is fair. As you noted, the failure of methods such as contrastive decoding may stem from a fundamental mismatch with discrete token representations. **This is not a result of unfair evaluation, but rather a limitation inherent to methods designed for continuous encoders. Our findings highlight this gap and constitute a meaningful contribution, rather than a flaw**. We appreciate your feedback and welcome further discussion. \n\n---\n\nW2: Suboptimal baseline models: Models that unify understanding and generation, such as Chameleon, Emu3, and Janus-Pro, use discrete tokenizers and perform worse than current SOTA LVLMs (e.g., Qwen2.5-VL, Intern3-VL) in visual language understanding tasks. The benchmarks only focus on understanding tasks and lack evaluation on generation tasks.\n\nA2: LVLMs with discrete tokenizers have recently attracted rapidly growing attention, making it important to investigate their behavior with respect to hallucination. In this paper, we focus exclusively on LVLMs with discrete image tokenizers. Models such as Qwen2.5-VL and Intern3-VL rely on continuous image encoders and are therefore beyond the scope of our study. \n\n**Hallucination in VLMs, as defined by previous research [1, 2, 3, 4], is characterized by VLMs generating textual descriptions that include objects not present in or inconsistent with the target images**. The benchmarks we employ (AMBER, Object HalBench) are established and well-adopted standards in the hallucination-related literature. We strictly follow previous works and use these benchmarks to ensure fair comparison.\n\nWe are not certain what you mean by “generation tasks”. If you are referring to image generation tasks, they fall outside the scope of our paper, as they conflict with how multimodal hallucination has been defined in prior work, focusing on visual understanding. If you are referring to text generation, multimodal hallucinations are identified based on the generated textual output, as defined in prior works. Our setting follows this established definition. We are happy to address any additional concerns you may have.\n    \n    [1]: Rohrbach, Anna, et al. \"Object hallucination in image captioning.\" arXiv preprint arXiv:1809.02156 (2018).\n    [2]: Li, Yifan, et al. \"Evaluating Object Hallucination in Large Vision-Language Models.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. (2023)\n    [3]: Wang, Junyang, et al. \"Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation.\" arXiv preprint arXiv:2311.07397 (2023).\n    [4]: Liu, Jiazhen, et al. \"PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2025).\n\n---\n\nQ1: Baseline relevance: Please verify how the proposed method compares to the SOTA model (e.g. Qwen2.5-VL) for comprehension tasks.\n\nA3: **Our experimental design follows standard practices in hallucination mitigation research, where the appropriate comparison is between different mitigation methods applied to the same base model, not across different architectures.** Comparing against Qwen2.5-VL would be out of the scope of this paper. As clearly stated in our abstract and introduction, we specifically target LVLMs with discrete image tokenizers and do not cover models with continuous image encoders, such as Qwen2.5-VL.\n\n---\n\nQ2: Benchmark scope: Are existing benchmarks sufficient to evaluate the baseline model's hallucination in generation tasks? Please verify on a generation-centric dataset.\n\nA4: As we clarified in the previous question, hallucination of VLM happens where models generate plausible but incorrect textual descriptions of visual content. The benchmarks we employ (AMBER, Object HalBench) are specifically designed and widely adopted for evaluating hallucinations in visual understanding tasks. Visual generation hallucinations (where models produce incorrect visual content) represent a fundamentally different research problem, which is beyond our research scope."}, {"review_id": "6iRY6WKt96", "reviewer": "Reviewer_qYkC", "summary": "This paper introduces a framework for modeling visual priors in LVLMs equipped with discrete image tokenizers. Motivated by the observation that certain image tokens frequently co-occur to represent common objects or scenes, the authors construct a co-occurrence graph of codebook entries and apply a GNN-based clustering method to uncover dominant token groups. By clustering over codebook embeddings rather than raw token embeddings, the method ensures both computational efficiency and improved semantic coherence. The resulting clusters are shown to be closely associated with hallucination-prone tokens, and the authors further propose a targeted mitigation strategy that edits intermediate representations based on dominant clusters.", "questions": "See Weakness section. I would consider raising the score if the identified concerns are properly addressed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n[1] The paper clearly motivates the concept of visual priors and effectively models them through GNN-based clustering over image tokens derived from a discrete tokenizer.\n\n[2] It also provides a compelling rationale for clustering over codebook embeddings rather than raw image token embeddings, highlighting both efficiency and semantic coherence.\n\nWeaknesses:\n\n[1] Regarding Section 3.2, please consider providing examples of the Top-K(C) object categories for each group—such as C1, C2, and C3—or examples of the tokens contained in each group, along with corresponding hallucinated objects (obj′) and segmentation annotations from the COCO panoptic dataset. The current experimental setup is somewhat difficult to follow without concrete illustrations.\n\n[2] In Section 3.3, please consider using a term like v_candidate instead of v_hal, as the current terminology may be somewhat misleading. This is especially relevant since hallucinated tokens cannot always be explicitly identified in certain benchmarks (e.g., MME or Object HalBench), unlike in AMBER.", "was_revised": true, "final_justification": "The authors’ response greatly clarified my understanding. Since all of my concerns have been addressed, I will keep my score unchanged.", "rebuttal": "Thank you for the positive evaluation of the **motivation and our proposed method**. We will address your questions below.\n\n---\n\nW1: Regarding Section 3.2, please consider providing examples of the Top-K(C) object categories for each group—such as C1, C2, and C3—or examples of the tokens contained in each group, along with corresponding hallucinated objects (obj′) and segmentation annotations from the COCO panoptic dataset. The current experimental setup is somewhat difficult to follow without concrete illustrations.\n\nA1: Consider the following example. For the third case in Figure 8, the top-5 object sets associated with $C_1$,$C_2$ and $C_3$ are: $\\text{Top-5} (C_1)=('table', 'sand', 'towel','pizza','cake')$, $\\text{Top-5} (C_2)=('sand','bowl','table','cake','water')$, $\\text{Top-5} (C_3)=('skateboard','table','ski','knife','light')$. In here, the hallucinated object 'bowl' is the second most significant object associated with $C_2$ and does not appear in the top-5 objects associated with $C_1$ and $C_3$. This demonstrates our central finding: absent tokens from dominant clusters ($C_2$) show stronger correlations with hallucinated content (the object \"bowl\" in this case) than present tokens or tokens from non-dominant clusters. Please note that although $\\text{Top-5}(C_2)$ also contains several ground truth objects, removing $C_2$ tokens' influence primarily eliminates hallucination-prone associations while preserving essential visual information through $C_1$ tokens. This selective intervention maintains recall for actual visual content while reducing false positive generations. We will include this example in our revision for better readability.\n\nTo supplement, we also present here how to derive the order of Top-K(C). We first construct a token-object association dictionary for each image token using the COCO panoptic dataset, where we compute the frequency of this token with which it is used to represent objects from different segmentation classes. A token is considered to represent an object if it lies within the segmented area corresponding to that object. The association dictionaries of all image tokens are then aggregated within token groups ($C_1$, $C_2$, $C_3$) to summarize how frequently tokens in each group are used to represent different segmentation classes, based on which we construct Top-K (C) sets ranked by association frequency. \n\n---\n\nW2: In Section 3.3, please consider using a term like v_candidate instead of v_hal, as the current terminology may be somewhat misleading. This is especially relevant since hallucinated tokens cannot always be explicitly identified in certain benchmarks (e.g., MME or Object HalBench), unlike in AMBER.\n\nA2: Thank you for your suggestion! We will update our notations in the revision."}, {"review_id": "Erxg0i8lOd", "reviewer": "Reviewer_WjUQ", "summary": "The paper proposes a novel hallucination mitigation method specifically designed for LVLMs based on discrete image tokenizers. The method is based on first constructing a co-occurrence graph to characterize image tokens that frequently co-occur based on their spatial proximity and semantic objects they are associated to. It then uses a GNN to extract image token embeddings and clusters them to represent the co-occurring tokens. Hallucinated objects are associated to tokens of the most dominant (\"commonly present\") clusters that are not detected in the current image. Finally, during inference, latent representations are edited to reduce influence of of all such tokens.", "questions": "1. How did you arrive to the 3x3 grid division for determining the spatial proximity. I expect it might not make a major difference but did you experiment with other choices?\n\n2. I am not entirely convinced by the need to use GNNs for extracting token representations. Did you try using a simple way of extracting node representations for clustering, for instance, just a vector summarising the edge strengths of a node? I am also not sure if line 237 references the same baseline. What representation did you use there for clustering?\n\n3. Line 315: Could you please describe more precisely the combination of CGC with other methods.\n\n4. If you performed a simple clustering step on continuous visual embeddings to manually discretize, does it not allow the method to also easily generalise to LVMLs using continuous visual tokens.", "limitations": "I am not sure the method generalizes to architectures that use Q-Former to process vision tokens even if discrete image tokens, or LVLMs which integrate vision modality through cross attention maps at all layers.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- The writing and paper presentation is very clear in most parts.\n- The problem is relevant, contemporary and the proposed method is also clearly novel.\n- Experimental evaluation is very solid, showing consistent improvement on multiple datasets, models and against multiple relevant baselines.\n\nWeaknesses:\n\nI do not believe the paper has any major weakness. I would like to rate the paper as 5, However, I do have doubts regarding certain design choices or experiments in the paper. Please refer to the questions section for the same.", "was_revised": true, "final_justification": "1. I already found the paper solid with no major weakness and was inclined to score it 5\n2. The authors adequately responded to my questions\n3. In my opinion, the only major concern I have seen in other reviews is that the method does not extend to LVLMs using continuous image tokens. Thus it has limited value. While an understandable concern, the authors have been very upfront about it in the paper from the beginning. To me the contribution is still valuable.\n\nOverall, I will increase my score to 5. At the moment, I am still open to discussion with other reviewers/AC. So this score could still change until deadline (Aug 13).", "rebuttal": "We appreciate your positive feedback about **high novelty, clear presentation, and solid experiments**. We address your questions below.\n\n---\n\nQ1: How did you arrive to the 3x3 grid division for determining the spatial proximity. I expect it might not make a major difference but did you experiment with other choices?\n\nA1: We have indeed explored other choices. We evaluated grid sizes from 1×1 to 6×6 on Janus-Pro-7B using the Object HalBench benchmark: \n| **Metric** | **1x1** | **2x2** | **3x3** | **4x4** | **5x5** | **6x6** |\n|------------|---------|---------|---------|---------|---------|---------|\n| CHAIR-s    | 14.95   | 10.78   | 10.43   | 12.52   | 12.11   | 13.27   |\n| CHAIR-i    | 9.94    | 6.11    | 5.81    | 6.19    | 6.32    | 7.01    |\n\nThe 3×3 grid achieves optimal performance, though 2×2 to 6×6 all provide reasonable results within acceptable ranges. The 1×1 setting performs poorly due to insufficient edge connectivity during graph construction, which impairs GNN learning quality by creating overly sparse co-occurrence relationships.\n\n---\n\nQ2: I am not entirely convinced by the need to use GNNs for extracting token representations. Did you try using a simple way of extracting node representations for clustering, for instance, just a vector summarising the edge strengths of a node? I am also not sure if line 237 references the same baseline. What representation did you use there for clustering?\n\nA2: The GNN learning step is essential, as it is widely recognized that GNNs are effective for node clustering due to their ability to learn node embeddings that capture graph structural information. In our case, the co-occurrence graph is constructed based on spatial proximity and semantic coherence, so effectively modeling its structure with a GNN amounts to capturing visual priors. We have shown in Table 1 that trivial clustering methods like naive K-means on the original codebook representations (Cluster-Base method) yield substantially inferior performance compared to our CGC+VTD approach. This supports our decision to employ GNNs.\n\nThe \"baseline\" method referenced in line 237 uses naive K-means clustering directly on the original VQVAE codebook embeddings without any graph-based representation learning.\n\n---\n\nQ3: Line 315: Could you please describe more precisely the combination of CGC with other methods.\n\nA3: We have provided details of how to combine CGC with other methods in Appendix B.2. Due to the page limit, we could not put them in the main body of our submission. We hope this can address your concern.\n\n---\n\nQ4: If you performed a simple clustering step on continuous visual embeddings to manually discretize, does it not allow the method to also easily generalise to LVMLs using continuous visual tokens.\n\nA4: This is a great question that we have also considered. While the intuition appears straightforward, the practical implementation presents significant technical challenges. Our method fundamentally relies on identifying \"absent tokens\" from a finite, discrete token vocabulary. Adopting continuous visual embeddings can be understood as introducing infinite visual tokens, making it impossible to get the set of \"absent tokens\" anymore. \n\n--- \n\nLimitation: I am not sure the method generalizes to architectures that use Q-Former to process vision tokens even if discrete image tokens, or LVLMs which integrate vision modality through cross attention maps at all layers.\n\nA5: While architectures like Q-Former and intra-layer cross attention were important early milestones in LVLM development, they fall outside the scope of our work. Our work focuses on a new class of LVLMs that adopt a unified token processing paradigm, where discrete image tokens are processed directly alongside text tokens within a single transformer architecture. In these models, there is typically no need for modality-specific components such as Q-Formers (as in BLIP variants), intra-layer cross-attention(as in Flamingo), or lightweight MLP adapters (as in LLaVA), since discrete tokenization naturally supports seamless cross-modal interaction without explicit alignment mechanisms. As the design space for unified LVLMs continues to evolve, our work offers foundational insights into hallucination behavior grounded in visual priors. These insights may help inform intervention strategies across future architectures, and we would be happy to engage in further discussion if you are interested."}]

Author rebuttal:

Meta-review: This paper proposes the CGC method to analyze how co-occurrence and semantic coherence among discrete visual words affect the hallucination rate in LVLMs. Based on the findings, the authors further introduce VTD, which reduces hallucinations during decoding by removing feature components prone to causing them. Extensive experiments demonstrate the effectiveness of the proposed methods.

Pros.

1. This paper identifies a new problem, which studies the hallucination mitigation in LVLMs with discrete image tokenizers.
 
2. The motivation of the method is clear. The concept of visual priors is effectively modeled by using GNN-based clustering on codebook embeddings. 

3. The method of using latent space editing for hallucination reduction is novel. 

Cons.

The applicability of the study is limited to some extent. Most existing methodologies used to process the image adopt continuous vision feature, while this paper studies the discrete tokens. 

Overall, this paper is novel in its problem and methodology but limited in its applicability.
Decision: Accept (poster)