Discovering Compositional Hallucinations in LVLMs
Authors: Sibei Yang
Ge Zheng
Jiajin Tang
Jiaye Qian
Hanzhuo Huang
Cheng Shi
Venue: NeurIPS 2025 poster ()
Keywords: Hallucination
LVLMs
Primary area: applications

Abstract: Large language models (LLMs) and vision-language models (LVLMs) have driven the paradigm shift towards general-purpose foundation models. However, both of them are prone to hallucinations, which compromise their factual accuracy and reliability. While existing research primarily focuses on isolated textual- or visual-centric errors, a critical yet underexplored phenomenon persists in LVLMs: Even neither of textual- or visual centric errors occur, LVLMs often struggle with a new and subtle hallucination mode that arising from composition of them. In this paper, we define this issue as Simple Compositional Hallucination (SCHall). Through an preliminary analysis, we present two key findings: (1) visual abstraction fails under compositional questioning, and (2) visual inputs induce degradation in language processing, leading to hallucinations. To facilitate future research on this phenomenon, we introduce a custom benchmark, SCBench, and propose a novel VLR-distillation method, which serves as the first baseline to effectively mitigate SCHall. Furthermore, experiment results on publicly available benchmarks, including both hallucination-specific and general-purpose ones, demonstrate the effectiveness of our VLR-distillation method.

Discovering Compositional Hallucinations in LVLMs
Sibei Yang Ge Zheng²
Jiajin Tang² Jiaye Qian Hanzhuo Huang²
Cheng Shi
School of Computer Science and Engineering, Sun Yat-sen University
TShagghaTTec University
Schooo of Computing and Data Science, The University of Hong Kong
Abstract
Large language models (LLMs) and vision-language models (LVLMs) have driven the paradigm shift towards general-purpose foundation models. However, both of them are prone hallucinations, which compromise their factual accuracy and reliability. While existing research primarily focuses on isolated textual- or visual-centric errors, a critical yet underexplored phenomenon persists in LVLMs: Even neither of textual- or visual centric errors occur, LVLMs often struggle with a new and subtle hallucination mode that arising from composition of them. this paper, we define this issue as Simple Compositional Hallucination (SCHall). Through an preliminary analysis, we present two key findings: (I) visual abstrac- tion fails under compositional questioning, and (2) visual inputs induce degradation in language processing, leading to hallucinations. To facilitate future research on this phenomenon, introduce custom benchmark. SCBench, and propose novel WLR-distillation method, which serves as the first baseline to effectively mit- igate SCHall. Furthermore, experiment results publicly available benchmarks, including both hallucination-sseeiffi and general-purpen ones, demonstrate the effectiveness of our Vis--iittlllation method.
Introduction
Large language models (LLMs) 3 62, g and large vision-language models (LVLMs) 12. 37 78
have driven the paradigm shift from task-specific general-purpose approaches, cementing their role as the de-facto foundation in natural language processing and computer vision research. However, both LLMs and LVLMs are prone to hallucinations (24 73 55
77
15), posing
significant risks in real-world applications [74] 68). In LLMs, hallucination research primarily addresses discrepancies between model outputs and real-worlh facts or user inputs- -ie. factuality hallucination and faithfulness hallucination (21). Compared to LLMs, LVLMs incorporate visual understanding, which naturally extends hallucinations include visual recognition errors textual responses inconsistent with the referenced phage-particularky, in object categories [55 33 77]
attributes, and relationships (64. 27). To suppress these hallucinations, recent work has achieved
promising results through improved architecture [6]
7], inference interventions (2 22
66 28
and auxiliary training data or strategies 777 26
72
52
76
focus on isolated forms of hallucina- tioneeither textual factuality and faithfulness enrors or visual recognition failures (see POPE [33 and TruthfulQA 34 in Figure [)). But what if neither occurs on its own? Intuitively, if an LVLM answers both simple vision-centric and a simple language-centric question correctly, without hallucination, should also succeed when the two are composed into single query. Yet, unexpectedly. it hallucinates. We observe that when these seemingly reliable components are combined into a single question, the LVLM fails- hasssshalallnannnnggg where no error existed before. For example, as shown in Figure [b), the LVLM independently recognizes the goldfish the image and understands that adding more
Despite recent progress, most existing studies
B3 55
34
11
Cooresponding author is Sibei Yang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Actussly
They thoves gridts
Scaple Visust
Staple Teut
Crapoctrton
Vinsa
102%
untc acrouctic
Not-hall
1111
25%
Teaple
IVEE
sking
Sel
9.%
Expected
plaiig vow looks
sking reerpture Soil placing Sed
27%
Compestien
Stmpbe Tee
13 Elack-1 6
Internll
5P7-46
(c)
Figure 1: (a) Comparison of the research scope of our SCHall with prior work. (b) Example of SCHall. LVLMs provide accurate answers to simple visual- or textual-centric questions but fail to reason compositionally when these questions are combined. (c) LVLMs exhibit SCHall on our compositional benchmark, showing performance drop of 20% compared individual questions.
plants increases the oxygen content of the fishbowl. However, when asked the compositional question "which action would increase the oxygen content of the object", it hallucinates and produces an incorrect answer.
In this paper, we define this phenomenon Simple Compositional Hallucination (SCHall): new and subtle hallucination mode that does not arise from textual- or visual-centin questions individually. but from their compositions parioonssspttctlarrry when each component question is simple and independently hallucination-free To better investigate the SCHall phenomenon, we construct curated benchmark, namely SCBench, comprising triplets: one more simple visual-centric questions, a simple textual-centric question, and their corresponding compositional question. To ensure diversity, the visual questions cover object classification [16 10] attribute recognition [2]
and OCR recognition 12 [48. while the textual ones span commonsense inference [41] factual
verification [46 57). and numerical reasoning 44 655) Triplets are generated semi-automatically:
simple visual and textual questions are automatically created per image using GPT-3.5 (51), verified to
be correctly answered by most LVLMs, including LLaVA series (37), Qwen-VL series MiniGPT- 4 [78) and Intern VL series [7]. then paired into compositional questions and manually filtered and revised for both quality and difficulty. Notably, our benchmark differs fundamentally from existing LVLM hallucination benchmarks, such as POPE [33 (Figure [b) which primarily target isolated visual recognition errors. In contrast, we focus on failures that emerge from composing questions that are individually simple and reliably answered. It is also distinct from knowledge-centric VQA benchmarks (e.g., OK- VQA (46), where the bottleneck lies in the textual subproblems, corresponding to limitations in external knowledge. More importantly, neither the data construction process nor the evaluation in OK-VQA considers compositionality. Unlike recent reasoning benchmarks [44] that emphasize multi-step reasoning chains, we instead target single-step compositions that unexpectedly induce hallucination.
65 69
Further, we validate that the SCHall phenomenon is widespread across a variety of LVLMs, rather than being confined isolated cases, as evidenced by evaluations on both our compositional benchmark and general-purposs benchmarks. As shown in FigureT(c) LVLMs such as the LLaVA series 38), InternVL. 7. and GPT-4o (50) exhibit substantial performance drops-accurant on compositional questions decreases by nearly 20% compared to their near-perfect accuracy on the corresponding standalone visual- and textual-centric questions. When evaluating general- purposse benchmarks (e.g.. MMBench [41] MME [12], MMVet (71). we decompose each question into visual- and textual- centric sub-questions (detailed in SecB.11 While visual recognition a major source of hallucination, notable pattern emerges: when the visual sub-question is answered correctly, hallucinations less frequently result from errors in the textual sub-question, which is also more likely answered correctly. Instead, they arise from the composition of sub-questions that are otherwise independently answerable (Figure 2). Interestingly, this phenomenon is more pronounced in stronger models such as Qwen and Vn [7] compared to 1.5 [37. As their visual recognition improves, hallucinations from visual errors decrease, while those caused by composition become more prominent- further underscoring the importance of studying the SCHall phenomenon.
To probe the underlying causes of the SCHall phenomenon, we conduct a series of analyses and identify two preliminary factors that may contribute to it. First, LVLMs struggle with composition- ality, particularly in targeting relevant visual content, more notably, abstracting it into textual understanding. We find that masking irrelevant visual regions-forcing the model to rely solely

---

on relevant content- significaanl improves performance, indicating that compositionality hinders accurate largeting of critical visual information. Similarly, inserting textual cues into the question that directly reference key visual regions also yields gains, suggesting that failures in abstracting visual content into textual understanding can induce hallucinations even in simple compositional settings (see Sec. for details). Second, the mere presence of visual input-even blank canvas devoid of meaningful content-can degrade the model's language processing performance. We observe that on the ScienceQA dataset [43] attaching a blank image to purely textual questions-i.e. those answerable without visual input- to a noticeable drop in performance. Moreover, when comparing compositional and textual-centric question pairs, the answer logits for compositional questions are substantially lower (often by half), and correct answers tend to appear later in the output sequence. All these findings (detailed in Sec. 3.3) suggest that language processing is disrupted in compositional settings.
Based on the aforementioned definitions and findings, we propose a novel baseline, VL---distilatton as the first attempt to address SCHall. To promote effective visual information extraction and representation, we introduce innovative token type, referred to as the Vision Language Registers (VLRs), which serves as a bridge between the visual and linguistic modalities. Designed to represent the mnestion-relevant image information while also engaging in textual understanding like text tokens, the VLRs fulfills the roles of both visual localizers and abstract semantic encoders, thereby reducing the model's 's functional gap between recognition and compositional tasks. Furthermore, to counter the degradation of language processing capabilities caused by visual integration, we introduce a textually-enhancee
ILR-distillation strategy. Leveraging the inherent strength of language models in textual reasoning, we employ a text-represented visual branch as the teacher to guide the LVLM student, enabling it to preserve its language processing while effectively incorporating visual context.
To validate our findings and the effectiveness of the VLR -iR-distillattin, extensive experiments across various benchmarks demonstrate that the VLRs and distillation learning strategy not only yield signif- icant improvements on our SCBench but also prove effective on different hallucination benchmarks and general VQA benchmarks. This further supports the validity, necessity, and generalizability of our proposed SCHall for LVLMs.
In summary, our contributions are as follows: (1) We identify a pervasive and fundamental SCHall phenomenon and introduce SCBench to systematically assess it, revealing significant limitations in LVLMs and paving the way for advancing hallucination research. (2) We conduct a thorough analysis of the challenges associated with this phenomenon, i.e., attending to relevant visual content while preserving accurate and fluent language processing. (3) We propose VLR-distillation to mitigate hallucination, yielding substantial improvements not only on SCBench but also across diverse hallucination benchmarks and general VQA benchmarks.
2 Simple-Composed Benchmark Construction
This section describes the data generation process of our SCBench benchmark. Following a bottom-up strategy, we first construct atomic questions targeting visual recognition and textual understanding. These are then composed into simple-composed questions. Finally, we perform cross-validation to filter out those that are likely to induce SCHall. For further details of SCBench benchmark, please refer to the Appendix.
Atomic Questions Generation. We collect images from established datasets, including MM- Bench [1), MME [12 and SEEDBench (31] as well as from various online sources. In addition, we manually synthesize images containing texts, numbers, and geometric shapes increase diversity. For each image, we generate captions using popular LVL.Ms including LLaVA series (37] Qwen-VL. series (2]. MiniGPT-4 (78) and InternVL series 77. identifying common content "casily" recog- nized by these models. Based on these captions, we use GPT-3.5 [51] to formulate corresponding recognition questions, which are subsequently filtered and refined through manual verification.
To construct textual cognition questions, we prompt GPT-3.5 using the recognized content as contex- tual input, encouraging it generate questions from diverse linguistic perspectives. After manual verification, we evaluate the same LVLMs on these questions and retain those with high accuracy as "easy" instances.
Cimple-Composed Question Generation. By replacing the text-represented visual content in the textual cognition questions with corresponding image inputs, we construct candidate simple-
3

---

MMRennch
MMB
MMVel
LLaVA-I
Chit
InternVi
18.4%
15.9%
19:1%g
27.5%
2%
5%
22.1%
54.8%
45.2%
28.
28.7%
30.35
24.4%
29.4
23.5%
Textubl contric?
Textual centric
Textusl contric?
Testual centric?
Textual contric?
Textual centric?
Benelimrkk-
Modele
Figure Proportions of error attributed to visual recognition and textual understanding failures across different benchmarks and models. When visual recognition is hallucination-ffes (the first line in each square), hallucinations occur more frequently in questions that have correctly answered text-centric sub-questions (top left corner) than in those with failed ones (top right corner).
composed questions, where both the visual recognition and textual components are individually "easy" for L.V.Ms. From these candidates, we identify questions that remain challenging for those with relatively low accuracy across models-whiee are then manually reviewed and refined to construct the final benchmark.
Probing Simple-Composed Hallucinations
In this section, we first present statistical evidence that SCHall occurs in general-purpose benchmarks (see Sec. Sec.3.1] 3.T), supporting its broad prevalence, as also demonstrated by our benchmark introduced in Sec. Based on our benchmark, we then examine two primary factors contributing to SCHall under compositional settings: (1) the model's failure to effectively target and abstract question- guestion-relevant visual content (see Sec. (.2). and (2) the degradation of language processing pathways induced by the integration of visual inputs (see Sec. 3.3
3.1 Beyond Isolated Flaws: Simple-Composee Hallucinations
Motivation. Research hallucinations in LVLMs primarily emphasizes recognition errors. However, once these recognition tasks become "easy", do unique hallucinations specific to LVLMs continue to persist? In this context, we present empirical evidence (see Figure supporting a negative conclusion. To further establish the universality of this hallucination, we conduct a statistical analysis over diverse of samples drawn from general-purpose benchmarks, evaluating multiple LVLM series. We include experiments with different decoding strategies in Appendix.
Setting. We uniformly sample variety of question types from multiple datasets, including MME [12. MMBench [41]. and MM- Vet [7]] Each question is annotated with its decomposed components, including more recognition and language questions that together cover the visual and linguistic capabilities required to answer the original question. Our experiments are conducted based on LLaVA1.5-7b 37,, Qwen-VL.
2 and Intern VL 77. involving sampled set of 300 instances.
Result & Discussion. The results are presented in Figure As noted by previous research, recognition errors account for substantial fraction of overall failures (41.2% & 26.5% MMBench on the bottom-left corner). However, considerable portion of the remaining errors associates with simple textual- and visual-centric questions. These errors evident across both benchmarks and models dimensions, highlighting the prevalence this hallucination: Even when both the textual- and visual-centin questions are individually hallucination- free for LVLMs, their compositions still pose unexpected challenges. We further observe that the proportion this type hallucination increases from LLaVA (19.1%) recognition capability. This trend highlights the growing importance of addressing such hallucinations LVLMs continue to improve in perceptual accuracy.
InternVL. (27.5%), likely due the latter's stronger visual
3.2 Visual Abstraction Fails under Compositional Questioning
Motivation. In recognition tasks, the model identifies relevant elements based on explicit query. In contrast, compositional questions render the query implicit, as they often entail multiple intertwined sub-goals, thereby introducing new challenges. We thus hypothesize that potential cause of

---

Mullinadalig
Text
Find
LLaVA 37 Intern VL
Origin
41.3
64.1
M
Image masking
48.9
71.7
Text insertion
43.4
66.3
SQA-text 43
67.4
92.9
rand image
61.7
84.3
black image
65.1
90.6
Table I: Results of LLaVA and InternVL with various settings.
Figure 3: Analysis: Logit Lens analysis on our benchmark.
SCHall may lies in failures of visual abstraction triggered by implicit queries compositional settings.
Setting. To validate the hypothesis, we use two input modification strategies to provide recognition cues: (a) Image masking, where the original image is masked to retain only the region corresponding the queried content; (b) Text insertion, where additional textual tokens are inserted to highlight the relevant visual content. The inserted text is constructed solely based on existing visual information in the question, ensuring no information leakage (see Appendix for details). We conduct experiments using LLa --BB-BBBBBBBB-. and Intern VI. on subset of our benchmark.
Result & Discussion. As shown in Table both manipulations reduce SCHall, supporting our hypothesis: while the model attend effectively to relevant regions isolated recognition tasks, this selective ability becomes bottleneck in compositional tasks. Besides, image masking proves more effective than text insertion, as it directly eliminates misleading visual input, whereas text insertion only provides additional contextual cues for visual abstraction. Notably, using manually annotated masks resulted in modest improvement (8%), suggesting that other underexplored factors may contribute to SCHall.
3.3 Visual Inputs Induce Degradation in Language Processing
Motivation. However, what happens when the visual input is simple and free of distractions? We find that the model still exhibits the SCHall phenomenon. A typical failure mode involves the model conduct directly matching when answering questions, while neglecting the other context, exhibiting shortcut reasoning behavior. These observations lead us to hypothesize that the integration of visual information disrupts the language processing capability, ultimately giving rise to SCHall.
Setting. We conduct both statistical and visualization analyses on LLaVA115-7B to verify this hypothesis. (a) We first focus on 1,434 text-only questions from ScienceQA. To assess the influence of visual input, we pair each sample with an unrelated or visually uninformative black image, and examine the resulting changes in model performance. (b) To better understand the mechanism, we employ a logit lens analysis across transformer layers and token positions and take averages based on the benchmark. We focus on the final answer token, as well as intermediate result tokens that correspond to sub-answers of decomposed recognition questions. This approach enable us to trace how visual and linguistic signals are progressively integrated by the model during inference. Please refer to appendix for more details.
Result & Discussion. (a) Table shows a notable 5.78% performance drop when paired with unrelated images, and a 2.1% degradation with black images, revealing that visual inputs can even
negatively affect the model's language processing capabilities even when the visual input contains no meaningful content. (b) As shown in Figure 3 the layer-wise visualizations at the final input token (the first line) indicate that the answer logits in the purely textual condition are significantly higher than those in Figure 3 reveal a positional distinction: multimodal inputs exhibit notably weaker activations at earlier token positions compared text-only inputs. This finding suggest a delay in model's language processing under compositional settings, validating hypothesis.
compositional condition-off
nearly twice as large. While the second line
5

---

Model
Lenging
Language
Pretrain
LxL
Logils
Inference!
"Response'
Causal Attention
Section 5,3
Causal Amention
Cancal Attemtion
Logis
Ours
CLIP VIT
Tokemnierr
Language Registers (VL.Rs)
Tolomizer
Tokeniesr
CLIP ViT
Tolsenizer
Pretrained VLRs
Image
"Text
Captionce
"Image Caption"
"Text Input"
Image
"Test Input"
Figure 4: An overview our VL.R-distillation. Training Stage 1: Pretrain the VLRs with masked self-attention to learn selective image querying. Training Stage2: Distillation learning via teacher branch augmented with additional captions, providing enhanced language-gua -uided supervision. Infer- ence: Use pretrained VLRs in the same manner training to generate responses.
VLR-DDistillation Method
An overview of proposed method, VLR-Distillation, is depicted in Fig. 4 preliminary LVLMs Sec. Next, we introduce Vision Language Registers (VLRs) (Sec. 4.2)
We begin with
and employ distillation learning strategy (Sec. 4.3) to alleviate the perturbations in language processing that arise when visual inputs are integrated.
4.1 Preliminary
The Architecture of LVLMs. We consider LVLMs as consisting of vision encoder, vision- language connector, and a language model. Given an input image and an instruction T, the vision encoder processes the image to extract features, which are then projected into text-aligned feature space. Simultaneously, the instruction T tokenized into text tokens Xr, with their embeddings computed for subsequent processing. These projected visual features X I and text embeddings X serve as inputs to the language model. The reasoning process of the language model, leading to the output Y, can be articulated as follows:
p xY XT)
II SEL
po X X7, Y a)
where L denotes the set of output positions, and Y is predicted output before current token yi-
Causal Attention. Following LLMs, LVLMs employ causal attention to ensure that each position is unable to access information from future positions:
M,,, (Treer <
(2)
False, otherwise
where M,, indicates whether the token position has access to the token at position
4.2 Vision Language Registers: Absorbing Visual Content conditioned Language
To promote effective visual information extraction and, we introduce additional VLRs that selectively absorb relevant visual content aligned with the input textual query. Specifically, we randomly initialize N tokens in the feature space of text tokens, denoted as the sequence XVI.R. This sequence then concatenated with both the image input and the instruction input, enabling the generation of the answer Y be expressed as follows:
PY X X
II REC
pe (vill X1, XT, X VLLS. Y Yeil).
(3)
Pretraining. ensure that the randomly initialized VLRs exhibits visual abstraction capability, we first pretrain VLRs in which all other components of LVLMs remain fixed. We modify the attention

---

mask to prevent output tokens from directly attending to image tokens, thereby compelling the VLRs serve bridge by effectively aggregating information from the image tokens, as follows:
False, L. and E X
M
M r,s otherwise
4.3 Distillation Learning from Textual Enhanced Branch
To mitigate the degradation of language processing capabilities under compositional conditions, we propose Distillation Learning strategy. Specifically, we leverage the strength of language models on text-only questions to guide the model in preserving its inherent linguistic competence during multimodal inference. The text-only branch substitutes the image input in the form of image tokens X, with image captions tokenized as text tokens X The approach can be expressed as follows:
p PY
T)
II SEC
po (v/ll X7, Y'ail
We subsequently compute the Kullback-Leible divergence between the introduced text-only branch (5) and the original branch (3), denoted as:
LK Ev,r(DKI X
X XT)
(6)
The final loss L is formulated as follows:
Leeg L'reg + LKL
where Leg and reg indicates the language modeling loss of the original branch and the text-only branch.
5
Experiments
5.1 Experimental Settings
Datasets and Baselines. To evaluate the effectiveness of our method across various architectures, we experiment with LLaVA1.5 37 Qwen-VI-CCht VL-Chat (2), and MiniGPT-4 78 as primary baselines. For training. we employ subset of the training data from the instruction tuning (IT) phase of these models. Given that MiniGPT-4 is trained exclusively on caption data, exhibits limited capability in addressing broader VQA tasks. Therefore, we finetune MiniGPT-4 using subset of the IT training data from LLaVA1 as the baseline, which also serves as the training data for our proposed method. For inference, first conduct experiments with popular hallucination mitigating methods, to demonstrate the effectiveness of our proposed VLR-distillation. Additionally, report results popular ballucination benchmarks including POPE [33], MME-hall [12] and general-pupos VQA benchmarks encompassing ScienceQA [43 MMBench 41, HallusionBench [14 and MM-Vet (71..
proposed SCBench, comparing our methods
Implementation details. For training, we have two training phases: pretraining stage for VLRs and distillation learning, both following the alignment learning and instruction tuning stages of the baseline model. During pretraining, we use batch size of 128, freezing all other parts the model and training only the VLRs. In the distillation learning phase, we employ batch size of 64 with 2 accumulation steps, freezing the pretrained VLRs and training the LoRA [19) of the language model. For each baseline, set the number of VLRs N to 4. All experiments epoch, utilizing the Adam optimizer on 8 A.00 GPUs. For inference, we follow VCD [29] using nucleus sampling for experiments POPE and MME, while applying greedy decoding for other benchmarks.
conducted for a single
5.2 Comparison on Simple-Composed Benchmark
Table 2 presents the results of standard baselines, representative hallucination mitigation methods, and our proposed -i--diitillttion approach on the SCBench benchmark. Despite the low complexity of the questions in our SCBench benchmark, all three basclines perform poorly, achieving average
7

---

Score in Various Question Type
Model
Perc.
Sci.
Comm.
Fact
Lang.
Scene
Math
Overall
LLaVA1.5-7b
46.55
30.61
43.59
46.67
36.67
23.33
19.30
33.75
(9)
48.28
36.73
41.03
46.67
36.67
26.67
21.05
35.60
PAT[40]
32.76
24.49
28.21
16.67
40.00
13.33
14.04
23.22
CODE
46.55
32.65
46.15
50.00
40.00
25.00
21.05
35.60
REVERIE 72
48.28
32.65
43.59
40.00
43.33
21.67
26.32
35.29
CC
44.83
38.78
35.90
30.00
36.67
28.33
22.81
33.75
ours
51.72
46.94
48.72
53.33
46.67
28.33
28.07
41.80
+5.17
+16.33
+5.13
+6.66
+10.00
+5.00
+8.77
+8.05
VwenVL.-Chal
51.72
42.86
53.85
63.33
23.33
41.67
24.56
42.41
VCD 29]
51.72
42.86
53.85
66.67
23.33
38.33
28.07
42.72
PAI[40
56.90
42.86
43.59
50.00
20.00
31.67
24.56
38.70
CODE 28
50.00
46.94
53.85
60.00
23.33
38.33
22.81
41.49
ours
56.90
48.98
58.97
70.00
26.67
43.33
29.82
47.06
+5.19
+6.12
+5.12
+6.67
+3.34
+1.66
+5.26
+4.65
MiniGPT-4
22.41
12.24
17.95
6.67
23.33
16.67
5.26
14.86
+ 2C
24.14
14.29
20.51
6.67
23.33
16.67
10.53
16.72
PAT[40]
18.97
24.49
15.38
23.33
36.67
18.33
1.75
18.27
CODE 28
15.52
16.33
17.95
10.00
26.67
15.00
10.53
15.48
+ ours
32.76
34.69
25.64
30.00
30.00
18.33
21.05
26.93
+10.35
+22.45
+7.69
+23.33
+6.67
+1.66
+15.79
+12.07
Table 2: Results on Our SCBench Benchmark. Although their decomposed visual-centric and texutal-centric questions are hallucination-free LVLMs struggle with this "simple" dataset. Full names of the categories in our benchmark: Perception, Science, Commonsense Reasoning, Factual Knowledge, Language Capability, Scene Understanding and Math.
accuracy of approximately 30%. These results are consistent with our analysis in Sec. which suggests that LVLMs are prone to ScHall.
We also evaluate several popular hallucination mitigation strategies, including zero-shot methods, in- cluding VCD [29] PAI 40) and CODE (28). well training approaches, including REVERIE [72] and CCA 667. The experimental results indicate that while these methods can suppress object hal- lucinations, they do not perform well in mitigating the ScHall. Compared existing methods that focus solely on object hallucinations, our approach consistently achieves substantial improvements on LLaVAii5-77 7.B... Qwen- VL, and MiniGPT-4 on our SCBench benchmark. In particular, yields no- table gains of 8.05% and 12.07% in overall accuracy on LLaVAI 1.B-BB and MiniGPT-4, respectively, indicating its effectiveness activating the models' latent capabilities under composed scenarios.
Setting
Model
w/ ours
Accuracyf
Precision
Recall
Scoref
83.29
92. 02.13 (30..))))
72.80
(10.4
LLaVAL.5
87.46 (20.42)
92.04 (2.04(20.40)
82.06
86.76/40.49)
Random
Qwen- VI
-4.73(20.30)
95.61 (+0.45)
72.81 (=0.88)
82.67(20.41)
87.59, (=0.44)
93.68 (+.0.9)) (9)6))
(0.
86.66(+0.47)
MiniGPT-4
74.85 (#00.27)
80. (0.50((1..00 (00882)
65. (55600/2.55)
72. (2018
83.99 (+0.95)
90. (1,778/4.000 (-0.02)
75. 15.68,,2 (0
82.54
81.88 (100.88)
88.93 (10000)
72.80
80.06(10.05)
LLaVAL5
85.28 17)
87.02
83.02 83.02
84.98 (40.19)
Popular
Qwen-VI VL
(40.8)))
94.31 (+0.43)
72. 72.64(20.46)
82.06(20.23)
85.68 (0222)
89.88 (+:0233
80.41 (=0.32)
(4.88/40.255)
71. 71.85(20.66
70. (4.770(+0.00) (:066))
66. 66.09(=0.90)
13.
MiniGPT-4
80.45 (0.)3)
84. 04 (00688)
75.20(=0.77))
79.37 (#:027)
78. 78.96 (40.52)
83, 83.06 (1:0.58)
72.75 72.75
77. (+0.37)
LLAVAL5
81.18 (0.
80.2 80.24 (1:0077)
82.96(
81.56
Adversarial
VL
82.26 (40030)
89.97 (+:0.33)
72. 72.61, (20.50)
80.37 (+0.37)
82.86 (0.)7)
84.37 (+:0333)
80.67 (=
80.48 (1:0288)
MiniGPT-4
(#0.43) 43)
72. 72.03 (30..9))))
66. 06.03/=2.00 (00.59)
68.90(20044)
78.13 10)
79. (0
75. 75.49
77.54
Table 3: Results on POPE MSCOCO. The best performances for baselines is highlighted in bolded.

---

Object-level
Attribate-Ievel
Model
w/
Total Scores
Existencef
Counrt
Position
Colort
LLaVA1.5
K
(=1.00
(47.88)
(+1100
(+1.67
510.33 ( (226.66)
191.00(13.74)
(36.67
116.33 (11..22)
142.67
585.67 (122.89)
K
180.83(+5.34)
120.8 120.83 (110013))
115.28
168.61(28..8)
585.56((:12.40
Qwen-VL
186.67 (12.38)
(34.44(112..0
123.89 (18.44)
173.33 118.55)
618.33
MiniGPT-4
(47.0)
(9100())0000)
63.33 (=13.46)
(7533
371.67 (4:26.2)))
168.67 (14.144
88.33 (411.79)
71.67 (17.75)
111.67
440.33 (47.10)
Table 4: Results on MME-hall. Higher scores indicate better performance and fewer hallucinations.
5.3
Comparisons on Other Hallucination Benchmarks and General-purpose Benchmarks
The results on the POPE dataset are detailed Table Our method achieves substantial improvements across the random, popular, and adversarial setups for LLaVA1.5. Qwen-VL. VL. and MiniGPT-4. Notably, we observe enhancements on aceuracy of +9.14, 14, +8.60, and +7.94 over the MiniGPT-4 baseline in the three respective setups. Furthermore, our method shows significant im- provement in recall, with average values of +9.05, +8.97, and +9.12 in the three setups, highlighting that our VLRs prioritize visual information that is often overlooked and susceptible to interference from redundant data.
POPE.
MME. As shwon in Table 4 our method performs favorably on the benchmark. showing consistent improvements over bascline models across all splits. Notably, we achieve an improvement of 75 on
LLaVA baseline.
General-purpose Benchmarks. As shown in Table experiment MMBench 41 for compre- hensive evaluation, ScienceQA [4 for scientific questions, HallusionBench [4] for challenging hallucination questions, and MM-Vet [71] for open-ended hallucination questions. Our method consistently demonstrates an improvement of approximately 1.5 across these general benchmarks. Notable advancements, +1.9 and 2.2,, demonstrated in the HallusionBeench and MM-Vet, which focus on hallucinations.
Hallusion Bench [4
Model
MMB 41
SQA 3
MM-Vet
BI.I--2
61.0
Instruct L.P 9
39.8
63.1
45.26
25.6
MiniGPT-4 78
30.5
35.78
22.1
Qwen-VL. 2
38.2
67.1
39.15
Qwen-VLL VL-Chat
60.6
68.2
1LaVAv1.5 37
64.3
66.8
+ ours
65.4
67.8
49.5
33.3
Table Results on the general-purpose VQA benchmarks.
5.4
Ablation Study
VLRs
DL
Acc
Random
Fl
Acc
Popular
Fl
Adversarial
Acc
Fl
#VLRs
Acc
Random
F1
Popular Acc
F1
Adversarial
Acc
F1
X
83.3
81.3
81.9
80.1
79.0
77.6
2
87.4
84.7
84.8
80.2
86.5
85.9
84.1
83.7
80.2
80.4
87.5
85.0
83.4
84.3
83.0
81.4
80.4
8
87.7
87.4
85.3
85.2
80.1
80.8
87.5
86.8
85.3
85.0
81.2
81.6
16
87.0
86.3
84.7
84.3
80.9
81.5
Table 6: Ablation study POPE using LLaVA-v1.5 baseline.
To demonstrate the effectiveness of our proposed VLRs and distillation learning training strategies, we conduct ablation studies on POPE based on LLaVAI.5 baseline, shown in Table 6 (1) It can observed that each component- VL.Rs and the distillation learning strategy- strategy-indivvdually contributes to improvement in the model's performance on POPE. (2) It is noteworthy that the

---

independently used VLRs result in surprisingly sigmificant improvement. This indicates that the VLRs, as supplementary visual component, aids the model recognition. (3) Additionally, the distillation learning strategy for the caption branch, which reduce the internal gap on language-side, allows the model to learn to moderately prioritize image information and perform question-answering based on the primary visual content.
6 Related Work
Hallucinations in LLMs. The generation of meaningless or unfaithful sutputs-commann referred as hallucinations 44 54 55 in natural language generation has garnered considerable attention, as it poses significant risks to real-world applications of language models, particularly large language models (LLMs) [62 8 [. In the context of LLMs, hallucinations
typically classified into two primary types: factual hallucinations [47 60 211, where the generated output contradicts or cannot be verified by real-world facts, and fidelity hallucinations [21], where the output deviates from the input or fails to remain consistent with preceding output.
Hallucinations in LVLMs. Unlike hallucinations observed in LLMs, LVLMs introduce object
hallucinationns where generated content misaligns with the visual input [55] 331. This issue is
commonly attributed to language priors [66] [3),, statistical bias [66 771, modality gap [26].
Existing efforts mitigate object hallucinations by improving model architectures 61 7 67 curating
training datasets 36
77 70 72. designing learning strategies (6)]. or leveraging the intrinsic
properties pre-trained LVLMS
666
45).
Additionally, several studies discuss other types of hallucinations in LVLMs beyond object hallucina- tions. LRV 36 observe instruction-following failures, while more recent studies emphasize the issue of new types of visual hallucinations, including multi-object hallucinations [6), event hallucination [25 and prompted visual hallucination [39], respectively. Some recent approaches focus on probing the internal mechanisms (58) of LVLMs to attribute hallucinations, such identifying the different
causal pathways that lead to hallucinations 53 or understanding why longer contexts are more prone
to causing them 77)..
Benchmarks for Hallucination in LVLMs. To facilitate the study of hallucination in LVLMs, several benchmarks have been proposed. most of which primarily focus on object hallucinations. Early efforts, such as CHAIR, [55), concentrates on hallucinated objects in image captioning. Subsequent
benchmarks 33 12 14 42 adopt more structured formats, including yes/no and multiple-choice questions settings, to simplify evaluation. More recent efforts expand both the scope and evaluation protocols. For generative tasks, GPT-based tools [36] 59 71] offer flexible, context-aware evaluation, while FaithScore [27] provides fine-grained faithfulness assessment. On the discriminative side, go beyond objects to include attributes and relational inconsistencies.
recent benchmarks
20 25 39
7 Conclusion
This paper focuses on a phenomenon in LVLMs: despite accurately answer questions in isolated textual- and visual-centric questions, it still struggles in the compositional one. We also establish benchmark and conduct analysis. We further propose VLR-distillation and achieve high performance on our benchmarks and published ones. Limitation: It is important to acknowledge the potential ethical implications arising from LVLMs. Since our method leverages large vision language models like Llava and GPT-4o, it may also inherit biases and limitations present these models.
Acknowledgment: This work is supported by the National Natural Science Foundation of China under Grant 66.66226174 and No.62576365.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint or NXXY::2333.08774 2023.
[2] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Siman Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large laiiuala model with versatile abilities. arXiv preprint ar3iv:2308. 12966, 2023.
[3] Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005 00022000.11115.. 14165, 2020.
[4] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu. Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v22 large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.00M 2023.
[5] Keqin Chen, Zhao Zhang. Weili Zeng. Richong Zhang. Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
[6] Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang. David F Fouhey, and Joyce Chai. Multi-object hallucination in dision-language models. arXiv preprint arXiv: 2407.069996 2024,
[7] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su. Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisti tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198 2024.
[8] Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph Gonzalez, et al. Vicuna: open-souree chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general seneral-purpose lanounlangguae models with instruction tuning, 2023.
[10] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Icce, 2009.
[11] Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. Evaluating groundedness in dialogue systems: The begin benchmark. arXiv preprint 2rXi::2105.00077 4, 2021.
[12] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, al. Mme: A comprchensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[13] Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[14] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacooh, et al. Hallusionhench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. arXiv preprint arXiv:2310. 14566, 2023.
[15] Anisha Gunjal, Jihan Yin, and Erhan Bas. 8ohhea In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135- 1825-18143 18143, 2024.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[17] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song. and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[18] Zheng Yi Ho, Siyuan Liang, Sen Zhang, Yibing Zhan, and Dacheng Tao. Novo: Norm voting off hallucinations with attention heads in large language models. arXiv preprint 2rXiv:2410.08977 2024.
[19] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0906 2021.
11

---

[20] Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction uning. arXiv preprint arXiv:2309.02301. 2023.
[21] Lei Huang. Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang. Qianglong Chen, Weihua Peng, Xiancheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.022 2023.
[22] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection alloooatioooo.ion...m Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13427,, 2024.
[23] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[24] Ziwei Ji. Nayeon Lee, Rita Frieske. Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii. Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38. 2023.
[25] Chaoya Jiang, Hongrui Jia, Mengfan Dong. Wei Ye, Haiyang Xu, Ming Yan, Zhang, and Shikun Zhang. Hal-eval: A universal and fine-grained hallucination evaluation framework for large vision language models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 525-534, 2024.
[26] Chaoya Jiang. Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20036,2704,,22 2024.
[27] Liqiang Jing. Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du Faithscore: Evaluating hallucinations in large vision-language models. arXiv preprint arXtv:2311.01477, 2023.
[28] Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. Code: Contrasting self- eele-genertted description to combat ballucination in large multi-modal models. arXiv preprint arXiv:2406. 019200. 2024.
[29] Sicong Leng. Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vasion-language models through visual contrastive decoding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13872-
13882, 2024.
[30] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
[31] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Secd-bench: Benchmarking multimodal lims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- -magee pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[33] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ri-Rong Wen. Evaluating object ballucination in large languae models. The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URI httpttppp///ppeeeviieeeee net. teet:/fforummiixxozzooozzz
[34] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. 2021.
[35] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[36] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
[37] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22296-26306, 2024.
12

---

[38] Haotian Liu. Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen. and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024. /t io/ 01----0------lava
[39] Jiazhen Liu, Yuhan Fu, Ruohing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: A prompted visual hallucination evaluation dataset. arXiv preprint arXiv:2403. 11,, 2024.
[40] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in Ivims. arXiv preprint arXiv:240.21177, 2024.
[41] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo I., Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision. pages 216-233. Springer, 2025.
[42] Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object pres- ence evaluation (nope) to measure object hallucination in vision- saiiuule models. arXiv preprint arXiv:2310.05338, 2023.
[43] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 35:2507-25211 2022.
[44] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathwista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference Learning Representations (ICI.R), 2024.
[45] Xinyu Lyu, Beitao Chen, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Alleviating hallucinations in large vision- laiguale models through hallucination-induee optimization. arXiv preprint arXiv:2405.15356 2024.
[46] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozheh Mottaghi. Ok-vqu: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[47] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint wxXiv:2005.0066 00661, 2020.
[48] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
[49] Nasrin Mostafazadeh, Nathanall Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint 1 1604.01696, 2016.
[50] OpenAL. Igpt-4o, 2024.
[51] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
[52] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data only. Advances Neural Information Processing Systems, 36:79155 79172, 2023.
[53] Jiaye Qian, Ge Zheng, Yuchen Zhu, and Sibei Yang. Intervene-all Unified mitigation of Ivlm hallucinations across alignment formats. Advances in neural information processing systems, 2025.
[54] Vikas Raunak, Arul Menezes, and Marcin Junczys- Dowmunt. The curious case of hallucinations in neural machine translation. arXiv preprint arXiv:2104.0660 2021.
[55] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv: 08880000.500. 2018
[56] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-100 2021.
13

---

[57] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[58] Cheng Shi, Yizhou Yu, and Sibei Yang. Vision function layer in multimodal llms. Advances in neural information processing systems, 2025.
[59] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:23009. 14525, 2023.
[60] Craig Thomson and Ehud Reiter. A gold standard methodology for evaluating accuracy in data-to-text systems. arXiv preprint arXiv:2011.0333 1.03992, 2020.
[61] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1ms.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[62] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302.1399 13971, 2023.
[63] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue; A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804,07461, 2018.
[64] Junyang Wang. Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. Amber: An free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:231 100397,, 2023.
[65] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thihty- Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL httpa:/7
openrevieu net//forumiid--/TCcccaa
[66] Xintong Wang. Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision- language models with instruction contrastive decoding. arXiv preprint arXiv:2403. 2024.
[67] Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. Advances in Neural Information Processing Systems, 777-2012--20333 2024.
[68] Yue Xu, Chengyan Fu, Li Xiong. Sibei Yang, and Wenjie Wang. Auto- nuto-search and refinement: An automated framework for gender bias mitigation in large language models. arXiv preprint arXi::2502.1155, 2025.
[69] Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression comprehension. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4644-4653, 2019.
[70] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Yc, Bosheng Qin, Siliang Tang. Qi Tian, and Yucting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12944-12953, 2024.
[71] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint 23Xiv:3308.02490 2023.
[72] Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. Reflective instruction tuning: Mitigating hallucinations in large vision-languag models. arXiv preprint arXiv:2407.11222 2024.
[73] Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 135334 2023.
[74] Yulin Zhang, Cheng Shi, Yang Wang, and Sibei Yang. Eyes wide open: Ego proactive video-IIm for streaming video. arXiv preprint 1rXiv22510.15560 2025.
[75] Ge Zheng, Jiaye Qian, Jiajin Tang, and Sihci Yang. Why Ivims are more prone to hallucinations in longer responses: The role of context. International Conference on Computer Vision, 2025.
[76] Xiaoling Zhou, Mingjie Zhang, Zhemg Lee, Wei Ye, and Shikun Zhang. Hademif: Hallucination detection and mitigation in large language models. In The Thirteenth International Conference on Learning Representations.
14

---

[77] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXi::331000754. 2023.
[78] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv2304.10542. 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions and align well with the theoretical and experimental results presented the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper explicitly discusses key limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not contain any theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed descriptions of the experimental setup, data preprocessing, model architecture, and training procedures, ensuring that the main results can be reliably reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will be released after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee http:///ggu.eee/ooeSSuuuiiiisss Coooeooomisssssseeeeeeee www.siiiccciiiiiccoooooooiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The paper provides detailed experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The performance improvements are significant, and the paper provides detailed analyses to support our conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides resources details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::://eurrpps.ccomo /ttoe///gggttprr//pooooooooooooa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwparripp..ccoooeeeeeeeeeea /t/CCooffeeencss//2222 for what should or should be described.
22

---

A Appendix Overview
This appendix provides further details on the SCBench benchmark, experimental settings, and additional results to support the main paper. The contents are organized as follows:
Sec B
Details of SCBench Benchmark
Sec
B.1
Data Distributions
Sec B
Data Sources
Sec B.3
Details and Prompts for Data Construction
Sec C
Analysis Details and Supplementary Results
Sec
C.1
Analysis on various decoding strategies
Sec C22
Detailed settings for image masking and text insertion experiments
Sec
Detailed settings for logit lens analysis
Sec
D
Additional Experimental Settings and Results
- -Sec
D.1
Additional Implementation Details
Sec D.2
Additional Experiments on POPE
Sec
D.3
Additional Experiments MME Remaining Subset
Sec
Visualizations
Image Sources
Number Proportion
COCO
35
20
6.2%
18.0%
17.6%
MMBench
41
98
30.3%
MME 12
36
11.1%
ScienceQA
43
15
4.6%
Internet
124
38.4%
15.2%
Constructed
30
9.3%
18.6%
Problem Sources
Number Proportion
ScienceQA
15
4.6%
WinoGrande 56
7
2.2%
9.3%
MMLU 17
10
3.1%
9.3%
WSC 30
1.9% 6
Story 49
1.5% 5
Perception
Languago Capability
MNLI 63
7
2.2%
Science
Scene Understanding
QQP 6
1.5%
Commonsense Reasoning
Math
GPT-3.5 Generated
268
83.0%
Factual Knowledge
Table Data sources of SCBench Benchmark.
Figure 5: Data distribution of SCBench benchmark.
B Details of SCBench Benchmark
This section presents a detailed overview the SCBench benchmark, covering the dataset distribution, the specific data sources utilized, and the prompt design strategy employed during dataset construction. Visualizations of representative examples are provided Appendix
B.1 Data Distributions
We construct the SCBench benchmark, comprising 951 questions in total -23 compositional and 628 decomposed- curated from diverse perspectives. The distribution question types is visualized in Figure
The dataset primarily focuses on questions that involve both visual- and textual-centric decomposed questions, accounting for 82% of the total. To address distinct class of failures, we introduce
23

---

GPT-3.5 Prompt
Given fact about image, transform this fact into concise and relevant question, and provide corresponding answer. The question must explicitly include the word "image" and be appropriate the level of the fact (object, attribute, relation, or event).
Format your response strictly follows: Question: (Your generated question] Answer: [Your generated answer]
Below are several examples:
Object-level example:
Input Fact: There a dog the image.
Question: What is the animal in the image?
Answer: Dog.
Attribute-level example:
Input Fact: There a white dog the image. Question: What is the color of the animal in the image?
Answer: White.
Relation-level example:
Input Fact: The dog is lying on the bench.
Question: What is the relation between the dog and the bench?
Answer: The dog lying the bench.
Event-level example:
Input Fact: The dog is sleeping.
Question: What is the dog doing?
Answer: The dog sleeping.
Now, apply this format to the following input: Input Fact: (fact).
Table 8: Prompts used for visual-centric atomic question generation in the SCBench construction pipeline.
a Perception category. This category captures cases where the model correctly identifies relevant content but still fails to answer accurately when the information reformulated in MCQ format. These failures represent specific type of compositional challenge, in which additional textual choices hinder accurate comprehension. By including these examples in the perception split, we aim to improve the overall coverage of the benchmark. Additionally, we introduce Language Capability category, specifically designed evaluate models' abilities handle complex linguistic phenomena.
B.2
Data Sources
We provide the sources of the images and questions included in our benchmark, detailed in the Most questions are carefully constructed following the pipeline described in the main text. Only language capability questions and portion of science questions are adapted from existing NLP datasets [56 17
Table 7
63) and the ScienceQA
43 dataset, respectively.
B.3 Details and Prompts for Data Construction
Visual-centric atomic question construction. As introduced in the main text, first prompt popular LVLMs with diverse captioning instructions to identify commonly recognized content- such objects, attributes, relations, and events. Based this content, then construct visual-centric questions using the prompt template shown in Table
Textual-centric atomic question construction. Based on commonly recognized image content, we prompt GPT-3.5 to generate questions and options. For each category, we first obtain set of diverse perspectives using GPT-3.5 (c.g., typical animal behavior in Science) and formulate questions

---

GPT-3.5 Prompt
You imaginative and highly creative language model. Given the caption of image and a question related this image, your task is generate five correct answers and five incorrect answers for the given question.
Your answers should be realistic, logically sound, and plausible. Correct answers must accurately address the question, while incorrect answers should be clearly wrong misleading, yet still sound superficially plausible. The answers have grounded in the image caption, but may optionally relate to it.
Strietly follow the format below:
Example:
Image caption: The image shows dog lying on a bench at sunset.
Question: Which of the following is not typically a behavior exhibited by the animal in this image?
Correct answers:
1. Lying bench
Being very lazy
Writing with a pen
4. Using a litter box
5. Climbing trees
Incorrect answers:
Barking at strangers
2. Wagging their tails
3. Digging holes
4. Snifting around Herding sheep
Now, apply this format to the following input: Image caption:
(image caption)
Question: (question)
Table 9: Prompts used to generate answer options for textual-centric atomic questions in the SCBench construction pipeline. The input questions also generated using GPT-3.5 with simple prompts to provide diverse perspectives on the given visual content across different categories (e.g.. typical animal behavior in Science).
grounded in appropriate visual contexts (e.g.. Which of the following typically a behavior exhibited by the dog?). We then prompt GPT-3.5 to generate corresponding answer options, using the template shown Table
Exception on specific splits. For the language capability split, we select questions from NLP datasets whose answers can be visually represented. Then expressing answers through spatial references (e.g., "the image on the left/right" or "above/beloww") For the science split, adapt ScienceQA questions that originally solvable without images into versions that require visual information for correct reasoning.
concatenated images image input,
C Analysis Details and Supplementary Results
C.1 Analysis on various decoding strategies
As discussed in the main text, SCHall hallucinations are observed across a range benchmarks and models. Here, we present additional experiments across different decoding strategies. We conduct experiments using the LLa 1.5-7b (37 model and the results are shown in Figure 6 The results demonstrate that SCHall is observed consistently across all decoding strategies.
C.2 Detailed settings for image masking and text insertion experiments
Image masking. We use manually annotated images with masks. Specifically, we annotate bounding boxes that enclose the content necessary answer the question, and mask out all other regions. example is shown in the Figure (a).
25

---

Sample
Visual- Visual-certricc certric
Visual- canal-Cantric??
Visuall centric
Visual Visugl-ccntried col--cotric??
1.00%
v
-3.55% 8.8%
11..9%-5.4%
<
15.9%~6.88
45.2%
28.7%
X
41.2%
26.5%
X
37.8% 37.8%
*
55.5%
22.7%
50
x
X
Textual-centric?
Textual-centric?
Textual-cenne centric?
Textual-centir
40
Beam Search
Visual canmrci
Visual- centric?
Visual contrre
Visunl- cenulcccntiic
17.9%~7.55
20.6%-11.89 11.8%
20.6% 5.9%
1.3.2%~5.3%
x 41.5% 33.0
*
41.2%
26.5%
X
-3.22% 35.3%
44.7%
36.8%
x
x
x
x
Textuul--cetrrcc?
Textual-centur
T eextual-centric?
Textual-centrn
10
Greedy
Decoding
Visuel- certiie
Visual centric?
Vissal-ccattiie
Visunl-cantric? contriii?
16.8%-11.2%
15..6%-1.55%
2..7%-17.1%%
1000%%550%%
45.8% 26.2% %
*
53.1%
18.8%
X
31..4%
25.7%
x
52.55%
32.5%
Textual-centhe
Textual-centric?
Textual-centric,
Textual-centric?
Overall
MMBench
MME
MMVet
Figure Proportions of error attributed recognition and textual understanding failures across different decoding settings. When visual recognition is hallucination- free (the first line in each square), ballucinations occur more frequently in questions that have correctly answered text-centric sub-questions (top left corner) than in those with failed ones (top right corner). This phenomenon occurs consistently across sampling, beam search, and greedy decoding strategies on all datasets.
(a) Image Masking
(b) Text Insertion
Which type energy needed complete action the image? sound
Which following not typically behavior
exhibited by the animal this image?
A. Wagging their tails
B. mechanical
B. Being very lazy
heat
Barking
light
D. Snifling around
Barking
X
c C, heat
X
Which the following not typically behavior exhibited the animal in this image?
Look type of animal image. Ignore any other image information.
B. mechanical
A B. Being very luzy /
Figure 7: Examples for image masking and text insertion experiments with LLa
26

---

USER: What the object in the
image?
Mid Result
Token
ASSISTANT:
Tape
USER: What can the instrument in the picture measure?
A. The volume of a water cup
B. The mass of an apple
C. The temperature of the weather
D. The length desk
Selected Position
Answer question with option's letter.
c ASSISTANT: D. The length a desk
Figure 8: An example of logit lens analysis. The red-highlighted "tape" indicates the intermediate token trace. Underlined tokens mark the positions of interest that focus and visualize. Text insertion. We prepend a simple textual prompt, "Look at (image content). Ignore any other image information.", to explicitly highlight the relevant visual content. The placeholder (image content) is populated with text extracted from the question itself, as illustrated in Figure (b).
C.3 Detailed settings for logit lens analysis
Target tokens. We visualize types of tokens: the final answer token and the intermediate result token. The final answer token corresponds to the ground-truth answer (e.g.. if the answer D, track the probability of token D in the logit lens). The intermediate result token refers the token associated with the answer to the decomposed visual-centric sub-question. For example, in the question shown in Figure the intermediate result token is the first token of "Tape". This setup allows us to examine the model's reasoning trajectory, where the intermediate result token is expected appear earlier than the final answer token.
Layer dimension. To investigate how the probability of a target token evolves across layers, we compute its average probability over all input positions following the question prompt. For instance, the example shown in Figure & the selected range spans from "Answer" to "ASSISTANT:" This yields layer-wise trajectory of the target token's 's likelihood.
Position dimension. To analyze how the target token's 's probability changes across positions, we average its probability across all layers at each position. The visualized range also spans from "Answer" to "ASSISTANT." resulting in a position wise trajectory of the token's likelihood.
Settings
LLaVAv1.5-7b
Qwen-VL-Cha MiniGPT-4
01 a22 A3 batch size Ir
1, 1, le5
128
2c-4
lc-5
3e-5
Ir schedule
Cosine Decay
Ir warmup ratio weight decay epoch
0.03
0.01
0.05
0.05
0.05
optimizer
AdamW
DpepSpeed stage
Table 10: Hyperparameters for our VLR-distillation methods. a1, a2 and ag are the coefficients for Lregs D' and LKL, respectively.
D Detailed Experimental Settings and Results
D.1 Additional Implementation Details
In this section, we present the model-specific implementation details. For LLa 1.5-7 [37] we
utilize subset of its instruction-tuui -unnin datasets, specifically VQAv2 [13, OK- OK-VQA [46). GQA 23]
27

---

Dataset
Setting
Model
w/ ours
Accuracy?
Preciston
Recall
F1 Scoret
83.45(=0.48)
87 24
78.36
82.56
5
87.57 (=0.37)
85. 05.86, #0.40)
89.75 #0.50) (00.5)
20.37)
Random
Qwen-VL
86.67 (=0.48)
40.
79.16 (40.50)
85.59 (=0.53)
88.07 (40.2))
89. 13; 49.14
86. 72 (10.55)
87. 87.91 (1094
72.38 (=
75. 66
66. (2:1000))
70. 70.49; (00.49/20.00
MiniGPT-4
80.08 (20.88)
82 20.83)
75 75.91 (40.04)
79.21 (40.00)
79.90 (9.90(=0.88) (00.38)
78.36
79.59 (00.7)
LLaVA1.5
82.45 e00.00)
78. 20
89 99.91 (40.31)
83.68; (00.27)))
A A-OKVQA
Popular
Qwen-VI.
85. 55.56 L.0.3))
144,1..00
79.53 (10.84)
14.63,_1 (4.663,1.44 (10.42)
85,80 ..0.20)
85 28,
86 86
85.91,10.00
68 66 (0.38))
69.7 (=0.40)
66.00
67.80; 67.80
MiniGPT-4
75.45 (00688)
75.14;ee (4/400070
76.09 +0.70)
(5.66
LLVVAL
(22.34)
72. 2.0.00..
78. (10.9)))..3)) (10.38)
15,
75.06 (5.06,10.18)
69 69.24, 10..4
90.20 (10.50)
78.34
Adversarial
DL
79.57 (200331)
79.7 79.77;25 (2207))
79. (9.23
79. 79.50; (00.30))033))
78. 38 (20118)
74. 20. 20.24)
86.33 (b0.50)
79.97 (2011))
63.51 (=0.38)
63. (0
64.85
63.99(=0.20 (00.999(.02))
MiniGPT-4
70.97 (=0.24)
68. 80 19)
76.
(40.5)) (0.55)
72.55
83. (3.73))))27))
87. 37.16(=0.39)
(+0.85)
82.90;
LLaVAL
86.37 =00077)
84.86 84.86
88.58 (10.41)
86.68 (o0.18))
Random
Qwen-VT.
80.97 (11.332
88 (10.34)
71 64 (10.57)
79.01
87.11 (30.88)
80 10.40.49
(10.57)
86.66,10.41)
70.93 (00.5))
73. (3
66.21 (#0.00)
(90.6))
MiniGPT-4
80.24
82.96,
76.
79.39
LLaVALS
78.17 (100.17)
77 77..64..0.00
79. (0.35))
78.37
GQA
78.91 (10.48)
74 74.24 1...2
88 28.600 (10.86)
80.79 (1.055))
Popular
Qwen-VL
75.99 (0033))
78. 62, 40)
71.40 (1040 (20.58)
74..8 74.84 74.84;22034
81.26 (0.36)
79. 82, (0..5)
83 (20-9))
81.70; :0036)
65.96 (0.45))
(G5.767e 76
66.61 (+1.06)
66.18; (6018))0.50)
MiniGPT-4
74.40, =0.30)
73. (0..5)
75 (40.27)
74.78, (00.28)
75.08 (10000)
73. 100.49
79. 79.16
76. 76,00;220
LLaVA1.5
74.44 (=0.8)))))))
09. 199119:aa (9
88 20 (0.07)
77.55, 77.55(=00.3))
Adversarial
Qwen-VI.
75. (=0.63)
77. (7.92 73)
71.07 (+0.97)
74.33, (
79,41
77 04
83 (1 (10.13)
80.28; (10038)
MiniGPT-4
(=0.64)
62. 12.15
66.48 (6048( (+0.88)
64.24 24.24/20..6 (20.68))
70.60 (00288)
08.74/=0.20)
75.37 (+0.26)
71.99, (+0.2))
Table 11: Results on POPE, The best performances for baselines in each setup is highlighted in bolded.
w/ ours
Perception Total
Model
Posters
Celebrity
Scene
Landwark
Artwork
OCR
130.1443.7 100.06g1.52
144.35am.70
127. 227.70
73.00
99.50
674.7448.00
1LaVA155
136.12
116.0614.4
153.30
140.05
140.05
101.25
101.0018.1
747.78
148. 1943
117.79
158. 75.1.08
147. 42.
115.50
86,25
773.90
Qwen-VL
165.48
126.62
164.00
153.6322.01
129.50.2.77
87.50
826.72.29.aa
Table 12: Results all MME perception-related tasks. The best performance of each setting is bolded.
Common Sense
Numerical Calculation
Text Transtation
Code Reasoning
Recognition Total
Model
w/ ours
Reasoning
52. 26
s.5..
17. 17.50
11.00
164.36
LLaVAL5
/
97.71
57.50
74.00 29.50
68.00 10.90
297.21
Qwen-VL
122.7444.02
19.58_10.94
121. 121.25
73.75
367. 242.32223.4
126.79
58.75 28.75
139.17 16
76.67 1.677214.40
401.37 2.3773300000
Table 13: Results on all MME cognition-related tasks. bolded.
The best performance of each setting is
and OCRVQA 48 the training data. Similarly, Owen-VL-CCi 2 is trained using datasets that
28

---

include VQAv2, GQA, and OCRVQA. For MiniGPT-4 77), we adopt the same datasets used for LLaVAv1.5-7b. The corresponding hyperparameters are summarized in Table
D.2 Additional Experiments POPE
We present a comprehensive performance evaluation of our VLR-distillation method applied to POPE, Iwo additional datasets: A-OKVQA and GQA. As shown Table II our approach outper- forms the baselines across nearly all configurations, particularly when compared to the MiniGPT-4 baseline, with average improvement of 7.3% accuracy and 8.4% in FI score. Furthermore, we observe significant improvements for LLaVAvl 1.5 on A-OKVQA and for Qwen- VL on GQA, with average gains on accuracy of 2.6% and 12%, respectively.
D.3 Additional Experiments MME Remaining Subset
We evaluate the performance of our proposed method on the MME remaining set, with results presented in Tables 12 and [3] Table T2 shows the performance of the perception-related tasks, while Table T3 focuses on the cognition-related tasks. Our method consistently outperforms the three baseline approaches across both perception and cognition tasks. Notably, exhibits sig- nificant improvements in cognitive performance, which we attribute to its effective handling of SCHall-potential key factor influencing cenition-related tasks in the MME dataset.
E
Visualizations
We provide visualizations of our SCBench benchmark in Figure and demonstration of the effectiveness of our "LR--istillaatin in Figure 10
Specifically, we present representative samples for each category in our SCBench benchmark, as shown in Figure 9 can be observed that the images and questions our benchmark are not particularly challenging for current powerful LVLMs. However, these models still struggle to answer the questions. Besides, Figure[[Oilllssee illustraass the effectiveness of our method on each category in our SCBench, with each background color corresponding to a distinct category in the benchmark.
29

---

Commonsense Reasoning
Perception
nature, the species shown the picture not possess which following attrbuttes under realistic condition? Swimming the water large boat Having furry surface
Which on picture blue?
Cake
B. Flowerpot (v)
Cup
Plate
D. Being able to produce sound
Science
Factual Knowledge
do people turn after drinking the drink this image?
Which of the following events did not take
place in the city shown in this image?
A, Acelic add accumulates in the body Ethanol accumulates the body Ethylene accumulates
B. The Battle Waterloo in 1815. (V)
body Acetaldehyde acoumulates the body (V)
C. The opening of the Victoria and Albert Museum 1852. opening
1000 1003
Bridge
late 19th century.
Linguistic Capabillty
Math
Given two objects in the image, which object's name does the pronoun "r"" refer to? water
answer the
the backpack that would be handy.
arithmetic problem the image?
13 + 8
A. The left object. right object.
13
C. 21 (v)
C. Cannot seduced.
29
D. I don't know:
Understending
group people top right picture likely doing? Eating in an open-air McDonald's. s.
image,
B. Watching movie about pirates. (V)
Having fun Having class about dolphin.
Figure 9: Visualizations of questions in SCBench Benchmark. Our benchmark considers both visual- and textual-centric tasks which are likely to induce SCHall. The ground-truth answer for each question is indicated with
30

---

Purception
Which object
the smallest reality?
Lamp (V)
Chair
Sools
Window
Hlava
Commonsease Reasoning
point object object image?
Hoth lave the capacity to hold water.
Both are typically made of plastic. (v)
Roth are used for disposing wastc.
typically found
oUPS
llva
Science
Will the substance the solid, liquid, when heated to 70 degnees Celsius?
Liquid
Gas
None of the above
CX
Factual Knowledge
following NOT associated was originally created modicinal tenic the late century.
logo appears image?
The company that makes this crink is of the world's largest beverage compuny, offering over 500 brands in more than 200 countries.
introduced
glass bottl,
primunily composed carborsted water, sugar, caffeine, natural flavorings, giving it its distinct tret..
X
Linguistic Capabilitt
Given two objects the image, which objon's some does the pronoun "ege"! to?
map will show building: very object. (v')
The right object.
Cannut deduced
know.
X
Math
Each story a building is 14.3 fect tall. If the height of the building is 183. feet, how many floces does the building havee?
(.3))) C. L7
V
Tlava
Understanding
Which of the Bollowing reasoning is the least appropriase about this image?
Becmusc the riler' shudow the be soen in the pictare, the westher clear. coad, weather good(v))
Because
Because many
road, riding challenging. Because the pesson cures shout his safety, weasing helnet.
B
OuEs
lava
Figure 10: Visualizations of the effectiveness of our VLR-distilllttion method in SCBench Benchmark.
31


Reviews:
[{"review_id": "9FHqe3Eqmb", "reviewer": "Reviewer_r87F", "summary": "This work identifies and analyzes a fundamental hallucination phenomenon in multimodal large models, termed SCHall. To address this, this work introduce the VLComp benchmark for evaluating SCHall and conduct a comprehensive assessment of existing models and mitigation techniques, revealing their significant limitations. Based on the findings, this work propose VLR-distillation, which achieves substantial improvements across VLComp, multiple hallucination benchmarks, and general VQA tasks.", "questions": "1. What is the relationship between the evaluation angle proposed in the article and the multimodal alignment problem? It is natural for the model to have better accuracy on a single mode, so are there other angles and views on the combined modal challenge proposed in the article compared to multimodal alignment? From Figure 1, the effect of simple-text is lower than that of simple-visual. Is there a corresponding explanation?\n\n2. From another perspective, does the solution proposed in the article make the multimodal alignment of mllms perform better? I still want to ask the author to introduce the motivations and effects of the three stages in the method part.", "limitations": "- Benchmark:\n1. The appendix or the main text lacks a more detailed description of the statistical information of the dataset, as well as the evaluation process and indicators\n2. The evaluation on the current mainstream open source and closed source models or the evaluation by human experts to verify the rationality of the benchmark\n\n- Method:\n1. The motivation description of the three stages proposed\n2. Experiments on more types and sizes of open source models to verify the universality of the method\n\nIf the author can solve most of the above problems, I am willing to reconsider the contribution of the article", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n- The evaluation angle of MLLMs proposed in the article is novel\n-  This work proposes an effective solution and baseline based on the problem\n- The description of motivation and conclusion is very clear, and the writing is very logical\n\nWeakness:\n- The statistical information and construction process of the benchmark need to be introduced in more detail (with the help of charts, etc.)\n- There is a lack of evaluation on the current mainstream mllms (including open source and closed source models) to verify the rationality of the motivation and the quality of the benchmark\n- In the method part, the mllms used is a bit outdated", "was_revised": true, "final_justification": "- The author has detailed additional statistical analysis experiments for the benchmark.\n- The author also supplemented the analysis experiments for the currently mainstream closed-source MLM framework, and obtained conclusions consistent with those in the article.\n- The updated methodology includes the latest Qwen2.5-VL model and detailed evaluation experiments.\n- The article addresses most of the questions and writing issues.\n\nThe article provides a novel perspective on constructing an MLM benchmark, providing the community with new conclusions and findings.", "rebuttal": "Thank you for the valuable feedbacks and suggestions. Our response is as follows: \n\n> **W1 & Limitations 1.1:** Statistical information and construction process of the benchmark.\n> \n\nWe appreciate your suggestion to provide more detailed information about the statistical properties and construction process of our benchmark.\n\n1. **Statistical Information**:\n    \n    We provided detailed statistical information in the supplementary materials (Appendix C.1 and C.2), including data sources, data volume (951 total questions with 323 compositional and 628 decomposed), and category distribution (18% Perception, 15.2% Science, 12.1% Commonsense Reasoning, 9.3% Factual Knowledge, 9.3% Language Capability, 18.6% Scene Understanding and 17.6% Math). We will include more details (e.g. sub-problems distributions) and make sure to reference these clearly in the main text for better accessibility. \n    \n2. **Benchmark Construction Process**:\n    \n    We agree that a clearer explanation of the benchmark construction process would improve the clarity of the paper. Our benchmark construction process includes two stages: Atomic Questions Generation and Simple-Composed Question Generation. \n    \n    **Atomic Question Generation**\n    \n    - **Image Collection**: Images are collected from datasets (MMBench, MME, SEEDBench) and online sources.\n    - **Caption Generation**: LVLMs (LLaVA, Qwen-VL, MiniGPT-4, InternVL) generate captions for each image.\n    - **Visual-centric Question Formulation:** GPT-3.5 generates recognition questions from captions, e.g., \"What animal is shown in the image?\"\n    - **Textual-centric Question Formulation:** GPT-3.5 generates diverse questions related to the visual content, e.g., \"What is typical behavior of the rabbit?\" along with answer options grounded in the visual context.\n    - **Evaluation and Filtering**: Questions are filtered based on LVLM performance, retaining only \"easy\" questions.\n    \n    **Simple-Composed Question Generation**\n    \n    - **Image Substitution**: Textual-centric questions are converted into Simple-Composed Questions by replacing text with images.\n    - **Filtering and Refinement**: Difficult questions are identified through LVLM performance and manually refined for the final benchmark.\n    \n    We will include a **visual pipeline diagram** to make this process more transparent and easier to understand. \n    \n\n> **W2 & Limitations 1.2:** Evaluation on the current mainstream mllms.\n> \n\nWe include evaluations on several mainstream models, including open-source (Qwen2VL) and closed-source models (GPT-4.1, Gemini 2.0, Flash). As shown in the table below, these models demonstrate high accuracy on both visual- and textual-centric questions individually. However, when combined, they exhibit noticeable drop in performance (over 10%) compared to the individual task accuracies.\n\nThis performance drop actually supports our motivation and quality of benchmark. Current mainstream MLLMs still show room for improvement on our VLComp benchmark, highlighting the quality and challenge of our benchmark.\n\nTo further validate our benchmark, we also evaluated three human experts (with access to search engines) and report their average score. Their consistently high performance demonstrates that the benchmark is solvable by humans and confirms its clarity and diagnostic value.\n\n|  | Visual-centric questions Acc | Textual-centric questions Acc | Compositional questions Acc | $\\Delta$ Performance Drop |\n| --- | --- | --- | --- | --- |\n| Human Expert | 100.0 | 95.1 | 94.1 | 1.0 |\n| *Closed Source Models* |  |  |  |  |\n| GPT-4o | 95.0 | 93.8 | 79.0 | 14.8 |\n| Gemini 2.0, Flash | 97.3 | 90.4 | 80.0 | 10.4 |\n| *Open Source Models* |  |  |  |  |\n| Qwen2VL | 96.4 | 78.1 | 67.3 | 10.8 |\n\n> **W3 & Limitations 2.2:** In the method part, the mllms used is a bit outdated\n> \n\nThank you for the suggestion. We apply our method on mainstream Qwen2VL and evaluate them on VLComp. The results are as follows:\n\n|  | Perception | Science | Commonsense | Factual | Text | Scene | Math | Overall |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Qwen2VL | 74.5 | 71.7 | 62.5 | 69.0 | 60.7 | 63.9 | 66.7 | 67.3 |\n| + ours | 76.4 | 73.9 | 67.5 | 69.0 | 64.3 | 66.7 | 68.5 | 69.8 |\n\nThe results indicate that our method also achieves the improvement on the mainstream baseline, which exhibits fewer hallucinations. Due to time constraints, we use only a very limited amount of training data from QwenVL (previous version in Qwen Family).\n\n> **Q1.1 & 2.1:** Discussion with multimodal alignment problem\n> \n\nThank you for the question. SCHall arise not only from insufficient multimodal alignment, but also from failures in compositionality.  To further probe the role of modality alignment, we conduct an additional experiment using Qwen2VL. Specifically, we first prompt the model to generate a caption for the image alone, then combine this caption with the original textual question for inference. Interestingly, the model’s performance drop from **67.3% to 62.2%.** Among these, 8.3% of cases show improvement, indicating that alignment is indeed a bottleneck. However, the majority of new errors fall into two categories:  (1) 45.5% where the caption lack question-relevant visual details (reflecting visual abstraction failures, as discussed in Section 3.2), and (2) 54.5% where the caption is informative, but reasoning still failed—highlighting limits in compositional reasoning. These findings support our claim that **multimodal alignment, while necessary, is not sufficient to resolve SCHall cases**. Visual abstraction and other underlying factors remain critical challenges.\n\nOur proposed method also contributes to improving alignment. Among the 8.3% alignment-limited cases, our method improves 22%, demonstrating its effectiveness in addressing this subset of failures.\n\n> **Q1.2:** From Figure 1, the effect of simple-text is lower than that of simple-visual. Is there a corresponding explanation?\n> \n\nSince most existing hallucination benchmarks focus on visual recognition, we intentionally craft visually simple questions to minimize the impact of recognition errors and better diagnose SCHall. \n\n> **Q2.2 & Limitations 2.1:** Motivations and effects of the training stages in the method part.\n> \n\nThe motivations and effects of two stages is as below:\n\n**Training Stage 1: VLR pretraining.** **Motivation:** As discussed in Section 3.2, a key contributor to SCHall is the failure of visual abstraction. To address this, we introduce explicit VLRs to encourage the model to abstract visual information effectively. Specifically, we pretrain the model by blocking access from the output tokens to the image, encouraging it to rely entirely on VLRs for extracting and transforming key information. **Effect:** This pretraining improves the model’s ability to extract and represent visual information accurately. However, it does not yet equip the model to jointly reason with images and VLRs, which is addressed in subsequent training stages.\n\n**Training Stage 2: Distillation learning.** **Motivation:** As shown in Section 3.3, the presence of visual inputs can degrade the model’s language processing ability. To address this issue, we distill language reasoning knowledge from a branch that uses only image captions to guide the main branch, which jointly reasons over images and VLRs. **Effect:** This stage improves the model’s language understanding in multimodal settings and encourages more effective use of VLRs.\n\nThank you for the feedback. We will make the motivations and effects of each training stage clearer in future revisions of the paper."}, {"review_id": "yNXJ3tbcgf", "reviewer": "Reviewer_cKHX", "summary": "This paper investigates a previously underexplored type of hallucination in large vision-language models (LVLMs), termed Simple Compositional Hallucination (SCHall). This form of compositional questioning consistently triggers hallucinations, despite the models achieving high accuracy on the decomposed sub-questions when handled independently.\n\nTo systematically study this phenomenon, the authors construct a new benchmark called  VLComp, which comprises triplets of questions: vision-only, text-only, and their compositional combination. They demonstrate that SCHall is a widespread issue and becomes more pronounced in models with stronger visual recognition capabilities. \n\nTo mitigate SCHall, the authors propose VLR-Distillation, a novel framework that introduces Vision Language Registers (VLRs) along with a textual distillation training objective, aiming to preserve language modeling capacity under visual conditions. Experimental results show consistent improvements over several strong baselines on the proposed VLComp benchmark, as well as on other hallucination or general-purpose VQA datasets.", "questions": "1. Which model is applied for captioner?\n2. In the ablation study, how many VLRs were used in the with/without DL (distillation learning) settings? Could increasing the number of VLRs (e.g., setting it to 8) achieve similar performance to using distillation learning?\n3. In the ablation study, why does increasing the number of VLRs lead to degraded performance?\n4. What are the key differences between your VLRs and LLaVA-Mini[1]?\n\n\n[1] Zhang, Shaolei, et al. LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token. arXiv:2501.03895, arXiv, 2 Mar. 2025. arXiv.org, https://doi.org/10.48550/arXiv.2501.03895.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper is clearly written and well-structured. The motivation is well-explained with illustrative examples (e.g., Fig. 1), and the experimental setup is carefully detailed. \n2. While the paper provides solid empirical evidence, it lacks a deeper theoretical analysis or formalization of why compositionality causes hallucination at the model architecture or optimization level. The explanations are plausible but remain largely observational.\n3. The paper addresses a previously unrecognized and important phenomenon—Simple Compositional Hallucination (SCHall)—that affects the reliability of modern vision-language models. Given the increasing deployment of LVLMs in real-world scenarios, understanding and mitigating subtle failure modes like SCHall is of practical and theoretical significance.\n4. The identification and formalization of SCHall is novel. The idea that hallucinations can emerge only when combining independently “easy” visual and textual components is counterintuitive and has not been highlighted in prior work. The proposed VLComp benchmark also offers a novel way to evaluate this specific type of hallucination, distinct from existing hallucination datasets.\n\nWeaknesses\n1. There are some writing mistakes, such as a formula error in the causal attention mask in Section 4.1.\n2. While the paper provides solid empirical evidence, it lacks a deeper theoretical analysis or formalization of why compositionality causes hallucination at the model architecture or optimization level. The explanations are plausible but remain largely observational.", "was_revised": true, "final_justification": "The paper seems novel but still lack of theoretical analysis. I will keep the rating.", "rebuttal": "We thank the reviewer for the insightful feedbacks and questions. Our response is as follows:\n\n> **W1:** Writing mistakes in Section 4.1\n> \n\nThank you for pointing out the formatting issues. We will ensure that the correct condition $s \\leq r$ is clearly stated and carefully revise the formatting in the final version.\n\n> **W2:** Lacks of deeper theoretical analysis\n> \n\nIndeed, our analysis of SCHall is primarily empirical. We perform error analysis on both VLComp and general benchmarks to reveal its widespread presence, and use experiments and visualizations for preliminary attribution. The competitive performance of our method on VLComp further substantiates our analysis.\n\nWe acknowledge the need for deeper theoretical analysis of why compositionality leads to hallucination at the model architecture and optimization level. Our hypothesis is that the training paradim, which supervises only the final textual output without intermediate guidance, leads the model to rely on spurious correlations and dataset biases instead of grounded compositional reasoning, thereby increasing SCHall risk. In response to this suggestion, we plan to conduct further analyses, including intermediate representation probing with logit lens and intervention validation via attention transfer.\n\n> **Q1-3:** Experimental Setting Details\n> \n\nThank you for your question. During training, in cases where ground-truth captions are not available, we employ the baseline model itself as the caption generator. In the ablation study (left table of Table 5), we used 4 VLRs for both the with and without distillation learning (DL) settings. We further conduct additional experiments by increasing the number of VLRs without applying distillation learning (DL). The results are shown below:\n\n| #VLRs | Random Acc | Random F1 | Popular Acc | Popular F1 | Adversarial Acc | Adversarial F1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| 4 | 86.5 | 85.9 | 84.1 | 83.7 | 80.2 | 80.4 |\n| 8 | 86.7 | 86.4 | 83.5 | 83.5 | 79.3 | 80.1 |\n| 16 | 85.0 | 84.9 | 82.0 | 82.4 | 78.4 | 79.7 |\n\nThese results indicate that simply increasing the number of VLRs does not match the performance achieved when combining VLRs and DL. Notably, performance even degrades when using 16 VLRs. This may be related to the nature of the POPE benchmark, where 4 tokens are sufficient to capture the necessary visual information. Adding more VLRs may introduce redundancy and noise, which could lead to hallucinations—this also addresses the concern raised in Q3.\n\n> **Q4:** Key differences with LLaVA-Mini\n> \n\nThank you for your question. LLaVA-Mini employs **explicitly designed fusion and compression modules** to generate compressed and **pre-fusion tokens**. In contrast, our approach introduces VLRs, a novel token type that operates entirely **within the LVLM,** enabling it to autonomously learn to extract question-relevant image information while also engaging in textual understanding. Specifcally, the key differences are as follows: \n\n- Our VLRs are independent tokens that bridge vision and language, whereas LLaVA-Mini uses pre-fusion tokens to replace text tokens.\n- The multimodal interaction in our method is conditioned on the LVLM’s rich internal knowledge, while LLaVA-Mini performs a simpler fusion of modalities. By leveraging this internal knowledge, VLRs achieve more flexible and semantically meaningful multimodal representations."}, {"review_id": "vvGWASLq6C", "reviewer": "Reviewer_4hUj", "summary": "The paper introduces Simple Compositional Hallucination (SCHall) — a novel and subtle failure mode in large vision-language models (LVLMs). SCHall arises not from visual or textual input alone, but from their composition, where individually “easy” visual and textual questions—answered correctly in isolation—lead to hallucinated answers when combined.\n\nTo systematically study this issue, the authors build a benchmark called VLComp, composed of question triplets: simple visual, simple textual, and their compositional form. Surprisingly, strong LVLMs like GPT-4o, InternVL, and LLaVA exhibit up to a 20% drop in accuracy when moving from isolated to compositional questions.\n\nThe authors identify two contributing factors: (1) failure in abstracting and aligning relevant visual content during composition, and (2) degradation in language processing performance when visual inputs are present. To address this, they propose VLR-distillation, a novel training strategy that incorporates Vision Language Registers (VLRs) and a distillation branch to enhance visual grounding and preserve linguistic reasoning.\n\nExtensive experiments show their method significantly reduces SCHall and improves performance across multiple hallucination and VQA benchmarks, demonstrating the generalizability of their findings.", "questions": "1. What types of compositional hallucinations are most common, and how do different models fail in different ways?\n2. You attribute SCHall to failures in visual abstraction and degradation of language processing. However, how do you distinguish SCHall from general shortcut learning or heuristic answering strategies that might also cause similar issues?\n3. My other questions are mentioned in the weakness. I am willing to raise my score if the authors can answer my questions.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. The paper identifies and formalizes a new type of hallucination in LVLMs—Simple Compositional Hallucination (SCHall)—which occurs even when both visual and textual components are individually handled correctly. This is a subtle, underexplored, and practically relevant failure mode that has not been systematically studied before.\n\n2. The authors introduce VLComp, a carefully curated benchmark that tests LVLMs' robustness to compositional queries. The benchmark is well-constructed using a mix of automatic generation and human filtering, with a diverse set of visual and textual question types.\n\n3. The paper presents compelling evidence that stronger models (e.g., GPT-4o, Qwen-VL) exhibit more pronounced SCHall, implying that current progress on visual recognition alone is insufficient.\n\n4. The proposed Vision Language Registers (VLRs) and distillation learning strategy are elegant, practically implementable solutions.\n\n**Weakness**\n\n1. While VLComp is well-motivated, it is relatively small and semi-automatically constructed, which may raise questions about generalizability and annotation bias.\n\n2. While VLR-distillation performs well, the paper provides limited comparative insight into why existing mitigation strategies (like REVERIE or PAI) fail in this specific scenario. A qualitative failure case analysis or sensitivity study could enhance interpretability.\n\n3. Can the author evaluate the method on more Hallucination evaluation benchmark like HallusionBench? What about other VQA benchmarks like MMMU, MathVista?", "was_revised": true, "final_justification": "My concerns are addressed so I will maintain my positive score.", "rebuttal": "We thank the reviewer for the insightful feedbacks and questions. Our response is as follows:\n\n> **W1:** Concerns about generalizability and annotation bias.\n> \n\nThank you for your feedback. The generalizability and quality of VLComp can be validated from the following aspects: \n\n- VLComp is specifically designed to diagnose and analyze SCHall. Inspired by existing hallucination evaluation benchmarks for LVLMs—such as LLaVA-Bench-in-the-Wild, MMHal-Bench, and MM-Vet, which comprise approximately 60, 96, and 128 samples, respectively—VLComp significantly expands both scale (323 samples with 951 compositional and decomposed questions in total) and scope. Importantly, it ensures strong diversity across key categories, including perception, science, commonsense reasoning, factual knowledge, linguistic capability, scene understanding, and mathematics.\n- While VLComp is relatively small than more general-purpose LVLM benchmarks, our experiments show that it effectively identifies and supports exploration of this novel form of hallucination. **More importantly, SCHall also emerge in general-purpose benchmarks** (Figure2), suggesting that our benchmark captures a representative and generalizable issue.\n- In addition, the semi-automated process for constructing VLComp involves minimal manual annotation, primarily limited to reviewing and refining the questions. This design allows for efficient and scalable expansion. In future work, we plan to extend VLComp with broader coverage and increased human involvement to further enhance its reliability and generality.\n\n> **W2:** Interpretation on failure of existing hallucination mitigation strategies on SCHall.\n> \n\nThank you for the insightful comment. The ineffectiveness of existing mitigation strategies in addressing SCHall can be attributed to two primary factors: \n\n1. While methods like PAI are effective at mitigating object hallucination and improving recognition accuracy, they are less suited to the SCHall setting, which primarily involves compositional reasoning rather than recognition failures. For instance, in Figure 5 (Math) of the Appendix, the image displays the equation “13 + 8,” which the baseline model answers correctly. However, PAI misleads the model to predict “53”—a partially visually plausible (due to the “3”) but semantically incorrect answer. This highlights that the difficulty in SCHall stems more from compositional challenge than from visual recognition errors.\n2. Meanwhile, REVERIE attributes hallucinations to a lack of fine-grained supervision and improves reasoning via Reflective Instruction Tuning. However, it does not explicitly teach selective integration of visual and textual cues. For instance, as illustrated in Figure 3(a) in Appendix, where a person is clearly chopping wood at sunset, the baseline model incorrectly predict \"thermal energy\" based on the visual salience of sunlight. REVERIE fails to correct this, likely because both the ground-truth and hallucinated answers are visually plausible.\n\nWe appreciate the suggestion and will include additional qualitative failure cases and comparative analyses for existing methods in the final version to enhance interpretability.\n\n> **W3:** More evaluation results on hallucination and other VQA benchmarks.\n> \n\nThank you for your suggestion! We evaluate our method with LLaVA-1.5-7B on additional hallucination benchmarks, including HallusionBench, as well as general-purpose VQA datasets, including MMBench, ScienceQA, and MM-Vet. Results are shown below. \n\n|  | MMBench | ScienceQA | MM-Vet | Hallusion Bench |\n| --- | --- | --- | --- | --- |\n| LLaVA-v1.5-7b | 64.3 | 66.8 | 31.1 | 47.6 |\n| + ours | 65.4 | 67.8 | 33.3 | 49.5 |\n\nOur method achieves consistent improvements across multiple benchmarks, demonstrating both the generality of SCHall and the effectiveness of our approach.\n\n> **Q1:** Types of compositional hallucinations and how do models fail in them?\n> \n\nThank you for your valuable suggestion! We identify several common types of **compositional hallucinations** observed in our VLComp:\n\n1. **Irrelevant Visual Feature Attribution.** When an image contains rich or distracting content, the model may attend to semantically irrelevant visual regions, leading to incorrect reasoning. For instance, in Figure 3a in Appendix, the image shows a person chopping wood at sunset, while the question concerns the type of energy consumed during the activity. Earlier models (e.g., LLaVA) are often misled by salient but contextually unrelated elements (e.g., the sunset), resulting in hallucinated answers.\n2. **Superficial Visual-Linguistic Mapping.** When there is superficial alignment between visual content and candidate answers, models may rely on pattern matching instead of grounded reasoning. \nIn Figure 3b, a dog is resting on a chair. The question asks about *typical behaviors* of the animal shown. Most open-source models mistakenly select “being very lazy” based on the dog’s current state, rather than reasoning about its typical behavior in general.\n3. **Cross-modal Misalignment.** In some cases, replacing the image with its caption leads to correct answers, indicating poor alignment between visual and textual modalities. In Figure 4 in appendix, the image contains a clearly visible ruler, and the question asks what it can be used to measure. While caption-based inputs (e.g., “A ruler is shown”) yield correct responses, the same models fail with the full visual input. Such failures on simple and unambiguous images are typically observed only in earlier models (e.g., LLaVA).\n\nNevertheless, the interplay of these challenges often leads to failure cases that challenge even close-source models. There remain many types of compositional hallucinations that warrant deeper investigation. We will incorporate the above discussion and related analyses in the final version.\n\n> **Q2:** Discussion with general shortcut learning or heuristic answering strategies.\n> \n\nWe thank the reviewer for the thoughtful question. We agree that shortcut learning and heuristic answering are important considerations and likely contribute to the observed SCHall behaviors, particularly those related to *Superficial Visual-Linguistic Mapping* (see response to Q1).\n\nHowever, we argue that SCHall extends beyond these mechanisms. In addition to superficial patterns, SCHall captures deeper model limitations such as *Irrelevant Visual Feature Attribution* and *Cross-modal Misalignment.* These cannot be fully accounted for by shortcut learning alone. We view our analysis as an initial step, and we welcome future work that further disentangles these contributing factors."}, {"review_id": "qfGB3pT8mL", "reviewer": "Reviewer_riVB", "summary": "This paper introduces a hallucination benchmark, Simple Compositional Hallucination (SCHall), constructed from the compositional structures of text- and vision-centric questions. Through a comprehensive analysis of the challenges posed by compositional hallucinations, the authors further propose a method called VLR-Distillation, which distills knowledge from a large language model (LLM) augmented with additional image captions. Experimental results on both hallucination benchmarks and general visual question answering (VQA) benchmarks demonstrate the effectiveness of the proposed approach.", "questions": "N/A", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper presents a thoughtful analysis of the challenges and underlying causes of compositional hallucination in multimodal reasoning tasks.\n2. The proposed method is thoroughly evaluated on both hallucination-specific and general VQA benchmarks, demonstrating its effectiveness and versatility.\n\nWeakness\n1. The distinction between the proposed SCHall benchmark and existing common-sense knowledge-based VQA benchmarks (e.g., OK-VQA, AOK-VQA) remains unclear. Based on Fig. 1 and the examples provided in the Appendix, the proposed benchmark appears largely similar to previous ones. As such, the use of the term hallucination may come across as more of a rebranding than a substantive contribution.\n2. The scale of the SCHall benchmark is relatively limited, raising concerns about its representativeness and generalizability.\n3. The paper’s writing and logical flow require improvement to enhance readability and clearly convey the motivation and contributions.", "was_revised": true, "final_justification": "I have carefully read the authors’ rebuttal. However, I still do not clearly understand the key differences between the proposed benchmark and existing commonsense-based VQA datasets, as both require models to extract information from the input image and question. Moreover, the distinction between hallucination and traditional VQA errors remains unclear, making it difficult to understand why this should be considered a hallucination benchmark. So I will keep my score.", "rebuttal": "We thank the reviewer for the feedbacks and questions. Our response is as follows:\n\n> **W1:** Distinction between the proposed benchmark and existing VQA benchmarks\n> \n\nWe would like to clarify that our proposed VLComp benchmark differs fundamentally from existing knowledge-based VQA benchmarks. As stated in Lines 56–59 in the main paper, while challenges related to **textual knowledge** can arise in these VQA benchmarks, VLComp focuses on questions with **simpler textual sub-questions**, with its primary challenge being ***compositional hallucination*.**\nTo further emphasize this distinction, we present the following comparison table as follows:\n\n|Benchmark|QuestionType|Visual|||Lingual|Easy Visual|Easy Lingual|Compositional|\n| ------------- | ------------- | ------ | --------- | -------- | ------- | --- | --- | --- |\n|||Object|Attribute|Relation|||||\n|OK-VQA|VQA|✔|✔|✔|✔|✘|✘|✔|\n|A-OKVQA|VQA,MCQ|✔|✔|✔|✔|✘|✘|✔|\n|POPE|Y/N|✔|✘|✘|✘|✘|✘|✘|\n|MME|Y/N|✔|✔|✘|✘|✘|✘|✘|\n|CHAIR|Open|✔|✘|✘|✘|✘|✘|✘|\n|AMBER|Open,Y/N|✔|✔|✔|✘|✘|✘|✘|\n|MMHal-Bench|VQA|✔|✘|✘|✘|✘|✘|✘|\n|HaluEval|QA,Short|✘|✘|✘|✔|✘|✘|✘|\n|DiaHalu|Short|✘|✘|✘|✔|✘|✘|✘|\n|VLHall(Ours)|MCQ|✔|✔|✔|✔|✔|✔|✔|\n\n\n> **W2:** Concerns about VLComp scale.\n> \n\nThe representativeness and generalizability of VLComp have been validated from the following aspects: \n\n- VLComp is specifically designed to diagnose and analyze SCHall. Inspired by existing hallucination evaluation benchmarks for LVLMs—such as LLaVA-Bench-in-the-Wild, MMHal-Bench, and MM-Vet, which comprise approximately 60, 96, and 128 samples, respectively—VLComp significantly expands both scale (323 samples with 951 compositional and decomposed questions in total) and scope. Importantly, it ensures strong diversity across key categories, including perception, science, commonsense reasoning, factual knowledge, linguistic capability, scene understanding, and mathematics.\n- While VLComp is relatively small than more general-purpose LVLM benchmarks, our experiments show that it effectively identifies and supports exploration of this novel form of hallucination. **More importantly, SCHall also emerge in general-purpose benchmarks (Figure2)**, suggesting that our benchmark captures a representative and generalizable issue.\n- In addition, the semi-automated process for constructing VLComp involves minimal manual annotation, primarily limited to reviewing and refining the questions. This design allows for efficient and scalable expansion. In future work, we plan to extend VLComp with broader coverage and increased human involvement to further enhance its reliability and generality.\n\n> **W3:** Concerns on writing and logical flow.\n> \n\nWe would like to clarify that our paper is structured to follow a logical progression centered around a concrete motivation: the discovery of a previously underexplored hallucination type in LVLMs, which we term SCHall (Simple Compositional Hallucination). The paper is organized as follows:\n\n1. We define SCHall and motivate its importance based on empirical observations (Introduction & Section 3.1).\n2. We construct VLComp, a targeted benchmark to systematically study this phenomenon (Section 2).\n3. We analyze its underlying causes, identifying two contributing factors: (Section 3.2 & 3.3)\n(i) difficulty with visual abstraction, and\n(ii) interference of visual input with language reasoning.\n4. We then propose VLR-distillation, a mitigation method informed by our analysis (Section 4).\n\nIn the final version, we will improve the transitions between sections to better highlight the logical flow and the relationship between sections."}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field. The paper received mixed reviews, i.e., 4 Borderline Accept, 4 Borderline Accept, 3 Borderline reject, 6 Strong Accept. 
After the discussion, the reviewers still have some concerns regarding the paper, such as
(a) The key differences between the proposed benchmark and existing commonsense-based VQA datasets.
(b) Not clear on the distinction between hallucination and traditional VQA errors.
On the other hand, the reviewers acknowledged the paper for its strengths in 
(1) The paper is clearly written and well-structured.
(2) The paper addresses a previously unrecognized and important phenomenon—Simple Compositional Hallucination (SCHall).
(3) The identification and formalization of SCHall is novel.
(4) The proposed Vision Language Registers (VLRs) and distillation learning strategy are elegant, practically implementable solutions.
Based on these positive reviews, the decision was made to recommend it for acceptance. We congratulate the authors on their acceptance!
Besides, authors should revise the paper taking into account the reviewers' comments.
Decision: Accept (poster)