Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
Authors: Jiaye Qian
Ge Zheng
Yuchen Zhu
Sibei Yang
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Language Models
Hallucinations
Primary area: applications

Abstract: Despite their impressive performance across a wide range of tasks, Large Vision-Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer’s causal architecture in LVLMs, integrating the effects of different intervention paths on hallucination. We find that hallucinations in LVLMs do not arise from a single causal path, but rather from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question–answer alignment format. Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.

Intervene-All-Path:: Unified Mitigation of LVLM Hallucinations across Alignment Formats
Jiaye Qian 1,2+ 1.2+
Ge Zheng
Yuchen Zhu?
Sibei Yang"
School of Computer Science and Engineering, Sun Yat-sen University
"ShanghaiTech University
Project Page: https://github.ccm htth://hhho.ccccbbllaaaaaaalhhha
Abstract
Despite their impressive performance across a wide range of tasks, Large Vision- Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer's causal architecture LVLMs, integrating the effects of different intervention paths hallucination. We find that hallucinations in LVLMs do not arise from single causal path, but rather from the interplay among image-to-input-tex, image-to- output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question- answer alignment format. Building these insights, propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.
Introduction
Large Vision Languae Models (LVLMs)
36
37
3
4
86
6
8
73 78 have emerged as foun-
dational models 13
11
51
87
13
for general purpose visual understanding [67 [6] 68 69 70
84 71
32
52
21
and reasoning 48
40
56
64
57 81, demonstrating strong capabilities across 50
31 34 [2]. They are able to handle diverse queries to open-ended captions and dialogues. However, despite these advances, LVLMs often suffer from hallucination, generating textual outputs that are inconsistent with the visual input, such as images [30 80 23
broad range of open-world world tasks 19
48
58
question formats [5) from binary judgments and multiple-choie
g 41 This
issue underscores a fundamental limitation LVLMS and raises concerns about user trust and potential for misinformation. Therefore, tackling hallucination has become a growing research focus, with increasing efforts to uncover its causes and mitigate its impact to enhance the reliability of LVLMs.
The causes of hallucination LVLMs are inherently more complex due to their foundation on text-only LLMs. Broadly, hallucinations may be inherited from the underlying LLMs or stem from vision-specific components and training. More specifically, several studies have examined the con- crete factors and proposed corresponding mitigation techniques. Contrastive decoding attributes hallucination to over-reliance on statistics and unimodal language priors, mitigating it by calibrating LVLM output distributions via comparisons with logits from distorted visual inputs 28 27 61]. Intervention based techniques diagnose internal attention weights or heads in LVLMs and directly intervene, enabling correction during inference with single forward pass and no additional compu-
tational cost 33 72 Among them, weight intervention [39 83 enhances visual grounding by
"Equal contribution. Work done during Jiaye's intersship SYSU, Conresponding author is Sibei Yang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Intervention Frameworks
MOQ-POPE
Vanilla
Canurl Atlention Mask
VCD
Image
IX- Input Text
X Ouiput Text
ICD
Image
AD-HH:
Image
Input Text
Output Text
AD&HH
CHAIR 7666
POPE
Iaput Text
Ours
88
Image
Input Text
Output Text
Ours!
Ourput
68
MME
(h)
Figure I: Leftt: Our AllPath intervention frameowrk comprehensively mitigates hallucinations from image-to-input-ext, toae-tto-uupput-text, and text-to-text paths. Right: AllPath achieves significant performance improvements over the baselines across all benchmarks.
reinforcing attention to the image, while head intervention 72 suppressing "lazy" heads that favor text and resemble those in the base LLM after tuning.
mitigates text-dominant behavior by
In this paper, we explore the causes of hallucination in LVLMs by analyzing their internal components, following the intervention-based line of work. As [22 shows that attention heads have a greater influence ballucination than MLPs, we focus on heads as the target components. We begin with an empirical observation that motivates our comprehensive intervention framework, which is validated through the identification of critical heads, counterfactual edits, correlation analysis, and empirical evaluation. This process comprises four stages, detailed in below. Notably, our primary analysis focuses on LLa LLaVA-vv.5-77B (36), with additional results for Owen-VL-Cha. ( and DW 4 are provided in the Appendix C
First, we propose comprehensive intervention framework aligned with the transformer's causal architecture in LVLMs, based on a thorough evaluation of existing hallucination studies. As shown in Fig. (b), a single hallucination mitigation method often struggles to achieve consistently strong per- formance across different benchmarks [29 45| [7]. Focusing on intervention methods (27 61 39 72),
we observe that they typically intervene along single causal path path-either from input image output text (PAI in Fig. ITa)) or from input text to output text (AD-HH in Fig. Tathhou considering
the system holistically. As result, certain methods [39 72 perform better benchmarks 445
that primarily rely on visual grounding, while others 27 are more effective on benchmarks [2
emphasizing textual coherence. Inspired by this, we propose comprehensive intervention framework aligned with the onput-output sequence and causal transformer architecture of LVLMs, as shown in Fig. Ours). This framework enables joint intervention of the inage-to-output-tex and image-to- ppuu-text-to-ouupptt-eee text causal pathways. Notably, although the input text is provided rather than generated, its hidden representations are influenced by the input image.
Second, we identify the heads in LVLMs that govern distinct intervention pathways within the comprehensive framework by introducing two novel methods, each targeting the image-to-text and text-to-text causal paths, respectively (detailed in Sec. 2. Although the framework involves multiple causal paths, they fall into two main types: text-to-text (input output text) and image-to-text (from image to input or output text). For the former type, as textual output logits are primarily influenced by preceding text tokens (38], identify the path's 's heads by measuring their differential impact on the logits of hallucinated versus non-hallucinated tokens. For the latter type, the image's influence on text is reflected in the attention contributions from image tokens to text tokens. Path heads identified by assessing whether they attend to image tokens that are semantically aligned with the corresponding text. Notably, both of our methods are significantly faster than previous approaches for identifying hallucination-reale
heads, detailed in Sec. [.44
Third, we validate the impact and roles of the identified heads both within individual causal paths and in their combined pathways, revealing key insights into the comprehensive intervention framework (detailed in Sec. 3). We find that:
Text-to-text intervened heads primarily contribute to LVLM alignment, particularly in adhering to cotpucctiioooolooiine formats. Heads are highly correlated and often overlap in questions with similar formats, even across different benchmarks, but differ significantly in more distinct formats (c.g., binary vs. open-ended captioning).
2

---

or MHA Modules
X-y+*
12T Heads List
Image
layer + I. head 0
X-y
Image-to- Text paths
Is there a dog in the image?
Input Text
No!
Yest
layer
Text-to-
T2T Heads
Text path
Yes
LPI Score
List
layer
head 0
(2-2)
Output Text
Heads Identification
Hallucination Mitigation
Figure 2: The overview of our proposed AllPath. AllPath first identifies the most critical text-to-text and image-to-text heads contributing to hallucinations using the Log Probability Increase (LPI) score and the ratio of key object attention to total image attention. Then, by applying adaptive heads interventions, AllPath mitigates hallucinations by manipulating the casual pathways in LVLMs.
The intervened heads along the image-to-inputttex and image-to-outputtlex paths differ, despite both exhibiting stronger attention to image tokens than to text tokens. This suggests that, although both paths "see" the image, the heads governing input and output text function independently.
LVLMs are "smarter" than expected, adaptively selecting the most suitable combined pathways based on the question type-suce both bote-depending on the relative visual and textual demands of the question.
mnage-to-output-exx image-to- sange-to-ipuut-ext--eee input-text-to-outpure
Finally, we propose simple yet effective intervention method for mitigating hallucination, grounded our earlier findings. It adaptively selects the appropriate pathway(s) within the comprehensive intervention framework based on question type and intervenes on the corresponding heads. Experi- ments demonstrate that our method consistently improves performance across diverse benchmarks while maintaining high efficiency.
In summary, our contributions multifold: (1) To the best of our knowledge, are the first to propose a multi-path hallucination intervention framework for LVL.M,, going beyond prior single- path approaches. (2) We introduce two novel methods identify critical heads across different causal pathways. (3) Through counterfactual edits and correlation analysis on attention heads, we uncover several novel findings-reportee for the first time-that offer insights into the internal mechanisms of LVLMs. (4) Built upon this framework, we demonstrate that simple interventions on selected heads yield consistent improvements across benchmarks with diverse question and alignment formats, while maintaining high efficiency.
2 Probing image-to-text and text-to-text heads along causal paths
Building on the intervention framework, we propose two lightweight and plug-and-play methods to identify critical attention heads associated with each type of causal pathway- Sehhway-text-t--feeeee text-to-text and image-to-text, An overview of the our probing method illustrated in Fig. 2
2.1 Preliminaries
We consider an LVLM composed of an image encoder, a modality connector and a base LLM with L transformer layers. At each generation time step f, the LVLM first encodes the input image
text prompt P, and the previously generated response tokens R&ç, into initial embeddings h (0) This embedding sequence then progressively updated through the transformer layers, with the hidden states at layer updated as follows:
S -
(1)
(1-1) 1
where (!!)) and F(I) the outputs of the multi-head attention (MHA) module and the feed-forward
network (FFN), respectively. The final hidden state the current time step, 1 (L) is then used to predict the next token rt+1, which is sampled from:
P(141 ( softmax(Unembenn (
3

---

Algorithm 1: Get Scores Set for Short An- swering Questions
Algorithm 2: Get Scores Set for Open-ended Questions
for each sample, each time step in the model generation process do if t is the first time step in generation then B f correct answer token of the
for each sample, each time step t in the model generation process do B 15 + W;
if rt is judged as hallucination then
question;
[, < {ret;
B, incorrect answer token(s) of the
else if is judged as non-hallucination
question;
then
else
BT {re):
, B, 6;
In this paper, we focus on the MHA module in transformer layers. Let denote the number of attention heads employed in each layer. For specific head indexed by (1, n), where I E {1, 2....... L} the layer index and n E {1, 21,2.-.. ..,,, N) is the head index within that layer, we denote the corre- sponding query, key, value, and output projection matrices as w(lx)), (l.m)) wllx)) and O(.,n)
respectively. Given the layer input Aie
(1-1) the output H1 (1.m) is computed as:
W (.,) (1-1) (..)) (
he (w
A
softmax St
Vdx
H!lm))
(1.m).
willm) (1-1) Olh,)
Age
where da is the scaling factor and A (1.n) is the attention weight for head (I, The final output of
gt
MHA at layer is the sum of N head output, and can be expressed as H E (1.1))
2.2
Discovery of crucial attention heads for the text-to-text path
Given that the textual output relies heavily on preceding text tokens, output probability distribution reflects the model's confidence in hallucinated content, refine the Log Probability Increase (LPI) score 776 77 to quantitatively assess the influence of individual model components in promoting mitigating hallucinations. Specifically, for generated token r, time step I, the LPI score of attention head (1, n) is calculated as:
logProb, (t.m) (r.) DngProba(n log Prr 11 (1-11) + H(())) log (rr 11 ((-1))
which measures the contribution of head (I) token rı- To further capture the effect of the head over a broader set of tokens, we extend the definition of LPI score by considering candidate vocabulary set B, at time and compute the modified score as:
logProb, (L.)) (B.) log Σ
PPb
(11) + H!("")) log Σ Pbb n n1""))
DEB.
DEB.
At the decoding step i, we designate hallucination set B, and non-hallucination set B," Specifi- cally, for short answering questions, prompt the model to produce direct response and analyze the first decoding step, assigning the correct answer to B and all incorrect alternatives to B, For open-ended questions, we focus positions that have been identified as hallucinated non- hallucinated. The corresponding token assigned exclusively to either B, or B with the other set left empty (see Alg. [ and 2 for details). Then, for each attention head (1, n), compute the IPI score on both sets, and then average the scores over all decoding time steps across all samples:
(l.m).- = Es, æ0 [looProb, (l.m) (B, )],
dd.)) = Egz ee [logProby (I.n) (B, SIZ
We then define the Text-to-Text (T2T) Score for each head as the difference between the two:
<(l.m) = s(l.a)++ s(t.a).-
where smaller value indicates stronger tendency of head (I, to promote hallucinations.

---

2.3 Discovery of crucial attention heads for the image-to-text path
In addition to textual context, visual information another crucial factor influencing the model's responses. To identify Image-to-Text (I2T) attention heads, we first select set of tokens which are most important for grounding the image, denoted as Tirr- Specifically, since LVLMs are causal models that later tokens integrate information from earlier ones through attention mechanism, focus only on the first occurrence each object during the question-answer input text P or the output response Rgg, these initial mentions are expected to be grounded in the relevant image regions rather than inferred from prior textual context. This makes these selected tokens especially useful for probing I2T heads.
process, either the
To distinguish whether the corresponding objects of these tokens are present the image, we then partition TIT into two disjoint subsets: TIT and TIzT. where TIT contains tokens whose objects are present in the image, and TET contains tokens whose objects are absent. When t E TTr. we expect the attention to be both strong and concentrated within the relevant region. Accordingly, we define the positive score $ s(l.n)++ as the average summed attention scores within the aligned region M, across all image tokens. Conversely, for E Tzr the attention is expected to be weaker and more diffuse. In this case, we define the negative score ST2T over the entire image region. Formally, compute:
<(l.a).- as the average summed attention scores
cla.). 12T
E
Σ (1a)]
A
Si2
s(.n),,
EET
Σ AA (l.)
M.,,
iET
Finally, similar to Text-to-Text Score, we obtain the Image-to-Text (12T) Score for each head as:
s(l.a)
= S
s(l.n),+
s(l.a).-
12T
2.4 Efficiency analysis of head probing methods
Previous work identifies hallucination-rrlated attention heads via either a zero-out strategy 722 or training-based methods (15). However, both approaches incur substantial computational overhead. The zero-out strategy requires ablating each attention head at every transformer layer to estimate its contribution to hallucination, necessitating full forward pass at each generation time step for every head. Meanwhile, training-basee based methods involve training an additional classifier on a large annotated dataset.
In contrast, our method computes Text-to-Text (T2T) Score and Image-to- -exx (I2T) Score for each attention head based solely on local behavior- the log- log-probability increase at that layer or attention shifts within the layer-without requiring full inference. This allows us to obtain all heads' T2T and I2T scores with only single forward pass, significantly enhancing both efficiency and practicality.
3 Analysis and discussion on identified heads in different paths
In this section, determine the specific roles of the heads identified by analyzing their statistical information and behaviors. First, in Sec. Sec.3.I] we investigate the behaviors the heads identified by our two methods and examine how they contribute the text-to-text and image-to-text causal pathways, respectively. Subsequently. in S.c. we show that LVLMs adaptively select different pathways across benchmarks, and that pathway choice significantly affects hallucination occurrence and mitigation across question types.
3.1 Functional interpretation of the identified heads
Text-to-text causal path is related to the alignment. To investigate the relationships between the heads identified in different datasets, we independently identify heads within each dataset and analyze the correlations between them. Specifically, we focus on comparing the heads identified in the POPE [29] COCO (33 adversarial dataset with those found in other datasets. For each dataset, we rank the Importance Scores s(f.n) and s(l.x) for each head. We then pairwisely plot the ranking positions of each head on two-dimensional graph and calculate the linear regression correlation
5

---

Rank Heads p 0.8427
Rank of Heads p 0.8358
Rank Heads p 0.1233
Rank Heads p 0.1650
Text-to-Texs Heads on POPE COCO adversarial
1024
512
0
512
1024
512
1024
512
1024
512
1024
Toxt-to-Text Heads on POPE COCO random
Text-to-Text Heads on POPR GQA adversarial
Text-to-Text Heads MCQ COCO adversarial
Text-to-Text Heads on CHAIR
Figure 3: Visualization of the rank distributions of text-to-text heads extracted from different datasets. denotes the correlation coefficient; a higher value (i.e., points concentrated along the diagonal) indicates greater similarity between the two sets of heads. Heads identified from datasets with similar question-answer alignment formats exhibit strong correlation and substantial overlap.
Runk of Heads P 0.4378
Image-to -o--to---ppu-eeeee Heads
Average Attention Ratio Over Dataset
Image-to-B Image-to-Outuu--l-. Out-puut--eettt Heads
1024
12T Heads
to
image 84.2%
Vitter
the"
10
All Heads
XNT
W
10
image 56.8%
43.2%
Image-to
Tonng--to-Outpur-TTe Output-Text Heads
"sclp'
dascrile
12T Heads
"Jeuil
Yhe
10
image 68.2%
31.8%
All Heads
512
1024
Imuge-to-Input-Toe Heuds
Teatures
10
image 54.0%
text 46.0%
Figure 4: Left: Visualization of the rank distributions of image-to-input-teeee heads and image-to- output-text heads, showing that the two are largely uncorrelated. Right: Compared to the average of all heads, all image-to-text heads identified, exhibit stronger focus visual content.
coefficient p between them. A higher p value indicates greater similarity in the ranking of heads across the datasets being compared.
we observe that the correlation between text-to-text heads under the same alignment format formm is very high, but the correlation is significantly lower under different formats. Specifically, for Yes/No questions. the correlation coefficient between the heads identified by POPE COCO adversarial and POPE COCO random reaches 0.8247, and the correlation coefficient between the heads identified by POPE COCO adversarial and POPE GQA 22 adversarial reaches 0.8358. However, even with the same images and similar questions, when switching to MCQ format, the correlation coefficient between heads drops only 0.1233. This demonstrates that T2T heads exhibit a strong correlation with the alignment format.
As shown in Fig. 3
image-to-input-tex causal path is different from image-to-output-te causal path. For the image-to-text path, we further investigate the behavioral differences between the heads in the image- to-input-text and maag-tto-output-toe
pathways. Specifically, we conduct experiments using CHAIR. Unlike the standard CHAIR [45] evaluation, we design a modified prompting strategy: we make certain objects appear in the input while prompting the LVLM to generate captions for other objects. We then extract the heads separately from the input and output paths and calculated the correlation coefficient o between these two sets of heads. Additionally, calculate the average attention weights of the top- identified heads and compared them with the average attention weights of all heads.
As illustrated in Fig. we observe that both image-to-inpure input-text heads and image-to-oune to-tooouttttteee text heads attend to the image more significantly than other heads. However, even within the image-to-text causal path, the heads identified in the image-to-input-ers and image-to-output-tes pathways differ considerably. This suggests that focusing solely on the output text when mitigating hallucinations may be insufficient, even though hallucinations only occur in the output.

---

POPE (COCO)
Causal Pathways
CHAIR
Method
Random
Popular
Advers.
POPR
Acc.
F1
Acc.
F1
Acc.
F1
Cs+
C1
w/o mask
85.1
83.7
83.7
82.4
80.9
80.0
52.2
14.6
CHAIR
w/ mask
83,2
81.2
82.0
80.2
79.7
78.1
62.4
32.4
A%
41.9
$2.5
11.7
J2.2
$1.2
11.9
110.2
117.8
Figure 5: Left: When entirely knocked out the attention weights of output tokens attending to image tokens, POPE's performance worsens very little, whereas CHAIR's performance gets worse significantly. Right: This indicates that LVLMs utilize different pathways for different question formats.
3.2 LVLMs adaptively select different causal pathways for different question formats
Motivation experiment setting. Given that both the to-output-text pathways may contribute to the prediction of final output tokens (i.e., the response), we further investigate whether LVLMs exhibit different pathway utilization patterns across different types of hallucination benchmarks. To demonstrate the significance of both causal pathways in LVLM hallucinations, following [26]. we attempt to remove the image-to-texs causal path from the image input to the output while preserving the image-to-text causal path from image input to text input and the text-to-text causal path from text input to output. Specifically, we utilize LLaVA (36 to mask the attention from output tokens to image tokens both POPE and CHAIR, as shown in Fig. 5
and image-
Finding & discussion. We observe that on POPE, even when the image input the output attention is entirely knocked out, performance only decreases by approximately 2%, underscoring the importance of the text-to-text causal path. However, on CHAIR, consistent with the findings of (26] the performance worsens by about 10% after knocking out, indicating the substantial contribution of the image-to-text causal path. This experiment also suggests that the causal paths taken by LVLM in answering questions differ across different question formats. Based on these findings. we contend that focusing solely on strengthening one causal path is insufficient for mitigating LVLM hallucinations comprehensively.
4 Mitigating hallucinations through multi-path head intervention
In this section, building on the heads identified in our intervention framework (Sec. 2 and their roles in different causal pathways (Sec. we propose a simple yet effective intervention method to reduce hallucinations.
Important claim: To demonstrate that study is general and not specific on questions, when identifying heads, we use images and questions that are entirely different from those used during testing, Specifically, for each tested dataset, we generate new dataset and calculate importance scores on it, ensuring that this newly generated dataset shares only the same format with the original dataset without sharing any images.
To identify attention heads with the most pronounced impact on hallucination, we use Importance s(l.n) Scores TTT and S12 indicators for head selection. Specifically, we select Z and Z the top and bottom & attention heads ranked by alignment scores, which are associated with hallucination suppression and promotion, respectively. Similarly, we select Z the top Ç attention heads ranked by image scores s(l.nn) which exhibit the desirable attention patterns over visual inputs.
g(l.a)
Finally, we derive the final sets of attention heads by combining the respective conditions:
== = ZTZT
z = Z U Z
To effectively mitigating hallucinations, we apply different scaling factors (1.n) to different heads according to - and Z+. Specifically,, we apply this modification to MHA, the updated output of the
7

---

MHA module can be expressed as follows:
(I, z if (1,n) z
A
Σ
x (1m) H (l.)
A(mm)
otherwise
where y+ and > are hyperparameters.
5
Experiments
5.1
Experimental settings
5.1.1
Datasets and metrics
POPE [29]. the Polling-basee Object Probing Evaluation (POPE) benchmark is a widely utilized benchmark for evaluating object hallucination in LVLMs. Built upon three established datasets. COCO 33 A-OKVQA [46), and GQA (22), POPE samples 500 images from each dataset and selects three ground-truth objects per image as positive instances. Correspondingly, three absent objects are sampled per image using one of three strategies: random, popular, and adversarial sampling. Each object is then queried in the binary format: "Is there a(n) (object) in the image?" This yields 3,000 questions per dataset, with balanced distribution of positive and negative instances.
MCQ-POPE. To extend the format of questions, we adapted POPE into multiple-choice questions, creating a new benchmark named MCQ-POPE. Specifically, for each image in each dataset each split, we extract all the three objects labeling Yes and three labeling No. For each object, we pair it with three other objects whose labels differ from the target object, shuffle the four objects, and present them in a multiple-choice question. The resulting MCQ-POPE benchmark also maintains total of 3,000 questions for each dataset each split, ensuring consistency with the original POPE benchmark.
CHAIR [45), the Caption Hallucination Assessment with Image Relevance is another widely used metric for evaluating hallucinations in LVLMs, focusing on open-ended image captioning tasks. Objects mentioned in the generated captions but absent from the ground truth are regarded as hallucinations. CHAIR evaluates such errors along two dimensions: sentence-level and instance-level, with their respective scores computed as follows:
Cs= |captions with hallucinated objects]| and G |hhallucinated objects]|
fall captions}]
[all mentioned objects]|
MME [17], the MILM Evaluation Benchmark is a benchmark designed to comprehensively evaluate the performance of LVLMS.. Similar to POPE, MME also requires the model answer yes no questions. Following [74 (7). and given our focus on hallucination, we report results on the hallucination subsets, specifically including existence, count, position, and color subsets. Consistent with their official implementation, we use the sum accuracy and accuracy+ + as the evaluation metric.
We conduct experiments on the LLa (36). In addition to the vanilla baseline, we also compared AllPath with some other training- free methods, including contrasive-ddecdding-aaa based VCD [27 and ICD [61. intervention-ben based PAI [39 and AD-HH (72).
Baselines.
5.1.2 Implementation details
Unless otherwise specified, we set the scaling factors to y = 2.0 and y = 0.0. For short-answering tasks, set & 20 and 10, while for open-ended tasks, set 40 and ( 50. For MME, we employ the same attention heads identified from POPE. See Appendix A for more details.
5.2 Comparison with state-of-the-art methods
As shown in Table T our AllPath exhibits the highest performance across all benchmarks with different formats. It can observed that on the POPE dataset, our method achieves at least a 2.1% increase in F1, significantly higher than the best baseline improvement of 0.9%. On MCQ-POPE, our method also achieves at least 5.2% increase. On the CHAIR dataset, we also achieve the highest F1 increase of 2.6%, exceeding the best baseline improvement of 1.9%. Additionally, find that while

---

POPE (COCO)
MCQ-POPE (COCO)
CHAIR
Method
Random
Popular Acc.
Advers. Ace.
Random Acc.
Popular Acc.
Advers. Acc.
Acc.
F1
F1
F1
F1
CxL
C4
Vanilla
85.1
83.7
83.7
82.4
80.9
80.0
72.8
72.9
68.0
68.0
64.4
64.5
52.2
14.6
VCD 27
86.3
85.2
84.5
83.5
81.4
80.9
78.2
78.2
72.2
72.2
67.9
67.9
58.2
16.1
ICD 66
84.4
82.2
83.8
81.6
81.5
79.5
69.1
69.1
67.2
67.2
64.7
64.6
51.4
14.7
PAI (39
86.4
85.0
84.9
83.6
82.5
81.4
78.0
78.0
71.2
71.2
68.0
67.9
28.8
7.9
AD-HH 72
85.0
83.6
83.7
82.4
80.9
79.9
78.5
78.6
71.1
71.1
68.1
68.1
33.2
7.5
AllPath
87.2
86.0
86.0
84.9
82.8
82.1
80.5
80.5
74.0
74.0
69.7
69.7
26.6
7.2
Table 1: Results POPE, MCQ-POPE, and CHAIR benchmark. The Fl score for MCQ-POPE
represents Macro Fl.
Object-Level
Attribute-Level
Method
Total
Existence
Countt
Position
Colort
Vanilla
180.0 (48.6))
113.9 (+10.05)
116.7 (+15.90)
129.4 (+18.28)
540.0 (#39.69)
VCD 2
177.8 (22.55)
122.8 (120.84)
122.2 (=1.92)
141.7 (12.89
564,4 4 (120.84)
ICD [61
185.0 (=5.00)
121.7 (+2.89)
118.3 (+6.01)
151.7 (+7.64)
576.7 (+11.67)
PAI (39
185.0 (25.00)
122.8 (#14.94)
114.4 (#4.19)
144.4 (25.09)
566.7 (+15.90)
AD-HH 772
179.4 (25.09)
117.8 (=12.51)
116.1 (#:3.37))
152.8 (=10.72)
566. (+:2.19) 19)
AllPath
188.3 (+2.89)
126.1 (#0.96)
132.2 (68.39)
153.3 (#7.64)
600.0 (+8.66)
Table 2: Performance on MME Benchmark [17]. Results are reported as the average of 3 test runs, with = indicating the sample standard deviation.
VCD shows noticeable improvements on POPE and MCQ-POPE, its performance on CHAIR suffers a significant decline of 1.4%. AD-HH, on the other hand, exhibits substantial performance gains on CHAIR and MCQ-POPE, but its performance on POPE similar that of the vanilla method. Our method achieves substantial performance gains across three benchmarks of completely different formats.
Our results on MME is shown in Table It can be observed that our method still achieves stable performance improvement the MME hallucination subset. Considering that still use the POPE heads for MME without extracting independent heads specifically for MME, this demonstrating the generalizability of our method under unified task format.
5.3 Ablation study
Ablation study on pathways. To evaluate the two key components of our method, the T2T heads and 12T heads, we conduct an ablation study on POPE. The results are shown in the upper part of Table We observe that employing either T2T heads or 12T heads individually leads to a certain degree of performance improvement. However, the most substantial performance gain is achieved only when all heads are utilized, significantly outperforming the previous two configurations. This result substantiates our claim that both proposed pathways are present and exert a considerable influence on L.LLM hallucinations. Therefore, considering only one pathway is insufficient for effectively mitigating LVLM ballucinations.
Ablation study on hyprpaarameters. To validate that our AllPath is insensitive to hyperparameter selection, we perform an ablation study on POPE COCO. As results report the lower part of [3 our method is largely insensitive to the choice of hyperparameters. Substantial changes to the hyperparameters lead only minor variations in performance. For instance, the accuracy on the random subset fluctuates only within the range of 85.6-88.3, consistently outperforming the baseline of 8..1. Notably, some hyperparameter settings this table result better performance on POPE. This is because rather than tuning hyperparameters for specific dataset, we select single set of hyperparameters that works best across various discriminative tasks. This shows that our method effective and stable without extensive tuning.

---

Hyperparameterss
Random
Popular
Adversarial
{
y²*
Acc.
F1
Acc.
F1
Acc.
F1
20
10
2.0
0.0
87.2
86.0
86.0
84.9
82.8
82.1
10
2.0
0.0
86.2
85.2
84,5
83.6
81.6
81.1
20
0
2.0
0.0
86.4
84.9
85.2
83.7
82.5
81.3
0
2.0
0.0
85.1
83.7
83.7
82.4
80.9
80.0
10
10
2.0
0.0
87.3
86.2
86.0
85.0
82.6
82.0
10
2.0
0.0
88.3
87.7
86.3
82,9
83.0
40
10
2.0
0.0
88.1
87.9
85.9
86.0
81.2
82.2
20
5
2.0
0.0
86.2
84.8
85.8
84.6
82.7
81.9
20
15
2.0
0.0
88.2
87.4
86.6
85.9
83.0
82.8
20
2.0
0.0
88.1
87.3
86.7
86.0
83.0
82.8
10
1.2
0.0
85.6
85.3
85.3
84.1
82.3
81.5
20
10
1.5
0.0
86.9
85.7
85.6
84.5
82.5
81.7
20
10
2.5
0.0
86.8
85.6
86.3
85.4
83.1
82.5
20
10
2.0
0.2
87.1
85.9
85.8
84.7
82.5
81.8
2.0
0.5
87.0
85.8
84.5
82.2
81.5
20
10
2.0
0.8
86.8
85.5
85.4
84.2
82.0
81.3
Table Ablation studies. Hyperparameters different from the default setting are highlighted.
Related work
Large Vision-Language Models. With the success of LLMs 59 6O
increasing 10
44 2
9 73). A typical LVLM architecture comprises vision encoder 43 with LLM, connected via modality alignment module such linear projection (36] or a Q-former [86 14]. To support effective cross-modal reasoning, these models follow multi-stage training pipeline. The pretraining stage focuses aligning visual and language modalities using large-scale image-text pairs. Building on this foundation, supervised fine-tuning (SFT) with essructtionfollooinng data enhances the model's ability generate coherent responses conditioned on multimodal prompts. Beyond standard training recipe, several works have further explored advanced adaptation strategies, including multi multi-phase pre-training [4] knowledge learning stage 337, and Reinforcement Learning from Human Feedback 55
number of studies focus on LVLMs [36
3 86
Mitigating hallucinations in LVLMs. Extensive researches have been conducted mitigate LVLM
hallucinations from various perspectives, including curating training datasets of models [35] 75 79]
modifying training objectives [24].
adjusting model architectures (65), and applying post-processing
methods to model outputs 855
63
Nevertheless, many studies also focus on mitigating hallucinations
during inference. Some work [16
62
61 82
primarily focused on Contrastive Decoding (28)
methods, while more recent approaches have focused on attention mechanisms, such as applying straightforward scaling techniques 39 or by dynamically adjusting scaling weights [25] on attention scores for range of layers and tokens. Meanwhile, some approaches focus modifying attention heads. 72 identifies hallucinatory heads by pero-ablation, and 7 identifies hallucinatory heads by
modifying input image tokens and training a binary classifier.
Understanding LVLMs. Currently, methods for understanding LVLMS can mainly be categorized into three types. The first type is based on averaging transformer weights. However, [54] 722 demonstrates that they do not effectively capture the specific regions that the model actually attends to. Additionally, Grad-CAM (47) visualize the model's 's focus by calculating the gradient of the output with respect to the input. Another approach the early exit method [42 (2), which obtains hidden state representations by extracting them before the unembedding process.
7 Conclusion
In this paper, we propose a unified framework for mitigating hallucinations in LVLMs. By distinguish- ing between Text-to-Text and image-to-Text pathways and analyzing their corresponding attention heads, we develop AllPath, a simple yet powerful method. Experimental results demonstrate that AllPath consistently achieves superior performance across all benchmarks.
Acknowledgment: This work supported by the National Natural Science Foundation of China under Grant 1o.62206174 and No.62576365.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar 2tt:::333008774,, 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, ct al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[3] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond, arXiv preprint arXiv:2308 1 12966, 2023.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Siho Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu. Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng. Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint wrXiv:2502.1392. 2025.
[5] Chaoqi Chen, Yushuang Wu, Qiyuan Dai, Hong- Yu Zhou, Mutian Xu, Sibei Yang, Xiaoguang Han, and Yizhou Yu. A survey on graph neural networks and graph transformers in computer vision: A tosk-oriented perspective, IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 46(12):10297-10318 2024.
doi: 10.1 1090//AA..224..44455544.
[6] Jun Chen, Deyao Zhu. Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoor- thi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for dision-language multi-task learning. arXiv preprint #rXiv22310.09477 2023.
[7] Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang. Lijie Wen, and Xumiigg Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models. arXiv preprint urXiv:2411.15260 2024.
[8] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu and Rui Zhao. Shikra: Unleashing multimodal Iim's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
[9] Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lui, et al. Intervvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
[10] Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[11] Qiyuan Dai and Sibei Yang. Curriculum point prompting for weakly-supervised referring image segmenta- tion. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 13711-13722. June 2024.
[12] Qiyuan Dai and Sibei Yang. Free the fly: Enhancing flexibility in test-time adaptation with online em, 2025. URL httpp://arxiv..ot wtgg/ba.///aa. abs/ 01222507069999
[13] Qiyuan Dai, Hanzhuo Huang. Yu Wu, and Sibei Yang. Adaptive part learning for fine-grained generalized category discovery: A plug-and-play enhancement, 2025. URL
org/ /bs/ htrg//bb/2507... 2507 009928
[14] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructhlip: Towards general-purpose vision-language models with instruction tuning, 2023. U bttps://aaxxi: wwtpsivvvoorgaeeeo 2b3/ 06500
[15] Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Truthprint: Mitigating Ivlm object hallucination via latent truthful-guie guided pre-intervention, 2025. URL httpp:///aaartt org/ abs 2503. 10602
[16] Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding, 2024. URL https://: http:://aaivvvvee org/ abs/ 2403. 14003
[17] Chaoyou Fu. Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.

---

[18] Anish Gunjal, Jihan Yin, and Erhan Bas, Detecting and preventing hallucinations in large vision language models. In AAAI Conference Artificial Intelligence, 2023. URL /etp://///pp..p.aaiiaaaaeaea org/Corpus I :2:o0pu722 260887222
[19] Xiang He. Sibei Yang. Guanbin Li, Haofeng Li, Huiyou Chang, and Yizhou Yu. Non-local context encoder: Robust biomedical image segmentation against adversarial attacks. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):8417 8424, Jul. 2019. doi: http://aiii330000331118000
URL httlthht///oop. aaai. org/index. php/ AAAI/ Tat.A//articeeeeveeeo 4857
[20] Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Frec-bloom: Zero-shot text-to-video generator with Ilm director and Idm animator. arXiv preprint arXiv:2309. 14494, 2023.
[21] Hanzhuo Huang. Yuan Liu, Ge Zheng. Jiepeng Wang, Zhiyang Dou, and Sibei Yang. MVTokenflow: High-quality 4d content generation using multiview token flow. In The Thirteenth International Conference on Learning Representations, 2025. URL htt.ittpee//pppeeeeiieeee
net wate/foorrst innerrummmiizii cBTPaDb
[22] Drew A Hudson and Christopher D Manning. Gqa: A new datasct for real-world visual reasoning and compositional question answering. Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[23] Ziwei Ji. Nayeon Lee, Rita Frieske. Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii. Ye Bang. Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1 2023.
[24] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model, 2023.
[25] Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:241 1.12211.1.724.. 2024.
[26] Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? a deep-dive into the vision of vision language models, 2024. htlL:htts::/// /tttps///axiiv..e org/ hbsp//11.....aeaaaaaaaeabbbtabtt 17491
[27] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:231 16922, 2023. URL /ttp:/ttts////xxiii www.wwwwww..........eeaeeeeeeeee wtwg/ba////// 2211 16922
[28] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemayer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization, 2023.
URL hhtll::tt/////. //tp:::////xxyy org/ arg/abs//210.... 2210. 15097
[29] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL /tttstsss////ttteeeee
[30] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[31] Han Liang, Jiacheng Bao, Ruichi Zhang, Sihan Ren, Yuecheng Xu, Sibei Yang, Xin Chen, Jingyi Yu, and Lan Xu. Omg: Towards ppen-vocabulary motion generation via mixture of controllers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 482-493, June 2024.
[32] Liang Lin, Pengxiang Yan, Xiaoqian Xu, Sibei Yang. Kun Zeng, and Guanbin Li. Structured attention network for referring image segmentation. IEEE Transactions on Multimedia, 24:1922-1932, 2022. doi: 10. 21199TTM.2022.007744
[33] Tsung- Yi Lin, Michael Maire. Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays. Pietro Perona, Deva Ramanan, Piotr Doll'a and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs//4405.033 2014. URL http://arxiv httt:://aaxiv.org/ab/////ae alee//05s/1440...... 0312
[34] Zhenxiang Lin, Xidong Peng, Peishan Cong, Ge Zheng, Yujing Sun, Yuenan Hou, Xinge Zhu, Sibei Yang. and Yuexin Ma. Wildrefer: 3d object localization in large-scale dynamic scenes with multi modal visual data and natural language. In ECCV (46), pages 456 473, 2024, URL https: org/10.. 1007/ 978-3- 1-8-30031 720--7722222222
12

---

[35] Fuxiao Liu, Kevin Lin, Linjie Li. Jianfeng Wang. Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023.
[36] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023.
[37] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL httpps://1laaaavll.gg blog/2024-01-222 01-10022----------0000- Llava-next/
io/
[38] Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: A chatgpt-prompted visual hallucination evaluation dataset. In CVPR, 2025.
[39] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention image: A training-free method for alleviating hallucination in Ivims. arXiv preprint arXiv:2407.21771, 2024.
[40] Xuyang Liu, Bingbing Wen, and Sibei Yang. Coq: Cross-class query network for partially labeled organ segmentation. Proceedings of the AAAI Conference on Artificial Intelligence. 37(2):1755 1763, Jun. 2023. doi: 10. 10.1609/ (t..///ai.v372...55222... 2572..25264. v126/
URL https::/ojs aaai. .rg index. www.dde...pppaeee.m articce
[41] Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination rision-language models, ArXiv, abs/2310.05338,, 2023. URL httphttsp://pp wet.app..smaatiii. wwwastemaiiieeeae semanticscholle cscholar Ttrg/CorpussD... Co0ppssID:263822610 263829510
[42] nostalgebraist. Interpreting gpt: The logit lens. https: wwwp:///www..ss.ongg.gooom com/ wwm/posts. AckKBBsoppaaN@vvvrr/nnttteteer http:////vvorrnntteeeeeeeeie gpt- tot-thh--ooit- lens 2020.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transfcrable visual models from natural language supervision, 2021. URL httpps://axxv..or///// 2103.00020
[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551 2020.
[45] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burs,, Trevor Darrell, and Kate Saenko. Ohject hallucination in image captioning. arXiv preprint arXiv: 1809. 02166, 2018.
[46] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022. URL https
//arxiv.orz abs/ 221.22226.000 01718
[47] Ramprasaath R. Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks gradient- localization. In 20/7 IEEE International Conference on Computer Vision (ICCV), pages 618-626, 2017, doi: 10. 1009/CCC. 2017.74.
[48] Cheng Shi and Sibei Yang. Spatial and Visual Perspective Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding, page 201-218. Peringer-Verlase Berlin, Heidelberg. 2022. ISBN 978-3-031- -200588-- doi: 10.1007/978-3-081-20055 12. https://t/////oo..rrr 1007 3 031 20059-550 12
[49] Cheng Shi and Sibci Yang. Edadct: Open-vocabulary object detection using carly dense alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
[50] Cheng Shi and Sibei Yang. Logopromptsyythhtii text images can be good visual prompts for vision- language models. In Proceedings of the IEFF/CVF International Conference Computer Vision (ICCV), Octoher 2023.
[51] Cheng Shi and Sibei Yang. The devil in the object boundary: Towards annotation-free instance segmen- tation using foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL httphh:://ooeereeiiiicco www.nneeooommdddaoooommm
[52] Cheng Shi, Yulin Zhang, Bin Yang, Jiajin Tang, Yuexin Ma, and Sibei Yang. Part2object: Hierarchical unsupervised 3d instance segmentation. arXiv preprint arXiv:2407.10088 2024.
13

---

[53] Cheng Shi, Yuchen Zhu, and Sibei Yang. Plain-det: A plain multi-dataset object detector. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol. editors. Computer Vision ECCV 2024, pages 210-226, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-0031-72652-1.
[54] Gabriela Ben Melech Stan, Raanan Yehezkel Rohekar, Yaniv Gurwicz, Matthew Lyle Olson, Anahita Bhiwandiwalla, Estelle Aflalo, Chenfei Wu, Nan Duan, Shao-Yo Tseng, and Vasudev Lal. Lvim-intreprett An interpretability tool for large vision-language models, 2024.
[55] Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14522,, 2023.
[56] Jiajin Tang, Ge Zheng, Cheng Shi, and Sibei Yang. Contrastive grouping with transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23570-23580, June 2023.
[57] Jiajin Tang. Ge Zheng. and Sibei Yang. Temporal collection and distribution for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15466- 15476, 2023.
[58] Jiajin Tang. Ge Zheng, Jingyi Yu, and Sibci Yang. Cotdet: Affordance knowledge prompting for task driven object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3068-3078, 3078, 2023.
[59] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2r7iv:2307.09928. 2023.
[61] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations large vision- language models with instruction contrastive decoding. In Findings of the Association for Computa- tional Linguisties ACL 2024, pages 15840- 15853, 2024. URL https: htttps://aclanttoooogyyy org. 2024 findings- acl.937
[62] Sangmin Woo, Donguk Kim, Jachyuk Jang, Yubin Choi, and Changick Kim. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405. 178200 2024.
[63] Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncovering object hallucinations in large vision-language models. arXiv preprint 1122 2024.
[64] Yu Wu, Yana Wei, Haozhe Wang. Yongfei Liu, Sibei Yang, and Xuming He. Grounded image text matching with mismatched relation reasoning, 2023.
[65] Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410. /5926, 2024.
[66] Sibei Yang. Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression comprehension. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4643 4652, 2019, doi: 10. N0.1101CCC..20190000 .04444444....
[67] Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal relationship inference for grounding referring expressions. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4140-4149, 2019. doi: 10. 2..1109/CPR..209.0000
[68] Sibei Yang. Guanbin Li, and Yizhou Yu. Propagating over phrase relations for one-stage visual grounding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 589 605, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58529-7
[69] Sibei Yang. Guanbin Li, and Yizhou Yu. Graphe-tructured referring expression reasoning in the wild. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 9949 9958, 2020. doi: 10. N0.11099CVR 2WW.CCCVVR22222222009999977
14

---

[70] Sibei Yang, Guanbin Li, and Yizhou Yu. Relationship-embeude representation learning for grounding referring expressions. IEEE Transactions on Puttern Analysis and Machine Intelligence, 43(8)22765-2779. 2021. doi:
[71] Sibei Yang. Meng Xia, Guanbin Li. Hong- Yu Zhou, and Yizhou Yu. Bottom-up shift and reasoning for referring image segmentation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2021. doi: 10.1 10.1109/CVPR444437.2200 200.PPPR4446372221111111
[72] Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Understanding and mitigating hallucination large vision-language models via modular attribution and intervention. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL btttps://oppererreeeee
[73] Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owi. Modularization empowers large language models with multimodality, 2023.
[74] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianho Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint or Xiv:2310. 16045, 2023.
[75] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang. Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data, 2023.
(76] Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3267-3280, 2024.
[77] Zeping Yu and Sophia Ananiadou. Understanding multimodal Ilms: the mechanistic interpretability of llava in visual question answering. arXiv preprint arXiv:2411. 10050, 2024.
[78] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction tunnnn--tdddddd audio-visual language model for vidco understanding. arXiv preprint 2rXiv:2306.02856 2023. URL / //axxiv..vvvva///////a.//avi org/ aba/ 2306
02858
[79] Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. Reflective instruction tuming: Mitigating hallucinations in large vision-language models. arXiv preprint arXiv:2407.11422 2024.
[80] Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint 23355 13534, 2023.
[81] Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-or cann-off-hooue thought prompting for multimodal reasoning in language models. arXiv preprint arXiv:2310. 16436, 2023.
[82] Ge Zheng, Jiaye Qian, Jiajin Tang, and Sihci Yang. Why Ivims are more prone to hallucinations in longer responses: The role of context. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 4101-4113, October 2025.
[83] Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang. Aiwei Liu, and Xuming Hu. Mitigating modality prior- induced hallucinations in multimodal large language models via deciphering attention causality. In The Thirteenth International Conference on Learning Representations, 2025.
[84] Hong- Yu Zhou, Chixiang Lu, Sibei Yang. Xiaoguang Han, and Yizhou Yu. Preservational learning improves self-supervised medical image models by reconstructing diverse contexts. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3479-3489, 2021. doi: 10.1 119/11144992222200 2et.22221.00000 100348.
[85] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large lision-language models. arXiv preprint arXiv:2310.000 007554, 2023.
[86] Deyao Zhu, Jun Chen, Xianqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4; Enhancing vasion-language understanding with advanced large language models. arXiv preprint 2304 10592, 2023.
[87] Yuchen Zhu, Cheng Shi, Dingyou Wang, Jiajin Tang, Zhengxuan Wei, Yu Wu, Guanbin Li, and Sibei Yang. Rethinking query-based transformer for continual image segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4595-4606, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions and align well with the theoretical and experimental results presented the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix E for our limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not contain any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed descriptions of the experimental setup, data preprocessing, model architecture, and training procedures, ensuring that the main results can be reliably reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will be released after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee htt.:///gg..eee/ooeSSuuuiiiisss Coooeooomisssssseeeeeeee www.siiiccciiiiiccoooooooiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The paper provides detailed experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The performance improvements are significant, and the paper provides detailed analyses to support our conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification:
The experimental results are not sensitive to compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:t://euirpps.ccomo /tt
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwparripp..ccoooeeeeeeeeeea /t/CCooffeeencs///2222 for what should or should be described.
22

---

Appendix
The appendix includes supplementary details, analyses, and experiments in support of the main text, structured as follows:
Section
A
Additional Experimental and Implementation Details
Section
A.
Additional Implementation Details
Section
Prompt Design for MCQ-POPE
Section
B
Additional Analysis
Section B.1
Adaptive Selection of Causal Path in LLaVA-v1.5-7B
Section B.2
Functional Interpretation the Identified Heads Qwen-VI.-Chal
Section
B33
Adaptive Selection of Causal Path in Qwen- VL-Chat
Section C
Additional Experimental Results
Section C.I
Experimental Setup for Qwen- VL-Chat
Section
Results of Qwen-VL-Chat POPE, MCQ-POPE and CHAIR
Section
C,3
Additional Results Qwen2.5-VL. on POPE
Section D Qualitative Results
Section
Limitations
A Additional experimental and implementation details
A.1
Additional implementation details
For POPE 29 and MCQ-POPE, attention heads identified using samples generated via
adversarial strategy, and subsequently applied across all three test splits: random, popular, and adversarial. Following the official LLaVA evaluation setup, prompts are appended with "Please answer this question with one word." All evaluations on POPE, MCQ-POPE, and MME [7 utilized nucleus sampling with a temperature of and does use beam search.
For CHAIR [55, we randomly sample 500 images from the COCO [33 dataset validation subset to identify attention heads. For both attention head identification and final evaluation, pair each image with the prompt "Please help me describe the image in detail.", " and responses are generated using greedy decoding with maximum length of 512 tokens.
A.2 Prompt for MCQ-POPE
Which of the following (appears does not appear) in the image?
A. (objectl]
B. [object2] }
C. (object3) }
D. [object4)
B Additional analysis
In this section, we first supplement the main text with additional analyses based on LLaVA-v1.5.
7B (36] in Section B. further support the conclusion that LVLMs adaptively select different causal pathways for different questions. We then replicate the same series of analyses using Qwen-VI. VL. Chat [3] in Section B.2 and Section B.3 Implementation details for Qwen- VL--Chat are provided in
Section C
23

---

Rank of Hends 0,7900
Heads 0.6709
Rank Heads = 0,0500
Rank Heads 0.0100
Text-ioTeest Heads POPE COCO adversarill
1024
1024
1024
512
512
512
512
0
512
1024
512
1024
512
1024
0
512
1024
Text Text Heads on POPE COCO random
Text-to-Text Heads on POPE GQA adversarial
Text-to-Text Heads MCQ COCO adversarial
Text-to Heads on CHAIR
Figure 6: Visualization of the rank distributions of text-to-text heads extracted from different datasets on Qwen-VI.-Chat [3]. P denotes the correlation coefficient; a higher value (i.e., points concentrated along the diagonal) indicates greater similarity between the two sets of heads. Heads identified from datasets with similar question-answer alignment formats exhibit strong correlation and substantial overlap.
LLaVA on POPE (COCO)
Qwen on POPE (COCO)
Method
Random Acc.
Popular
Advers.
Random
Popular Acc.
Advers. Acc.
F1
Acc.
F1
Acc.
Fl
Acc.
F1
F1
F1
w/o mask| 85.1
83.7
83.7
82.4
80.9
80.0
84.5
82.3
83.4
81.4
80.5
78.9
w/ mask
52.9
59.2
47.8
56.7
48.9
56.6
48.3
31.0
51.2
31.3
50.6
33.5
%
432.2
424.5
4.35.9
425.7
432.0
423.4
135.0
451.3
432.2
450.1
429.9
445.4
Table 4: When we knock out the attention weights of input text tokens attending to image tokens on LLaVA-v1.5-7b [36] and Qwen-VL-Che (3). POPE's performance worsens significantly.
B.1 Adaptive selection causal path in LLaVA-v1.5-7B
As supplement to the main experiments, further knock out the image-to-tens causal path from the image input to the text input while preserving the image-to-text path from image input to text output, as well as the text-to-text path from text input to output. Specifically, conducted on LLaVA, where we mask the attention from text input tokens to image tokens.
experiments
As shown in Table 4 (left), this modification results in substantial performance drop on POPE, with accuracy falling to around 50% comparable to random guessing. This suggests that the model's prediction on POPE heavily depends on the causal path from the image input to the text input.
B.2 Functional interpretation of the identified heads in Owen-VL-Chaa
Text-to-text causal path is related to the alignment. We present the correlation maps of heads identified in Qwen-VL Liven-VL-CHie VL-Chat in Fig. 6 Consistent with the findings from LLaVA, we can observe that the correlation between text-to-text heads under the same alignment format is very high, but the correlation is significantly lower under different formats in Qwen-VL-Chat Specifically, for Yes/No questions, the heads identified from POPE COCO adversarial show strong correlations with those from POPE COCO random and POPE GQA (2 adversarial, with correlation coefficients of 0.79 and 0.67, respectively. However, the correlation drops markedly when comparing POPE COCO adversarial with heads identified from the MCQ-POPE COCO (0.05), and also declines when compared to those from generation tasks (-0.01). These observations indicate that the T2T heads in Qwen- VL-Chat also exhibit a strong correlation with the alignment format.
image-to-input-tex causal path is different from image-to-input-te causal path. Consistent with the LLaVA-based analysis in the main text, Fig. presents the correlation between image-to- heads in Qwen-VL-Char VL-Chat under the modified prompting strategy
input-text and saaag-to-ouuput-tee on CHAIR, along with the visualization of their focus on visual content. The results reveal that Qwen- VL-Chat exhibits similar pattern: although both the image-to-input-ter and image-to-uupput-eex heads attend more strongly to the image than other heads, their attention patterns differ markedly. This suggests that the two pathways modulate the model's behavior in distinct.
24

---

Rank Heads -0.5596
Image-to Crotee-to-Inpleee Co--Innut-eeee Text Heads
Average Attention Ratio Over Dataset
Amige-to-Duupu-Teee Heads
1024
Stom
12T Heads
aife
could
y001 liller
ory
image 71.2%
28.8%
detaile
10
All Heads
Inn Sunt
image 42.7%
text 57.3%
512
Image-to- Maage-to-Qutpinn Outputt-preee Heads
10
Prraed
descriee
12T Heads
image 94.5%
detail
Sinn -
stint
Heads
512
1024
Imageeto-Inpuu--De Heads
image 48.5%
text 51.5%
Figure 7: Left: Visualization of the rank distributions of image-to-input-tes heads and image-to- output-text heads on Qwen-VL-Chat VL-Chat [3), showing that the two are largely uncorrelated. Right: Compared to the average of all heads, all image-to-text heads we identified, exhibit stronger focus on visual content on Swen--V-CChtt.
POPE (COCO)
Causal Pathways
CHAIR
Method
Random Acc.
Popular Acc.
Advers. Acc.
POPE
F1
Cs4
C1
w/o mask
84.5
82.3
83.4
81.4
80.5
78.9
43.4
13.5
CHAIR
w/ mask
83,4
81.1
83.1
80.9
80.9
78.9
78.4
48.0
A%
11.1
11.2
10.3
J0.5
10.4
10.0
135.0
134.5
Figure 8: Left: When we mask the attention weights of output tokens attending to image tokens Owen-VL-Chat (3), POPE's performance worsens very little, whereas CHAIR's performance gets worse significantly. Right: This indicates that LVLMs utilize different pathways for different question formats.
B.3 Adaptive selection causal path in Qwen- QW-Cn-V
We perform two counterfactual interventions based on output-text path by masking attention from text output tokens to image tokens, while preserving the image-to-input-rext and text-to-text paths; and (2) removing the image-to-input-tex path by masking attention from text input tokens to image tokens, while keeping the smage-to-output-er and text-to-text paths. The effects of both interventions are illustrated in Fig. g and Table (right).
(1) removing the image-to-
(1) As shown in the left panel of Fig. 8 the first counterfactual intervention results in a negligible impact on POPE, with an average performance drop of less than 0.5 points, whereas it causes substantial degradation on CHAIR. This suggests that the image-to-outpll path is not critical for POPE, but plays crucial role in CHAIR. (2) Table 4 provides further evidence of the critical role played by the accuracy to chance level, underscoring its essential contribution successful prediction.
causal path POPE: masking this pathway reduces the model's s
C Additional experimental results
C.1 Experimental setup for Dowen-V--Ciin
Following a similar methodology to -LaVA-v1.5-7B 336, we identify attention heads in Qwen-VL- Chat (3] using a generated dataset that entirely disjoint from the test set. This dataset is employed to compute both the T2T and 12T Scores for head probing. However, due to substantial differences between the base models of Qwen- VL-Chat and LLaVA, we introduce two key adaptations in our
s!l.n).+ This change is necessitated by the observation that attention heads Powen-VL-Chaa generally assign non-negligible attention to most image patches. As result, penalizing background attention would otherwise favor heads with uniformly low attention to visual inputs, which is not desirable. (2) We adopt & = 20 and for both discriminative and generative tasks.
implementation: (1) We modify the 12T Score to s((m) Si2T
25

---

POPE (COCO)
MCQ-POPE (COCO)
CHAIR
Method
Random
Popular
Advers. Acc.
Random
Popular
Advers.
Acc.
F1
Acc.
Acc.
F1
Acc.
F1
Acc.
F1
Cs
C1
F1
Len
Vanilla
84.5
82.3
83.4
81.4
80.5
78.9
61.8
67.3
55.9
58,9
52.5
56.1
43.4
13.5
76.4
98.1
VCD 27
85.5
83.4
84.3
82.3
82.0
80.2
66.6
71.9
60.9
63.6
55.3
58.3
46.0
12.9
76.9
100.8
ICD
85.4
83.6
84.1
82.4
81.7
80.3
74.9
77.7
67.3
68.8
62.0
63.3
50.4
14.4
73.9
100.5
PAI (39
85.8
84.1
84.9
83.3
82.4
80.9
57.9
66.1
52.0
56.6
48.6
54.6
17.2
7.9
70.1
36.8
AllPath
86.6
85.3
85.6
84.3
82.4
81.5
75.4
77.7
68.8
68.8
62.6
63.6
34.2
9.0
77.2
91.0
Table 5: Results of Qwen-VL-Chat 3 model on POPE [29]. MCQ-POPE, and CHAIR 45 bench-
mark. The Fl score for MCQ-POPE represents Macro F1. Notably, PAI compromises generative capability, resulting in a substantial reduction in generation length. In contrast, our method signifi- cantly improves performance across all benchmarks while largely maintaining generative capacity.
POPE (A-OKVQA)
POPE (GQA)
Method
Random Acc.
Popular Acc.
Advers. Acc.
Random Acc.
Popular F1
Advers. Acc.
F1
F1
F1
F1
Acc.
F1
Vanilla
86.3
85,8
82.7
82.7
75.9
77.5
85.1
84.5
79.6
80.0
76.6
77.6
VCD 20
86.2
86.2
82.3
83.0
76.2
78.5
86.4
86.4
78.3
79.8
75.6
77.7
ICD
661
86.0
85.1
83.2
82.6
77.3
77.8
84.5
83.3
80.6
80.0
77.5
77.6
PAI
39
87.8
87.3
84.1
84.0
77.2
78.5
86.2
85.6
80.5
80.8
77.5
78.4
AD-HH
722
86.2
85.7
82.7
82.7
76.0
77.6
85.0
84.5
79.6
80.0
76.5
77.5
AllPath
88.1
87.9
84.2
84.5
77.6
79.4
87.5
87.2
81.9
82.5
78.2
79.5
Table Results of LLaVA-v1.5-7b [36
model on POPE [29]
A-OKVQA 44 and GQA 22 split.
Our method achieves consistent improvements over previous approaches across all metrics.
C.2 Results of Qwen- VL-Chat on POPE, MCQ-POPE and CHAIR
The results of Qwen-VL-Chie
POPE 22
MCQ-POPE, CHAIR 45 are shown in Table 5
Notably, our method outperforms each method on at least one benchmark by a significant margin. Specifically, compared to VCD (27). method achieves average improvement of 8.0% % and 5.4% in terms of accuracy and Fl score on MCQ-POPE, respectively, and yields gain of 11.8 and 3.9 CHAIR, and CHAIR,, Relative to ICD 61], approach demonstrates significant margin 16.2 on CHAIR, g and 5.4 on CHAIR, As for the PAI method [39]. is noteworthy that it degrades the generative capability of Owen-VL-Chat, leading to 36.8) in generation length on CHAIR. In contrast, our method not only achieves a significant average improvement of 16.1 % on accuracy and 10.9% on Fl score MCQ-POPE over PAI, but also substantially reduces the metrics on CHAIRs and CHAIR, while largely preserving the original generation length of Qwen-VL-Chat.
substantial reduction (from 98.1
C.3 Results of Qwen2.5-VL. on POPE
To verify that our AllPath remains effective on newer and larger models, we evaluate its performance on POPE Dwen2.5-VLL-7B722 [4, as shown in Table 7 It is worth noting that since 72B version of Qwen2.5-VL. L has 5 times as many heads as LLaVA, we accordingly scale the number of heads used by factor of five, i.e., & 100 and Ç = 50. As can be seen from the table, our method also achieves consistent improvements on Qwen2.5.
D Qualitative results
D.1 Qualitative Results on POPE
Fig. and [0 each illustrate two qualitative results from POPE (29], where the ground-truth answers to the given questions are Yes and No, respectively. They also show attention distributions top- and bonked-rrnked heads by T2T Score during answer generation, and 12T Scores at the object token, with average attention for comparison. From these visualizations, we observe that heads with the highest T2T Scores consistently attend to object tokens, differing from the average pattern. Moreover,
26

---

Qwen2.5-VI.-7B POPE (COCO)
Qwen2.5-VI.-722. POPE (COCO)
Method
Random
Popular Acc.
Advers.
Random
Popular
Advers.
Acc.
F1
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Vanilla
87.6
86.0
86.6
85.1
85.5
84.0
87.7
86.2
86.6
85.1
85.6
84.1
AllPath
90.6
90.0
88.6
88.1
86.6
86.3
90.7
90.2
88.5
88.1
85.7
85.7
Table 7: Results of Qwen2.5-VL 4 model POPE
29
COCO 33 split. Our method achieves
consistent improvements across all metrics.
when the ground-truth answer is Yes, i.e., the queried object indeed exists in the image, the heads with the highest 12T Scores focus on the corresponding visual regions, again showing a distinct difference from the mean attention. These qualitative findings further demonstrate that our core metrics, the 12T Score and T2T Score can effectively identify the critical attention heads along the two pathways. This provides strong evidence that our method, AllPath, mitigates hallucinations by enhancing both pathways.
D.2 Qualitative Results on CHAIR
To further illustrate the behavior of our intervention, present qualitative examples highlighting both successful (Fig. 1T) and failure (Fig. 12 cases our AllPath on CHAIR [45]. We emphasize that method is particularly effective when the LVLM demonstrates correct reasoning just one of the two pathways (e.g., image-to-text or text-to-text). In such scenarios, our intervention adaptively enhances the pathway make more reliable, thereby reducing hallucination. However, our method is less effective when both pathways in the LVLM exhibit substantial errors, making it difficult for the intervention fully correct the output.
along
Success Case. The baseline model generates hallucination "potted plants", whereas our AllPath correctly describes the scene without hallucination. Visualization shows that when baseline is generating "potted plants", , the heads with top 12T Scores correctly attend to the flower in the image, while the heads with bottom T2T scores distribute attention across general objects such as "dog". "bench", and "fence". Our method adaptively amplified both the image-to-text and text-to-text pathways, reinforcing useful information and effectively suppressing hallucination.
Failure Case. The baseline response hallucinates two "bowls", and our AllPath still produces a hallucinated "cup". Visualization shows that when baseline generating "bowls". the heads with top 12T Score fail to focus on the relevant area in the image, while the heads with bottom T2T Score remain strongly influenced by textual priors, particularly the token "blender". Since neither pathway is reliable, our intervention could not mitigate this hallucination.
E
Limitations
This work focuses on understanding how hallucinations in LVLMs relate to their internal causal pathways. However, we do not investigate what causal pathways are most effective for different types of questions, nor we explore training strategies that could systematically improve the use of such pathways. Additionally, our analysis is constrained by the scope of existing hallucination benchmarks, which may not capture a broader range of unrecognized subtle hallucinations, These remain important open questions for future research.
27

---

Avg. Attn. 10 Heads with Lowest T2T Scores
Is there dog in the image? Please answer this question with word.
Prompt:
Meez
LLa AllPath: Yes
Avg. Alin. Heads with Highest I2T Scores
Avg. Altn, Heads with Highest T2T Scores
Micad
Spaeniee
1ee X
Ieee
Werd
15:1
55
15.1
WNT
WNT
tork
Avg. Alin. of All Heads
Avg. Attn. of All Heads
Fanc
Jcg
Yiare
V
dug
tx:
the
12x
tsugs
Theu
Maa
Maal
wedd
W
Avg. Attn. of 10 Heads with Lowest T2T Scores
Prompt:
Is there a person the image? Please
Merrz
5
1a
15r
answer this question with one word.
Read
Haa Ieee
LLaVA AllPath: Yes
Avg. Heads Highest 12T Scores
Avg.
Highest T2T Scores
Meer
17
Ine
Werd
ANT
Avg. Attn, of All Heads
Avg. Attn. of All Heads
SUARA
Bose
"Hann" 1ay
peroett
12002
Teagg
Plazse
Maa
genstorr
weee
W
5.5
555
104
Figure Qualitative visualization of attention weights of I2T and T2T heads the POPE dataset 29) Each case contains three rows and two columns. Left: The top row shows the input image, the corresponding question, and the model's predicted answer, where the object token in the question is highlighted in blue. The second and third rows visualize the attention from the object token to previous tokens: the second row shows the average attention of the 10 heads with the highest 12T scores, and the third row shows the average attention of all heads. Right: The attention weights when generating the final answer token. From top bottom: average attention of the 10 heads with the lowest T2T scores, average attention of the 10 heads with the highest T2T scores, and average attention of all heads.
28

---

Avg. Attn. 10 Heads with Lowest T2T Scores
Is there person image? Please answer this question with one word.
Prompt:
Meez
A No
Avg. Alin. Heads with Highest 12T Scores
Avg. Alnn, Heads with Highest T2T Scores
Micad
Spaeniee
1ee
Werd
X
15:1
X
55
15.1
WNT
1tm
ANT
tork
Avg. Altn. of All Heads
Avg. Attn. of All Heads
Fanc
V
Jensot
tx:
the
12x
tssgss
Thea
Plau
Maa
Maal
W
Avg. Attn. of 10 Heads with Lowest T2T Scores
Is there car in the image? Please answer this question with one word.
Prompt:
Mirrz
5
10
"aage
Mead
Haa leee
LLa AllPath: No
Avg. Heads
Highest 12T Scores
Avg. Heads Highest T2T Scores
101
Meer
17
Ma
Ine
Werd
ANT
Avg. Attn, of All Heads
Avg. Attn. of All Heads
STARA
Bose
1he
Teage
Plazae
Maa
wi1
MAC
gensterr
W
5.5
15T
355
Figure 10: Qualitative visualization of attention weights of 12T and T2T heads the POPE
dataset [29]. Fach case contains three rows and two columns. Left: The top row shows the in- put image, the corresponding question, and the model's predicted answer, where the object token in the question is highlighted in red. The second and third rows visualize the attention from the object token previous tokens: the second row shows the average attention of the 10 heads with the highest 12T scores, and the third row shows the average attention of all heads. Right: The attention weights when generating the final answer token. From top bottom: average attention of the 10 heads with the lowest T2T scores, average attention of the 10 heads with the highest T2T scores, and average attention of all heads.
29

---

Avg. Attn. Heads with Lowest T2T Scores
1000
Strgee
2115
11x2
3:40
1
1000
201
1909
Avg. Attn. of 10 Heads with Highest T2T Scores
Prompt:
descrite 11e
Please help me describe the image detail.
00
her
LLaVA (Baseline):
brgguc
T1x
80
The image features a dog sitting on a hench, possibly a park bench, with its tongue out. The dog is wearing red leash, and it appears to be enjoying its time outdoors. The bench is located near a fence, and there are several potted plants surrounding the area.
2400
1005
14.0
10
1300
Avg. Attn. Heads Highest Scores
14.41
In addition to the main dog, there are two other dogs in the scene, one located near the left edge of the image and the other towards the right side. The presence of multiple dogs suggests that this might be popular spot for pet owners to bring their pets for walk or a rest.
1erking
1-4
1000
180
2:2
LLa Alllaahh:
The image features dog sitting on a bench, possibly park bench, with its tongue out. The dog wearing leash, and it appears to be enjoying its time outdoors. The bench located near fence, and there are several potted plants surrounding the area.
scene, one located near the left edge of the image and other towards the right side. The presence of multiple dogs suggests that this might popular spot for pet owners to bring their pets for a walk or a rest.
addition main dog, there are two other dogs
Non- Non-hallucinated objects in model's response are highlighted in blue, while hallucinated objects are highlighted in red. The upper part of this figure presents the bascline output, with right column visualizes the model's average attention of different set heads when generating the hallucinated object "potted plants". From top to bottom: average attention of the 10 heads with the lowest T2T scores, average attention of the 10 heads with the highest T2T scores, and average attention of the 10 heads with the highest 12T scores.
Figure 11: Qualitative success of our intervention CHAIR 45
30

---

Avg. Atin, of 10 Heads with Lowest T2I Scores
Spage
Hind
Avg. Attn. of 10 Heads with Highest T2T Scores
The
seaad Teatecr
Prompt:
Please help me describe the image in detail.
LLaVA (Baseline):
24
The image features kitchen counter with blender sitting on The blender is filled with mixture, possibly smoothic milkshake, and ready be blended. The counter is also covered in brown substance, which could be result of the blending process spill.
Avg. Attn. Heads with Highest 12T Scores
The
In addition the blender, there are two bowls the counter. One bowl is located the left side of the blender, while the other positioned on the right side. The kitchen counter appears be bit messy, with the blender and the brown substance taking up a significant portion of the space.
10
135
LLLVA+
The image features kitchen counter with blender sitting on it. The blender is filled with mixture, possibly smoothic a smoothie-like like substance. The blender is turned on, and the blender cup filled with the mixture. The counter is also covered bit of mess, with few drops liquid spread across the counter.
Non-hallueinated objects in model's response are highlighted in blue, while hallucinated objects are highlighted in red. The upper part of this figure presents the baseline output, with right column visualizes the model's average attention of different set of heads when generating the hallucinated object "bowls". From top to bottom: average attention the 10 heads with the lowest T2T scores, average attention of the 10 heads with the highest T2T scores, and average attention of the 10 heads with the highest I2T scores.
Figure 12: Qualitative failure case of our intervention on CHAIR 45
31


Reviews:
[{"review_id": "KvslsDUhOd", "reviewer": "Reviewer_Zj3C", "summary": "The paper argues that hallucinations in Large Vision-Language Models (LVLMs) don't stem from a single error source but rather emerge from complex interactions between multiple information pathways within the model's architecture. The authors develop a comprehensive intervention framework that simultaneously targets both image-to-output-text and image-to-input-text-to-output-text causal pathways. They establish importance scores for individual attention heads across text-to-text and image-to-text pathways, revealing that heads in different pathways behave distinctly and vary depending on question formats. Through targeted intervention of attention heads—both suppressing those associated with hallucination promotion and enhancing those linked to hallucination suppression across image-to-input-text and image-to-input-to-output-text pathways—they achieve approximately 2% reductions in hallucination rates compared to baseline methods across multiple benchmarks. Ablation studies demonstrate that both causal pathways are essential for optimal performance and a single universal head configuration proves effective across different datasets.", "questions": "- Could you please elaborate on how the analysis on attention heads adds beyond the attention analysis in [1]?\n- Could you illustrate qualitative examples on where the intervention succeeds and fails?\n- There is a typo in line 162: `Image-to-input-text causal path is different from image-to-input-text causal path` – one of the `image-to-input-text` has to be corrected to `image-to-output-text`", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Jointly addressing causal pathways to treat hallucination outperforms prior single-path fixes, which is intuitive and engineered cleverly.\n- The analysis is granular and informs the method design well.\n- The method and evaluation setup is comprehensive. The authors probe both direct and indirect visual pathways across diverse question formats, two LVLM architectures (LLaVA v1.5 7B and Qwen-VL-Chat), and strong baselines, with methodology clearly documented.\n- The gains are consistent and high on multiple benchmarks.\n\n**Weaknesses**\n- The paper’s finding that the image-to-input-text and image-to-output-text pathways are governed by different sets of heads are similarly explored in previous work. [1] also separately analyzes the information flow from image-to-query tokens and from image-to-generated tokens. The idea of analyzing distinct stages of visual information processing was a known direction in the field, making the analysis aspect of the paper’s contribution incremental. \n\n[1] Kaduri et al., What's in the Image? A Deep-Dive into the Vision of Vision Language Models. *CVPR 2025*.", "was_revised": true, "final_justification": "All issues raised have been resolved. I am convinced that this submission makes unique contributions that differentiate it from the prior work mentioned in [1], and the qualitative examples effectively demonstrate when and how the proposed mechanism performs well.\nRegarding the failure scenario identified in the author rebuttal, I have suggested that the authors discuss it in the Limitations section to provide readers with a complete understanding of the method's boundaries.\n\nI will maintain my score as Accept.", "rebuttal": "Thank you for recognizing our findings on the causal pathways as well as our method and evaluation setup. We especially appreciate that you insightfully pointed out prior work [1]. Below, we provide detailed responses to the weaknesses and questions you raised:\n\n> **Weakness 1 & Question 1:** Analysis on attention heads adds beyond the attention analysis in [1].\n\nThank you for your insightful comments and questions. While our findings share some similarities with those in [1], our approach advances beyond [1] in several important aspects:\n\n1. **Broader Scope Enables the First Exploration of Hallucination in Alignment Formats.** Unlike [1], which focuses solely on the image captioning task, our study evaluate hallucination in different question formats, including yes/no tasks, multiple-choice tasks, and captioning tasks. This enables us to identify our core finding: **the importance of different pathways varies significantly across task formats. This is a phenomenon not previously observed in previous studies.** This explains why existing methods struggle to achieve consistent improvements across all benchmarks.\n2. **A Finer-Grained Metric (Our Identification of Attention Heads in Each Layer vs. Averaged Attention Weight Across Layers in [1]) Enables In-Depth Analysis and Yields Several Novel Insights.** While [1]’s primarily analysis rely on the average attention weight across layers, we focus on identifying key attention heads in each layer, which enables a more fine-grained and interpretable understanding of the underlying mechanisms. This methodological refinement leads to **several novel insights that go beyond [1]:**\n    1. **Although both yes/no and multiple-choice tasks rely on the Image-Prompt-Response pathway, they rely on very different attention heads** (see Fig. 3 the third column in paper). This reveals that even when different format of questions share a same pathway, the actual cross-modal attention heads they rely on are not the same, which advances beyond [1]’s general observations about the I-P pathway. What’s more, our method not only identifies these heads precisely, but also validates their significance via performance gains in hallucination mitigation experiments.\n    2. By identifying heads, we further uncover the detailed mechanisms in I2T and T2T interactions. While [1] observes that both input and output texts pay attention to the image (as we do in Fig. 4), **we go further by showing that the output text specifically attends to object-level keywords in the input text**, which is more obvious in yes/no and multiple-choice tasks. **These patterns are more obvious at the level of heads than in average attention.** Due to the rebuttal format's limitations, we describe this phenomenon in text, and we will provide the corresponding visualizations in the revised version of our paper.\n    3. Additionally, our head-level analysis yields other valuable insights. For example, we reaffirm that the importance of paying attention to image in mitigating hallucinations. These further supporting the contribution of our work.\n3. **Head-Level Analysis Enables Practical Utility in Hallucination Mitigation.** Because we can accurately identify hallucinatory attention heads, we are able to use these heads to effectively mitigate hallucinations and enhance performance. In contrast, [1] offers some high-level interpretability of the model’s behavior, but its findings offer less direct utility for improving the LVLM itself.\n\n> **Question 2:** Qualitative examples on where the intervention succeeds and fails.\n\nThank you for your question. Our method is particularly effective when the LVLM demonstrates correct reasoning even along just one of the two pathways (e.g., image-to-text or text-to-text). In such scenarios, our intervention adaptively reinforces the pathway to make it more reliable, thereby reducing hallucination. However, our method is less effective when both pathways in the LVLM exhibit substantial errors, making it difficult for the intervention—despite targeting certain heads—to fully correct the output. Below, we present one successful case and one failure case to illustrate this:\n\n- **Success Case**\n    - **Image**: A dog is sitting on a bench. Behind the bench, beyond a fence, there are various flowers planted directly in the soil. There are **no potted plants** in the image.\n    - **Prompt**: Please help me describe the image in detail.\n    - **Baseline Response (hallucination at ‘potted plants’)**:\n\n        The image features a dog sitting on a bench, possibly a park bench, with its tongue out. The dog is wearing a red leash, and it appears to be enjoying its time outdoors. The bench is located near a fence, and there are several **potted plants** surrounding the area.\n\n        In addition to the main dog, there are two other dogs in the scene, one located near the left edge of the image and the other towards the right side. The presence of multiple dogs suggests that this might be a popular spot for pet owners to bring their pets for a walk or a rest.\n\n    - **Our Method’s Response (no hallucination):**\n\n        The image features a dog sitting on a bench, which is located near a flower garden. The dog is wearing a red leash and appears to be enjoying its time outdoors. The dog is positioned in the middle of the bench, occupying a significant portion of the bench's length.\n\n        The flower garden is filled with various flowers, creating a beautiful and colorful garden. The flowers are scattered throughout the garden, with some flowers located closer to the dog and others further away. The garden creates a beautiful and serene environment for the dog to enjoy.\n\n    - **Behavior of Our Identified Heads.** When baseline is generating ‘potted plants’:\n        - The heads with the **highest I2T (image-to-text) scores** focused on the flower region in the background, grounding the description visually.\n        - The heads with the **lowest T2T (text-to-text) scores** distributed attention relatively uniformly across general terms like “dog”, “fence”, and “bench.”\n    - **Interpretation**:\n\n        This example demonstrates a case where the LVLM exhibited partial alignment on two pathways. Our method adaptively amplified both the image-to-text and text-to-text pathways, reinforcing useful information and effectively suppressing hallucination.\n\n- **Failure case**:\n    - **Image**: A kitchen counter with a blender and miscellaneous clutter. There are no bowls or cups present.\n    - **Prompt**: Please help me describe the image in detail.\n    - **Baseline Response (hallucination at ‘potted plants’)**:\n\n        The image features a kitchen counter with a blender sitting on it. The blender is filled with a mixture, possibly a smoothie or a milkshake, and is ready to be blended. The counter is also covered in a brown substance, which could be a result of the blending process or a spill.\n\n        In addition to the blender, there are two **bowls** on the counter. One bowl is located on the left side of the blender, while the other is positioned on the right side. The kitchen counter appears to be a bit messy, with the blender and the brown substance taking up a significant portion of the space.\n\n    - **Our Method’s Response (hallucination at ‘cup’):**\n\n        The image features a kitchen counter with a blender sitting on it. The blender is filled with a mixture, possibly a smoothie or a smoothie-like substance. The blender is turned on, and the blender **cup** is filled with the mixture. The counter is also covered in a bit of mess, with a few drops of liquid spread across the counter.\n\n    - **Behavior of Our Identified Heads.** When baseline is generating ‘bowls’:\n        - The heads with the **highest I2T (image-to-text) scores** failed to sufficiently attend to the cluttered counter area in the image where no bowls could be verified.\n        - The heads with the **lowest T2T (text-to-text) scores** still maintained strong attention to token ‘blender’, suggesting that text priors dominated the response generation.\n    - **Interpretation**:\n\n        In this case, both pathways exhibited substantial issues: the image-to-text heads failed to focus on the visually relevant regions, while the text-to-text heads remained overly influenced by strong textual priors (\"blender\"). Since neither pathway is reliable, our intervention could not mitigate this hallucination.\n\nWe will include these analysis and cases in the final version of our paper.\n\n> **Question 3 & Paper formatting concerns.**\n\nThank you for pointing out the typos and formatting issues. We will correct them carefully in the final version."}, {"review_id": "ILIuEoTpSW", "reviewer": "Reviewer_JSpS", "summary": "This paper introduces AllPath, an intervention framework designed to mitigate hallucinations in Vision-Language Models (LVLM) across diverse alignment formats. The framework employs a multi-path approach that analyzes three critical causal pathways: image2input-text, image2output-text, and text2text interactions within LVLMs. \n\nThe key innovation is a novel method for identifying critical attention heads along these pathways that contribute to hallucination generation. Empirical findings reveal that LVLMs dynamically adapt their pathway usage depending on specific question-answer formats. \n\nBuilding on this findings, the authors propose a targeted intervention method that selects optimal pathways and attention heads based on question type. The framework's effectiveness is rigorously validated through extensive experiments across multiple benchmarks, including POPE, MCQ-POPE, CHAIR, and MME, demonstrating consistent performance improvements over baseline methods in various alignment scenarios.", "questions": "- How sensitive is the AllPath method to the choice of hyper-parameters, particularly the number of selected heads?\n\n- Have you tested your approach on larger VLMs? Is the head issue present in larger VLMs?\n\n- The paper mentions that different question formats rely on distinct causal\npathways. Can you expand on it? It is unclear what is the exact difference here.", "limitations": "The authors have adequately addressed the limitations of their work. They acknowledge that their primary analysis focuses on one LVLM and provide additional results for other models in the appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThis paper introduces a comprehensive framework for reducing hallucinations in large vision-language models by analyzing multiple causal pathways, including how images and text interact within the model. \n\nThe authors provide novel insights into how these models use different pathways depending on the type of question asked. The proposed method achieves strong and consistent improvements across several standard benchmarks and question types, demonstrating its effectiveness in practice.\n\n**Weaknesses**\n\nDespite the experimental results, the paper does not provide a theoretical explanation for why the framework works. \nAdditionally, most experiments are conducted on a single model (LLaVA-v1.5-7B), so it is unclear how well the method would perform on other models, especially larger ones. Broader testing on different models would help confirm the general usefulness of the approach.", "was_revised": true, "final_justification": "The rebuttal and the additional ablations are informative. The additional experiments clearly provide evidence that this method can be applied on larger and more powerful VLMs. I'm raising my score.", "rebuttal": "Thank you for recognizing our causal pathway analysis and framework. Below, we provide detailed replies to each of the weaknesses and questions:\n\n> **Weakness 1:** The paper does not provide a theoretical explanation for why the framework works.\n\nThanks for your feedback and suggestion. Our rationale behind the overall framework, the design of the two metrics, and the intervention on attention heads to mitigate hallucinations is as follows:\n\n1. **Rationale of the Overall Framework.** Due to the causal attention mechanism of LVLMs, generation is influenced by both the image-to-text and text-to-text pathways. Therefore, we aim to analyze how each pathway contributes to hallucination and mitigate hallucinations by intervening in both paths. Within these two pathways, there are two distinct types of atomic paths, one is the image-to-text path, and the other is the text-to-text path. So we propose two separate metrics to identify the hallucinatory heads associated within each pathways.\n2. **Rationale Behind the Hallucination Metrics**:\n    - **Text-to-Text Score**. It has been suggested that unembedding the intermediate hidden states of LVLMs can approximate the model’s internal beliefs or predictions at each step [1]. Based on this, our Text-to-Text (T2T) metric uses these approximations to assess the influence of individual attention heads on generating non-hallucinated versus hallucinated responses. To achieve this, we employ the log-probability increase score to estimate the contribution of each individual head to hallucinatory and non-hallucinatory outputs. This enables us to identify attention heads that suppress or contribute to hallucinations.\n    - **Image-to-Text Score**. It has been proved that enhancing image-related attention helps mitigate hallucinations [2]. Inspired by this, we propose the I2T metric to quantify each attention head's contribution to the model's focus on image features, allowing us to identify attention heads linked to hallucinations.\n3. **Rationale behind our AllPath for hallucination mitigation** is that it precisely suppress attention heads that contribute to hallucinations while enhancing those that help the model produce correct outputs. Since we have already identified hallucination-related heads, hallucinations can be effectively mitigated through a simple strategy of reweighting these heads. Our quantitative results further validate that our identified heads play a meaningful and consistent role in hallucinations.\n\n[1] Understanding Multimodal LLMs: The Mechanistic Interpretability of LLaVA in Visual Question Answering\n[2] Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs\n\n> **Weakness 2 & Question 2:** Test on other models, especially larger ones.\n\nThank you for your valuable comment. We include results on Qwen VL, the newer and stronger model from the same series, Qwen2.5-VL-7B, and the much larger Qwen2.5-VL-72B, on POPE to further evaluate the generalizability of our method across models of different scales. Since Qwen2.5-VL-72B has five times as many heads as LLaVA, we accordingly increased the number of heads we used by a factor of five.\n\n| Qwen-VL  | ran. |      | pop. |      | adv. |      |\n| -------- | ---- | ---- | ---- | ---- | ---- | ---- |\n|          | Acc. | F1   | Acc. | F1   | Acc. | F1   |\n| Vanilla  | 84.5 | 82.3 | 83.4 | 81.4 | 80.5 | 78.9 |\n| Ours     | 86.6 | 85.3 | 85.6 | 84.3 | 82.4 | 81.5 |\n| $\\Delta$ | +2.1 | +3.0 | +2.2 | +2.9 | +1.9 | +2.6 |\n\n| Qwen2.5-VL-7B | ran. |      | pop. |      | adv. |      |\n| ------------- | ---- | ---- | ---- | ---- | ---- | ---- |\n|               | Acc. | F1   | Acc. | F1   | Acc. | F1   |\n| Vanilla       | 87.6 | 86.0 | 86.6 | 85.1 | 85.5 | 84.0 |\n| Ours          | 90.6 | 90.0 | 88.6 | 88.1 | 86.6 | 86.3 |\n| $\\Delta$      | +3.0 | +4.0 | +2.0 | +3.0 | +1.1 | +2.3 |\n\n| Qwen2.5-VL-72B | ran. |      | pop. |      | adv. |      |\n| -------------- | ---- | ---- | ---- | ---- | ---- | ---- |\n|                | Acc. | F1   | Acc. | F1   | Acc. | F1   |\n| Vanilla        | 87.7 | 86.2 | 86.6 | 85.1 | 85.6 | 84.1 |\n| Ours           | 90.7 | 90.2 | 88.5 | 88.1 | 85.7 | 85.7 |\n| $\\Delta$       | +3.0 | +4.0 | +1.9 | +3.0 | +0.1 | +1.6 |\n\n> **Question 1:** How sensitive is the AllPath method to the choice of hyper-parameters.\n\nThank you for your question. We conduct additional ablation studies on the hyper-parameters on POPE based on LLaVA 1.5 7B. The results are shown in the table below. It illustrates that the performance of our method is not sensitive to the choice of this hyper-parameter. Notably, alternative hyper-parameter settings resulted in better performance on POPE, demonstrating the robustness of our approach. Rather than tuning for a specific dataset, we selected a single set of hyper-parameters that works well across various discriminative tasks. This shows that our method is effective and stable without extensive tuning. We will include these results in the revised paper.\n\n|        |         |            |            | ran. |      | pop. |      | adv. |      |\n| ------ | ------- | ---------- | ---------- | ---- | ---- | ---- | ---- | ---- | ---- |\n| $\\xi$  | $\\zeta$ | $\\gamma^+$ | $\\gamma^-$ | Acc. | F1   | Acc. | F1   | Acc. | F1   |\n| **20** | **10**  | **2.0**    | **0.0**    | 87.2 | 86.0 | 86.0 | 84.9 | 82.8 | 82.1 |\n| **10** | 10      | 2.0        | 0.0        | 87.3 | 86.2 | 86.0 | 85.0 | 82.6 | 82.0 |\n| **30** | 10      | 2.0        | 0.0        | 88.3 | 87.7 | 86.7 | 86.3 | 82.9 | 83.0 |\n| **40** | 10      | 2.0        | 0.0        | 88.1 | 87.9 | 85.9 | 86.0 | 81.2 | 82.2 |\n| 20     | **5**   | 2.0        | 0.0        | 86.2 | 84.8 | 85.8 | 84.6 | 82.7 | 81.9 |\n| 20     | **15**  | 2.0        | 0.0        | 88.2 | 87.4 | 86.6 | 85.9 | 83.0 | 82.8 |\n| 20     | **20**  | 2.0        | 0.0        | 88.1 | 87.3 | 86.7 | 86.0 | 83.0 | 82.8 |\n| 20     | 10      | **1.2**    | 0.0        | 85.6 | 85.3 | 85.3 | 84.1 | 82.3 | 81.5 |\n| 20     | 10      | **1.5**    | 0.0        | 86.9 | 85.7 | 85.6 | 84.5 | 82.5 | 81.7 |\n| 20     | 10      | **2.5**    | 0.0        | 86.8 | 85.6 | 86.3 | 85.4 | 83.1 | 82.5 |\n| 20     | 10      | 2.0        | **0.2**    | 87.1 | 85.9 | 85.8 | 84.7 | 82.5 | 81.8 |\n| 20     | 10      | 2.0        | **0.5**    | 87.0 | 85.8 | 85.6 | 84.5 | 82.2 | 81.5 |\n| 20     | 10      | 2.0        | **0.8**    | 86.8 | 85.5 | 85.4 | 84.2 | 82.0 | 81.3 |\n\n> **Question 3:** Differences in how different question formats rely on distinct causal pathways.\n\nCausal paths including image-to-input-text, image-to-output-text, and input-text-to-output-text are always present across LVLMs (via casual attention mechanisms), but their significance varies with the question format.\n\nFor discriminative tasks like POPE, masking the image-to-output-text path results in a minor performance drop (Fig. 5 left in paper), which means that POPE requires little or much less the image-to-output-text path. However, in open-ended tasks such as CHAIR, masking the same path causes a more significant decline, indicating that this path plays a more critical role in CHAIR. Taken together, these findings indicate distinct causal paths for each task: POPE follows Input → Prompt → Response, while CHAIR introduces an additional path: Input → Response (Fig. 5, right). These have been discussed in Section 3.2 and Fig. 5 in our paper."}, {"review_id": "ltmAgZF2eh", "reviewer": "Reviewer_vdH4", "summary": "This paper focuses on mitigating large vision language model hallucinations in a head-intervented manner. The author first expands existing metrics (LPI scores) into a grouped verision (T2T, I2T), and proposes to use these metrics to analyze each head's role in model hallucinations. Based on score comparisons across different datasets with different formats, the author claims that this multi-path intervention consistently reduces hallucinations across different task formats.", "questions": "- For the open-ended QA, what model is used to judge whether an answer is a hallucination or not?\n- I know this might be a wording issue, but why do the authors think that their proposed method is using a \"causal path\"? To me, I do not understand why the whole problem and proposed solution is relevant to causality.", "limitations": "No. I think understanding LVLM hallucinations beyond existing benchmarks will be very interesting and important. I encourage the authors to explore more on this rather than focusing on a few existing datasets, which only cover a tiny aspect of Vision QA.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- **Important Problem:** The paper addresses the timely and critical problem of hallucination in LVLMs, which is a significant barrier to their reliable deployment in real-world applications.\n- **Practicality:** The proposed intervention is training-free, making it efficient and easy to apply to existing models without requiring expensive fine-tuning.\n- **Writing and Orginization:** This paper is writing in a relatively clear way, making the understanding of its intuition and algorithm easy.\n\n### Weaknesses\n\nThere are several major weaknesses that prevent this work from being accepted. Specifically, \n\n- The central analysis lacks methodological rigor. While the authors claim to uncover the roles of different \"causal-paths\" and \"heads\", the proposed score metrics (T2T and I2T) are presented as simple heuristics. There is no comprehensive explanation—either theoretical or empirical—for why these specific definitions can reliably quantify a head's contribution to hallucination. For instance, modeling a head's contribution as the log-probability difference from its isolated addition to the residual stream fails to account for the complex, non-additive interactions between multiple heads. A more rigorous approach, such as one inspired by Shapley values, would be needed to properly attribute contributions in such a system. The authors need to demonstrate what verifiable properties their scores possess; for example, just as word embeddings exhibit semantic regularities (e.g., king - man + woman ≈ queen), the authors should show what predictable, meaningful properties their scores hold.\n- The explanatory sections (3.1 and 3.2) are not self-justified and fail to build a convincing narrative. The paper frequently uses vague and anthropomorphic language to describe its findings, such as heads \"contributing,\" \"differing,\" or models being \"smarter\" (L69-78), instead of presenting precise, falsifiable claims. The term \"identified heads\" is used throughout, but there is no clear principle for *why* certain heads are selected over others beyond their ranking on an unvalidated metric. As a result, the analysis does not convincingly support the intervention that follows.\n- If I understand correctly, the whole algorithm for reducing hallucinations is basically a simple reweight of heads (L216). Were the proposed method outperform other training-free methods, I can ignore the novelty issue. Unfortunately, there are lots of training-free methods such as [1], that the authors do not compare with. Table 4 in [1] shows that for LLaVA1.5, their F1 score and Accuracy outperform the propose method on 2/3 setttings on POPE dataset. This paper is released in 2024 and its follow-ups have better performance. Given the limited novelty, I do not think that the paper meets the acceptance threshold unless their performance can be significantly better than existing approaches.\n\n[1] Reducing hallucinations in large vision-language models via latent space steering", "was_revised": true, "final_justification": "After the rebuttal, my concerns about the theoretical intuition and comparisons with existing baselines have been addressed. Therefore, I raise my rating to weak acceptance to reflect this.", "rebuttal": "We sincerely appreciate your time and effort. Below, we respond to the weaknesses and questions raised, offering clarifications and justifications to further highlight the novelty and effectiveness of our framework:\n\n> **Weakness 1 & 2:** There is no comprehensive explanation for why these specific definitions can reliably quantify a head's contribution to hallucination, and there is no clear principle for why certain heads are selected over others beyond their ranking on an unvalidated metric.\n\n### **Theoretical rationale behind our framework**\n\nOur rationale behind the overall framework and the design of the two metrics is as follows:\n\n1. **Rationale of the Overall Framework.** Due to the causal attention mechanism of LVLMs, generation is influenced by both the image-to-text and text-to-text pathways. Therefore, we aim to analyze how each pathway contributes to hallucination and mitigate hallucinations by intervening in both paths. Within these two pathways, there are two distinct types of atomic paths, one is the image-to-text path, and the other is the text-to-text path. So we propose two separate metrics to identify the hallucinatory heads associated within each pathways.\n2. **Rationale Behind the Hallucination Metrics**.\n    - **Text-to-Text Score**. It has been suggested that unembedding the intermediate hidden states of LVLMs can approximate the model’s internal beliefs or predictions at each step [1]. Based on this, our Text-to-Text (T2T) metric uses these approximations to assess the influence of individual attention heads on generating non-hallucinated versus hallucinated responses. To achieve this, we employ the log-probability increase score to estimate the contribution of each individual head to hallucinatory and non-hallucinatory outputs. This enables us to identify attention heads that suppress or contribute to hallucinations.\n    - **Image-to-Text Score**. It has been proved that enhancing image-related attention helps mitigate hallucinations [2]. Inspired by this, we propose the I2T metric to quantify each attention head's contribution to the model's focus on image features, allowing us to identify attention heads linked to hallucinations.\n\n**Furthermore, based on the transferability and strong generalization of the heads identified by our metrics, this in turn demonstrates the effectiveness of our heads.**\n\n- The heads identified on the POPE COCO adversarial subset and the MCQ-POPE COCO adversarial subset can be applied respectively across all POPE and MCQ-POPE tasks (see Table 1 in the paper);\n- Attention heads identified on POPE can be directly leveraged to improve performance on MME (see Table 2 in the paper).\n\nSuch transferability is neither trivial nor heuristic.\n\n**Regarding other metrics and joint interactions.** Notably, we have considered common techniques for interpretation like gradient-based and zero-out methods; however, these approaches are computationally expensive and may not scale efficiently in practice. On the other hand, the joint interaction between attention heads and model responses is a complex issue within the domain of interpretability, and it falls beyond the scope of our current work. Our focus is on identifying the key attention heads that influence hallucinations, aiming to effectively mitigate their occurrence.\n\n### **Empirical explanation of our framework**\n\nBased on your suggestions, we present empirical evidence demonstrating the effectiveness of our metrics:\n\n- **Text-to-Text Score**. We demonstrate the effectiveness of T2T Heads on the hallucination benchmark in Table 3. Additionally, we conduct an experiment where we amplify the identified heads. The results, shown in the table below, reveal that amplifying the top 20 heads (with the highest T2T scores) significantly reduces hallucinations, improving accuracy by at least 3.2% across each subset. Conversely, amplifying the bottom 20 heads leads to a sharp decline in accuracy, dropping by at least 44.7%. This indicates a strong correlation between T2T heads and hallucinations.\n\n    | LLaVA 1.5 7B           | ran. |      | pop. |      | adv. |      |\n    | ---------------------- | ---- | ---- | ---- | ---- | ---- | ---- |\n    |                        | Acc. | F1   | Acc. | F1   | Acc. | F1   |\n    | Vanilla                | 85.1 | 83.7 | 83.7 | 82.4 | 80.9 | 80.0 |\n    | T2T top-20 heads 8×    | 88.6 | 87.7 | 88.0 | 87.2 | 84.1 | 83.4 |\n    | T2T bottom-20 heads 8× | 40.4 | 39.8 | 35.8 | 37.0 | 33.7 | 36.7 |\n\n- **Image-to-Text Score**. We provide visual evidence for this metric in Fig. 4. The extracted I2T heads focus on object-relevant regions of the image, while using all heads results in more dispersed attention, indicating that our selected heads attend to query- and content-related visual information. We will include attention maps for all heads in the final version for comparison. The right side of Fig 4 further shows that I2T heads focus more on the image than averaging all heads, with these heads being highly correlated with hallucinations. Finally, the performance boost in Table 3 validates that the extracted heads effectively target hallucinations, with accuracy improving by at least 1.3%.\n\n> **Weakness 3:** Concerns about the novelty and performance.\n\n**The novelty of our method is multi-fold as follows:**\n1. While our method involves reweighting, **its core contribution is the identification of key heads in different types of paths**, which enables targeted reweighting to achieve substantial performance improvements. This is something that simple reweighting methods do not achieve.\n2. To the best of our knowledge, we are the first to investigate the impact of different pathways on hallucination, and we discover that **the importance of these pathways varies significantly across task formats**, offering several novel insights.\n3. Furthermore, we are the first to introduce **pathway-specific hallucination metrics for I2T and T2T heads**, and we **empirically demonstrate their effectiveness**.\n\n**On comparison with [1] and performance.** Please note that [1] uses a different prompt format and decoding setup: they adopt prompt “Is there a [object] in the image?” and use beam search (beam size = 5). In contrast, we follow the official LLaVA evaluation setup, where prompts are appended with “Please answer this question with one word.” and does not uses beam search when evaluating POPE following VCD [2] and ICD [3].\n\nWe have reproduced the performance of the VTI [1] under our setup using their official code. The results are shown in the table below. As seen, VTI underperforms both the vanilla baseline and our proposed method across all splits of the POPE benchmark. What’s more, on the CHAIR benchmark, as reported in respective papers, our method outperforms VTI by 9.2 points on $CHAIR_S$ and by 3.9 points on $CHAIR_I$, under similar baseline performances.\n\n| LLaVA 1.5 7B | ran. |      | pop. |      | adv. |      |\n| ------------ | ---- | ---- | ---- | ---- | ---- | ---- |\n|              | Acc. | F1   | Acc. | F1   | Acc. | F1   |\n| Vanilla      | 85.1 | 83.7 | 83.7 | 82.4 | 80.9 | 80.0 |\n| VTI          | 83.2 | 80.9 | 81.8 | 79.5 | 79.0 | 77.1 |\n| Ours         | 87.2 | 86.0 | 86.0 | 84.9 | 82.8 | 82.1 |\n\n| LLaVA 1.5 7B | $CHAIR_S$ | $CHAIR_I$ |\n| ------------ | --------- | --------- |\n| VTI          | 35.8      | 11.1      |\n| Ours         | 26.6      | 7.2       |\n\n[2] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding\n\n[3] Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding\n\n> **Question 1:** For the open-ended QA, what model is used to judge whether an answer is a hallucination or not?\n\nOur experiments are conducted on the CHAIR benchmark, using its official evaluation protocol for hallucination, which is a widely adopted standard in the field. Specifically, the CHAIR parser extracts object mentions from the generated captions and maps them to the 80 object categories defined in the COCO dataset. These extracted objects are then compared with the GT annotations to determine whether an object is hallucinated. This process does not involve any language models.\n\n> **Question 2:** Why the whole problem and proposed solution are relevant to causality?\n\nWe use the term **“**causal path**”** to refer to components in the model that have a causal influence on the final output, specifically on hallucination. This is not meant in the strict Pearlian sense, but rather follows an intervention-based perspective: when we intervene on a path (e.g., remove it), the output changes accordingly.\n\nImportantly, while all pathways exist in every generation, their causal effect on hallucination varies by format. For example, in POPE, the image-to-output-text path has minimal causal influence on the final answer, while in CHAIR, this path has a significant causal effect. This difference supports our claim that causal paths are format-dependent, as illustrated in Fig. 5 (right) — the former rely on the Image-Prompt-Response pathway and the latter introduces an additional path: Input → Response."}, {"review_id": "EbpmqFiAyF", "reviewer": "Reviewer_5FfF", "summary": "The authors found that the hallucinations in LVLMs do not originate from a single path, but are the result of the interaction of multiple paths, image-to-input-text, image-to-output-text, and text-to-text. The authors also found that LVLM will choose different paths according to the homogeneous format of \"question-answer\". Based on this, this article proposes Intervene-All-Paths, a method to identify and intervene in the key attention heads that drive hallucinations in each path, and to design intervention strategies for both discriminative and generative formats.", "questions": "1. Some hyperparameters mentioned in the article, such as the number of front heads, scaling factors, etc., directly give fixed values, but do not specify the basis and rationality of the setting, nor do they evaluate the sensitivity of their value changes to model performance. Maybe try to add some details about hyperparameter settings.\n\n2. The article mainly uses quantitative indicators to evaluate the effectiveness of the method. Perhaps several qualitative analysis examples can be tried to add to the appendix, such as showing the original output of the model under specific tasks and the output after using the method. This will more intuitively reflect the effectiveness of the proposed method.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1. The paper has a rigorous structure and is highly readable.\n\n2. The intervention is carried out at the reasoning stage, which does not require additional training, requires less code modification, has low deployment costs and is easier to integrate.\n\n3. The author found that LVLM automatically switches between different paths according to the format of the question, and this observation led to the final \"discriminative\" and \"generative\" intervention strategies, with a complete logical closed loop, which not only analyzes the mechanism but also guides the strategy design.\n\nWeaknesses：\n\n1. The method proposed in the article is able to adaptively select the key attention head and apply the scaling factor, but the potential impact of attention head intervention on the original accurate output of the model in the absence of \"illusion\" scenarios is not evaluated. Perhaps it is possible to consider supplementing the model's output changes before and after intervention in the hallucination-free sample. To verify the safety and stability of this method.\n\n2. In Sec.2.3, the label source of the set T_I2T is not specified in the text, which will cause readers to be unable to determine whether these labels are based on manual labels or automatically generated. It is recommended to add a detailed description of the annotation process of this data set in the article to improve the repeatability and credibility of the experiment.", "was_revised": true, "final_justification": "I would like to thank the authors for the rebuttal, which has somehow tackled weakness. I will retain my original score.\n\nBest regards,", "rebuttal": "Thank you for your recognition of our paper and our framework AllPath. Below, we provide detailed responses to the weaknesses and questions you raised:\n\n> **Weakness 1:** The potential impact of attention head intervention on the original accurate output of the model in the absence of \"illusion\" scenarios is not evaluated. \n\nThank you for your valuable feedback. To assess the robustness of our approach on originally correct cases, we analyze the POPE COCO benchmark. The table below shows the number of originally correct and incorrect cases and the changes after applying our method. Our results demonstrate that our method effectively suppresses hallucinations, while its impact on correct (non-hallucinated) cases is negligible, with a change of only 1.5%. This indicates that our method exhibits high stability and safety when applied to non-hallucinated inputs.\n\n| Before \\ After | Hal.         | Non-Hal.     |\n| -------------- | ------------ | ------------ |\n| Non-Hal.       | 116  ( 1.5%) | 7378 (98.5%) |\n\n> **Weakness 2:** Label source of the set T_I2T In Sec. 2.3.\n\nThe set $\\mathcal{T}\\_\\text{I2T}$ consists of tokens corresponding to the first occurrence of each object mentioned during the question-answering process, either in the input text $P$ or in the generated response $R\\_{\\leq t}$ (Appendix B.1).  Specifically, for short-answer questions like POPE and MCQ-POPE, the selected tokens are from the input text, while for open-ended questions like CHAIR, they are drawn from the generated response. In our implementation, we use the same parser for the CHAIR evaluation to automatically extract objects and remove synonyms, without relying on external models or manual annotations.\n\nThe rationale behind this design is that only the first mention of an object should attend directly to the image, while subsequent references can rely on prior textual context. We will include these details and clarifications in the final version of the paper.\n\n> **Question 1:** Details about hyperparameter settings.\n\nThank you for your valuable feedback. We include ablation experiments for the head and scaling factor. The results, as shown in the table, indicate that some alternative hyperparameter settings even resulted in better performance on POPE, which shows that our method is effective and stable without extensive tuning.\n\n|        |         |            |            | ran. |      | pop. |      | adv. |      |\n| ------ | ------- | ---------- | ---------- | ---- | ---- | ---- | ---- | ---- | ---- |\n| $\\xi$  | $\\zeta$ | $\\gamma^+$ | $\\gamma^-$ | Acc. | F1   | Acc. | F1   | Acc. | F1   |\n| **20** | **10**  | **2.0**    | **0.0**    | 87.2 | 86.0 | 86.0 | 84.9 | 82.8 | 82.1 |\n| **10** | 10      | 2.0        | 0.0        | 87.3 | 86.2 | 86.0 | 85.0 | 82.6 | 82.0 |\n| **30** | 10      | 2.0        | 0.0        | 88.3 | 87.7 | 86.7 | 86.3 | 82.9 | 83.0 |\n| **40** | 10      | 2.0        | 0.0        | 88.1 | 87.9 | 85.9 | 86.0 | 81.2 | 82.2 |\n| 20     | **5**   | 2.0        | 0.0        | 86.2 | 84.8 | 85.8 | 84.6 | 82.7 | 81.9 |\n| 20     | **15**  | 2.0        | 0.0        | 88.2 | 87.4 | 86.6 | 85.9 | 83.0 | 82.8 |\n| 20     | **20**  | 2.0        | 0.0        | 88.1 | 87.3 | 86.7 | 86.0 | 83.0 | 82.8 |\n| 20     | 10      | **1.2**    | 0.0        | 85.6 | 85.3 | 85.3 | 84.1 | 82.3 | 81.5 |\n| 20     | 10      | **1.5**    | 0.0        | 86.9 | 85.7 | 85.6 | 84.5 | 82.5 | 81.7 |\n| 20     | 10      | **2.5**    | 0.0        | 86.8 | 85.6 | 86.3 | 85.4 | 83.1 | 82.5 |\n| 20     | 10      | 2.0        | **0.2**    | 87.1 | 85.9 | 85.8 | 84.7 | 82.5 | 81.8 |\n| 20     | 10      | 2.0        | **0.5**    | 87.0 | 85.8 | 85.6 | 84.5 | 82.2 | 81.5 |\n| 20     | 10      | 2.0        | **0.8**    | 86.8 | 85.5 | 85.4 | 84.2 | 82.0 | 81.3 |\n\n> **Question 2:** Qualitative analysis examples.\n\nThank you for the suggestion. In response, we provide one specific qualitative example below. We will include qualitative examples in the appendix, showing the model's original and modified outputs for specific tasks, to better illustrate the effectiveness of our method.\n\n- **Image**: A dog is sitting on a bench. Behind the bench, beyond a fence, there are various flowers planted directly in the soil. There are **no potted plants** in the image.\n- **Prompt**: Please help me describe the image in detail.\n- **Baseline Response (hallucination at ‘potted plants’)**:\n\n    The image features a dog sitting on a bench, possibly a park bench, with its tongue out. The dog is wearing a red leash, and it appears to be enjoying its time outdoors. The bench is located near a fence, and there are several **potted plants** surrounding the area.\n\n    In addition to the main dog, there are two other dogs in the scene, one located near the left edge of the image and the other towards the right side. The presence of multiple dogs suggests that this might be a popular spot for pet owners to bring their pets for a walk or a rest.\n\n- **Our Method’s Response (no hallucination):**\n\n    The image features a dog sitting on a bench, which is located near a flower garden. The dog is wearing a red leash and appears to be enjoying its time outdoors. The dog is positioned in the middle of the bench, occupying a significant portion of the bench's length.\n\n    The flower garden is filled with various flowers, creating a beautiful and colorful garden. The flowers are scattered throughout the garden, with some flowers located closer to the dog and others further away. The garden creates a beautiful and serene environment for the dog to enjoy.\n\n- **Behavior of Our Identified Heads.** When baseline is generating ‘potted plants’:\n    - The heads with the **highest I2T (image-to-text) scores** focused on the flower region in the background, grounding the description visually.\n    - The heads with the **lowest T2T (text-to-text) scores** distributed attention relatively uniformly across general terms like “dog”, “fence”, and “bench.”\n- **Interpretation**:\n\n    This example demonstrates a case where the LVLM exhibited partial alignment on two pathways. Our method adaptively amplified both the image-to-text and text-to-text pathways, reinforcing useful information and effectively suppressing hallucination."}]

Author rebuttal:

Meta-review: This paper presents a training-free, head-level intervention framework that mitigates LVLM hallucinations by jointly targeting image-to-text and text-to-text causal pathways and shows these pathways shift with question format. Reviewers praised the novelty, consistent 2-4% gains across POPE, CHAIR and MME on LLaVA-v1.5, Qwen-VL and 72B Qwen2.5-VL, and the practical plug-and-play design; concerns about theoretical grounding, single-model focus, hyper-parameter sensitivity and comparison with prior steering methods were largely resolved through additional ablations, larger-model results and qualitative examples. Based on the above reason, this paper is recommended to be accpted.
Decision: Accept (poster)