SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding
Authors: Woohyeon Park
Woojin Kim
Jaeik Kim
Jaeyoung Do
Venue: ICML 2025 poster (2025)
Keywords: Vision-Language Model
Hallucination
Selective and Contrastive Decoding
Primary area: applications->computer_vision
TL;DR: We propose SECOND (Selective and Contrastive Decoding) to tackle perceptual hallucination in LVLMs by iteratively selecting and contrasting multi-scale visual information.

Abstract: Despite significant advancements in Vision-Language Models (VLMs), the performance of existing VLMs remains hindered by object hallucination, a critical challenge to achieving accurate visual understanding. To address this issue, we propose SECOND: Selective and Contrastive Decoding, a novel approach that enables VLMs to effectively leverage multi-scale visual information with an object-centric manner, closely aligning with human visual perception. SECOND progressively selects and integrates multi-scale visual information, facilitating a more precise interpretation of images. By contrasting these visual information iteratively, SECOND significantly reduces perceptual hallucinations and outperforms a wide range of benchmarks. Our theoretical analysis and experiments highlight the largely unexplored potential of multi-scale application in VLMs, showing that prioritizing and contrasting across scales outperforms existing methods.

SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding
Woohyeon Park
Woojin Kim
Jaeik Kim
Jaeyoung Do 12
Abstract
Q: there dog the image?
Despite significant advancements in Vision- Language Models (VLMs), the performance of existing VLMs remains hindered by object hallu- cination, a critical challenge to achieving accurate visual understanding. To address this issue, we propose SECOND: Selective and Contrastive De- coding. novel approach that enables VLMs ef- fectively leverage multi-scale visual information with an object-centric manner, closely aligning with human visual perception. SECOND progres- sively selects and integrates multi-scale visual information, facilitating a more precise interpre- tation of images. By contrasting these visual in- formation iteratively, SECOND significantly re- duces perceptual hallucinations and outperforms wide range of benchmarks. Our theoretical analysis and experiments highlight the largely unexplored potential of multi-scale application in VLMs, showing that prioritizing and contrast- ing across scales outperforms existing methods. Code is available https: htt///gtttt/tt com/ AIDASLab/ SECOND.
LLaVA-NeXT
SECOND
Pash Selection
Solection
Patch Selaction
Visual Patches
Visual Patches
Figure I. The comparison IL LLaVA-NeXT (Liu et al., 2024a) SECOND in utilizing visual patches. While LLaVA-NeXT uni- formly incorporates patches at two scales, SECOND accumulates object-relevant patches through patch selection strategy.
nation-a a phenomenon where models generate plausible yet inaccurate responses that are not grounded in the visual content Content-remaing, remains a persistent and critical challenge (Bai et al., 2024; Leng et al., 2024). This limitation poses sig- nificant barrier to their adoption in high-stakes applications demanding reliability and interpretability.
1. Introduction
LsiiunLanguugg Models (VLMs) have revolutionized Arti- ficial Intelligence (Liu et al., 2024b;a) by effectively bridg- ing vision and language modalities, enabling breakthroughs in tasks such as visual question answering (VQA) (Antol al., 2015), image captioning (Vinyals et al., 2015), and open-vocabulary semantic segmentation (Zhang al., 2023). These advancements have been driven by cross-modal align- ment strategies and the integration of large-scale vision and language data. Despite these achievements, object halluci-
Several studies have explored methods mitigate hallueina- tion in VLMMs. These approaches include designing robust datasets to reduce model biases (Liu et al., 2023; Hu et al., 2023), suppressing the influence of LLM priors (Huang et al., 2024; Leng et al., 2024; Kim et al., 2024a), and im- proving the performance of vision models integrated into VLMs (Tong et al., 2024; Chen et al., 2024b). Moreover, re- cent VLMs, such as LLaVA-NeXX (Liu al., 2024a), have incorporated multi-scale vision techniques (Li al., 2024; Wang et al., 2024a), which have been effective in traditional computer vision tasks like object detection (Guan et al., 2022) and semantic segmentation (He et al., 2022b). How- ever, prior research has largely focused on single-scale vi- sual representations, overlooking the full potential of multi- scale integration.
'Department of Electrical and Computer Engineering, Seoul Na- tional University, Seoul, South Korea Interdisciplinary Program in Artificial Intelligence, Scoul National University, Scoul, South Ko- rea. Correspondence to: Jacyoung Do <jacyoung do@ snu.ac..k. kr²>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
In this work, we align with the evolving trends in VLMS by optimizing multi-scale vision techniques address object hallucination effectively. Our analysis reveals that current multi-scale VLM designs often integrate visual information indiseriminately across different scales, which can impede precise extraction of dense, object-specific details. This challenge arises due to the fundamentally spatial nature of visual data in contrast to textual data- increasing the number of visual patches through up-scaling can lead to exponential information loss (He et al., 2022a).
including POPE (Li et al., 2023), VQAv2 (Antol al., 2015), MMStar (Chen et al., 2024a), and MM- Bench (Liu et al., 2025), highlighting its effectiveness.
2. Related Works
2.1. Vision-Language Model and Hallucination
The remarkable advancements in VLMs have been driven by the development of models such as LLaVA (Liu et al., 2024b), Instruct-BLIP (Dai et al., 2023), and Mini- GPT4 (Zhu et al., 2023), which have effectively bridged the modalities of vision and language. Despite these ad- vancements, perceptual hallucination-a phenomenon in which VLMs generate compelling yet ungrounded object re- sponses- remains persistent challenge (Li et al., 2023). To mitigate this issue, various approaches have been explored. Training-bai based methods aim to enhance model robustness, designing robust dataset (Liu et al., 2023; Hu et al., 2023), employing contrastive loss (Jiang et al., 2024), or adopting Reinforcement Learning (RL) (Zhao et al., 2023; Sun et al., 2023). On the other hand, pre/post-processing methods are served in training- frree manner. these works address hal- lucination leveraging auxiliary vision model (Tong et al., 2024; Chen et al., 2024b), refining visual input representa- tions (Huang et al., 2024; Leng et al., 2024), or indiscrimi- nately accumulating multi-scale visual features (Liu et al., 2024a; Li et al., 2024). Specifically, our approach identifies the limitation of the indiscriminate utilization of sclti-scale vision, and introduces an optimal strategy for selectively integrating multi-scale fine-grained visual features without requiring additional training or auxiliary models.
To address these limitations, we propose SECOND (Selective and Contrastive Decoding), a novel, training-fre frre framework designed to address object hallucination by leveraging multi- scale visual information with human-inspired selectivity and contrast (Purves et al., 2001). SECOND dynamically priori- tizes object-relevant patches and refines visual representa- tions through an iterative, multi-stage decoding process. By selectively incorporating coarse-to-fine -graine visual in- formation and focusing on high-resolution relevant patches, SECOND enhances both visual fidelity and interpretability, effectively mitigating hallucination. As illustrated in Fig. 1, this process systematically filters out irrelevant patches, en- abling the model to transition from broad-level scanning to fine-grained examination.
In addition, the multi-stage process of SECOND fosters a synergistic interaction between amateur and expert outputs, thereby introducing multi-stage Contrastive Decoding. The initial stage acts as an amateur, performing coarse image processing, while the final stage serves as an expert. deliv- ering detailed and comprehensive analysis. This approach enhances step-by-step progression of SECOND by contrast- ing the expert, which utilizes fine-grained and relevant in- formation from the image, with several amateurs across multiple stages. Unlike prior works on Contrastive De- coding in VLMs, which primarily focus on suppressing language priors (Leng al., 2024; Kim et al., 2024a; Wang al., 2024b), SECOND leverages selective contrast between sparse and dense visual information to effectively mitigate perceptual hallucinations. As result, the expert output in the final stage of SECOND outperforms several baselines independently, and moreover, incorporating contrastive de- coding further amplifies the improvement of performance (For more details, see Sec. 5).
2.2. Contrastive Decoding
Contrastive Decoding (CD) (Li et al., 2022), which contrasts the outputs of strong expert models with those of weak am- ateur models has emerged as a fundamental approach in LLMs, enhancing generation quality while ensuring fac- tual consistency. Building on the success of contrastive decoding in LLMs, this framework has been extended to VLMs, with primary focus on enhancing factual ground- ing in multi-modal generation tasks. VCD (Leng al., 2024) utilizes noised images to distort visual inputs for contrastive output distributions, reducing biases and im- proving alignment with visual inputs. CODE (Kim et al., 2024a) employs self-generated visual descriptions as con- trasting references to align textual outputs with visual con- tent. While these approaches primarily focus on suppressing the prior knowledge of LLMs, HALC (Chen et al., 2024b) contrasts the specific cropped focal of image to enhance vi- sual representation. Recent works further expand the single- step contrastive decoding paradigm through adaptive visual augmentations (Kim et al., 2024b; Park et al., 2025), di- verse contrastive sampling and fusion strategies (Lee et al.,
We propose SECOND, a novel, training- free framework that adaptively selects multi-scale patches and contrasts sparse visual data with fine- ine-grained details.
Our analysis of multi-scaled visual patches in VLMs offers insights into optimizing performance by balanc- ing coarse and fine-grained visual information.
Extensive experiments demonstrate that SECOND out- performs several baselines across diverse benchmarks,
2

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
2024), attention-guided ensembling (Cho al., 2025), and token-level introspective filterig (Huo al., 2025). Other approaches employ classifier-free grounding and prompt- based umplification (Wan et al., 2025; Favero et al., 2024b) to enhance multimodal alignment. In contrast to these meth- ods, which rely on a single contrastive reference, we em- ploy a multi-stage Contrastive Decoding, leveraging each intermediate output in SECOND as a contrastive reference, amplifying the distinction across multiple scales.
Hallucinated Non-Haalucinattd
0.7
0.6
0.5
Rabio
0.4
0.3
0.2
0.1
3. Core Principles
0.0
0.0-0.1
0.1 -0.2
0.2-0.3
0.3-0.4
0.4-0.5
Phal Range
3.1. Problem Definition
(a)
VLMs, such as LLaVA (Liu et al., 2024b), tokenize visual information from image patches and integrate with textual instructions. Given visual information v, an instruction x, and the model's generated answer y. we define the halluci- nation probability the likelihood that y diverges from the relevant information in (Favero et al., 2024a; Guerreiro et al., 2022).
Hallucinated Non-Hallen Non-Hallucciaated
Attention Dice Coefficient
022
Definition 3.1. Hallucination probability. Given the prob- ability P(y U, x) in VLMs, where denotes the generated text sequence for the given query x and visual contents v, the probability of object hallucination defined as PHat. representing the likelihood that the model does not generate the response y based on the given query and visual content:
0.0
0.0
0.1
0.2
0.3
0.4
05
Pial
(b)
Figure 2. (a) Distribution of PH with the LLaVA model on POPE. (h) Attention Dice Coefficient on Pitl
Paul 1 P④y 22
[
(Thm. 3.2), inspired by the widely used Dice Coefficient in image segmentation (Shamir et al., 2019), where a higher value indicates that the VLM is properly focusing on the relevant object.
= 1 II t=1
P(u (,I,,i<4)
where t and L each denote the timestep of the generation process and total length of generated tokens, with y, repre- senting the token generated at timestep & and Vict represent- ing all previously generated tokens.
Our approach integrates self-attention maps from the vi- sion encoder and cross cotss-attention maps between generated output tokens. To compute unified attention signal that reflects both visual and textual modalities, we multiply these two attention maps element wise. To assess this, we query the LLaVA model with "Is there a/an {object) in the im- age?" on MSCOCO (Lin et al., 2014), which provides mask annotations for each object. As shown in Fig. 2(b), we observe a strong correlation between the Attention Dice Coefficient and hallucination probability. Specifically, hallu- cinated responses (red points) tend to have lower Attention Dice scores and greater hallucination probabilities compared to non-hallucinated responses (blue points). These findings support our hypothesis that model's ability to focus accu- rately on target objects significantly reduces hallucination risk, summarized in the following observation:
Fig. 2(a) shows the distributions of PiLa for hallucinated and non-hallucinated answers by LLaVA on the POPE benchmark. Non-hallucinated responses predominantly ex- hibit lower hallucination probabilities, while hallucinated responses are more uniformly distributed. This distribu- tional disparity highlights that reducing PHil is essential for mitigating hallucination, aligning with Def. 3.1.
3.2. Theoretical Insights and Analysis
For VLMs generate accurate responses based on visual content, we hypothesize that the model's ability to precisely focus on the target object plays a crucial role in determin- ing the probability of hallucination (i.e., PHa)). To quanti- tatively analyze this, we measure the alignment between the model's attention and the object's 's segmentation mask. Specifically, we introduce the Attention Dice Coefficient
Observation: Pita decreases when model's visual attention aligns with the groundtruth object mask.
3

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
Based on this observation, we theoretically validate recent advancements in VLMS that aim to enhance visual infor- mation by utilizing small-sized patches (i.e., visual scaling) with respect to their Attention Dice Coefficient scores as detailed in Thm. 3.2.
selection increases the Attention Dice Coefficient, thereby reducing Pial-
Theorem 3.3. Let Dice denote the Attention Dice coef- ficient at stage & in a multi-stage framework. Multi-stage stage patch selection guarantees that the Attention Dice Coef- ficient at next stage, + 1 is lower-bounded by the Dice coefficient at stage s. such that:
Theorem 3.2. Consider an image composed of n patches, each of size m m pixels, where a E [0, 1] represents the visual attention score of the i-th patch. The ground truth object mask assigns value of to pixels within the object region and O to the background. The average ground truth value for each patch is denoted as gi E [0, 1]. The Attention Dice Coefficient for an m x m pixel-sized patch is defined as follows:
Dice (8) am S Dice (s+1) (
Proof. See Appendix A.
4. Methodology
2 Σ
a49i i=1
Building on the theoretical foundations outlined in Sec. 3, we propose SECOND (Selective and Contrastive Decoding), novel approach designed to mitigate perceptual hallucina- tions within a multi-stage framework. SECOND selectively integrates mne-grained information across stages based on the visual attention mechanisms of VLMS (Sec. 3.2) and en- hances representation by contrasting enriched dense visual information with sparse representations at each stage.
Dice
1
Σ 04 + Σ 9i
i=1
Given m' m. follows that Dice
amm when the model's attention correctly aligns with the object mask.
< Dice
Proof Sketch. VLMs tend to show increased Diceam in the absence of hallucinations (Fig. 2(b)), indicating higher atten- tion a is assigned to patches, where containing higher value of g. Assume that each pixel in the ground truth mask fol- lows a Bernoulli distribution with probability of p E [0, 1]. Then, as the patch size increases, the mean of g, remains p while its standard deviation decreases as /p(1 p)/m m.
4.1. Selective Multi-Scale Feature Integration
Since SECOND relies on effectively capturing multi-scale vi- sual features, it requires an expansion of the vision encoder's s scale. However, most existing models typically operate at only one or two scales (Liu et al., 2024b;a), and training additional multi-scale vision encoders for each VLM entails excessive computational costs. Therefore, a training- free ap- proach for handling multiple resolutions is highly desirable. Inspired by prior studies (Dosovitskiy, 2020; Beyer et al., 2023), we employ bilinear interpolation on the positional embeddings of the vision encoder and down-sample input images to generate visual tokens at lower scales. By append- ing vision encoder that processes down-sampled features to the existing model, we achieve multi-scale integration without additional training. Although the standalone perfor- mance of the down- wwwn-s vision encoder lower than the baseline, it remains effective for multi-scale integration, as demonstrated in Appendix B, proving its suitability for enhancing visual processing across multiple scales.
Consequently, the most of 9, values concentrate around p, reducing variability and limiting the Attention Dice Coef- ficient's growth. In contrast, smaller patch sizes allow for more dynamic Attention Dice allocation, supporting recent VLM advancements that improve object focus and reduce hallucinations by up-scaling visual content.
However, simply reducing patch size to scale up the model has inherent limitations in improving performance (He et al., 2022a; Wang et al., 2024a; Beyer et al., 2023). This limita- tion is due to the indiscriminate scaling of all visual patches, which increases irrelevant information while diluting object- specific visual context. As the number of patches grows, we observe that VLMs struggle precisely align their visual at- tention with ground-truth object masks. This misalignment induces perceptual hallucination by increasing the density of not only object-related but also background information. Instead of uniformly reducing patch size across the entire image, balaneing relevance and quantity of visual content proves more effective in enhancing the Attention Dice Co- efficient. To achieve this, we propose multi-stage patch selection, where object-relevant patches are iteratively se- lected at each scale and integrated. This approach preserves the density of object-relevant information while curbing the proliferation of patches unrelated to the object's 's focus.
For a detailed implementation, we apply SECOND on recent VLMS as a baseline, leveraging their scale-aware trained capabilities to maximize performance in a training-free set- ting and effectively adapt to the emerging trend of visual scaling in VLMs. We choose the maximum length of stages as 4 to balance decoding computational efficiency and per- formance, which will be discussed in detail in Sec. 5.4. Given a visual encoder's 's base resolution before scaling up x W with patch size m x m, SECOND incorporates two additional visual encoders into the early stages: one processing a resolution of H/4 x W/4 for the first stage
In Thm. 3.3, we theoretically prove that multi-stage patch
4

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
STAGE 1
STAGE
STAGE
STAGE 4
Is there dog the image?
Pathh
nech
Pach
LVLMs
SECOND
Orgetlogh
The
Expert
Yes.
Contrastive Decoding
Figure Architecture of SECOND. Starting with sparse visual information Stage 1, SECOND incorporates higher scales' fine-grained information selectively based on visual attention. By contrasting each interval of stages, SECOND achieves the final output effectively mitigating perceptual hallucination. See Sec. 5.3 for more detailed view of visual attention.
and another processing H/2 x W/2 for the second stage, both maintaining the same patch size. The vision encoder that processes the base resolution images baseline VLMs constitutes the third stage in SECOND. For some models utilizing double-scale, such LLaVA-NeXT, subsequently form the fourth stage, while single-scale models applied 3- stage design. This multi-stage approach preserves the scale gap between stages, effectively aligning with the existing twofold scaling utilized VLMss. Furthermore, we adopt SECOND on VLMs utilizing CLIP (Radford et al., 2021) and SigLIP (Zhai al., 2023). which are most widely used vision encoders. In the interpolation process, the embedding of the cls token in CLIP is preserved and does not participate in the interpolation, whereas SigLIP's positional embedding is fully interpolated due to the absence of cls token. This modification allows the model to process low-resolution coarse images with fewer patches while gradually allocat- ing more patches to high-resolution fine-grained images, thereby increasing visual density more robustly with the patch selection strategy described below.
patch selection mechanism that combines a hyperparame- ter A, with the entropy of the visual attention H (V). Low entropy in the visual attention indicates a clear distinction between the attention on objects and the background, sug- gesting that fewer additional patches are required for the next stage. In contrast, high entropy indicates that the model fails to focus appropriately on specific objects, necessitating the incorporation of more fine fine-graid grained information in the subsequent stage. Therefore, the patch selection percentage Pselect is designed as below:
Peelect exp(A H(V)) -1
eap(A)
By selecting Psetect patches with highest visual attention in each stage, SECOND intensively accumulates coarse-to- fine-grained visual information through multiple stages. A detailed algorithmic description of the patch selection pro- cess is provided in Appendix D.
4.2. Multi-Stage Contrastive Decoding
As shown in Fig. 3, the process begins at Stage by utilizing only the smallest scale vision encoder. At each subsequent stage, patches for larger-scale processing are selectively cho- sen based on the visual attention generated in the previous stage. As the stages progress, visual attentions at different scales are resized and accumulated align with the atten- tion map of the highest resolution. To selectively filter out object relevant patches, Pselect, we introduce a dynamic
Contrastive Decoding (CD) (Li et al., 2022) builds upon the perspective that when powerful expert model fails to perform optimally, weaker amateur model is likely to encounter even more pronounced failures. By contrasting the output logits of these two models with hyperparameter a, single-stage CD amplifies the output quality of expert model as following:
logit logit + allgitt logit
5

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
Table Results of POPE in SECOND and baselines. SECOND outperformed in 11 of 12 highlighted cases. Inside the parentheses below each model name indicates the type of vision model and its base resolution. Method
Model
LLM
CD
MSCOCO
OKVQA
GQA
Recall (1)
Acc. (t) fl (1)
Recall (t)
Acc. (1)
fl (t)
Recall (t)
Acc. (t)
fl (1)
baseline
78.8
87.7
86.5
86.7
89.1
88.8
84.8
86.6
86.3
VCD
81.1
88.2
87.3
88.3
88.0
88.1
86.3
84.6
84,9
Vicuna-7B
SECOND
80.1
88.6
87.5
87.6
89.9
89.6
84.9
86.5
86.3
VA-Next (CLIP-336)
SECOND
85.1
89.7
89.2
90.5
90.3
90.4
85.5
89.4
87.4
baseline
80.2
88.3
87.3
88.2
88.7
88.7
88.2
84.2
84.8
80.8
87.4
88.3
88.0
84.5
85.1
Mistral-7B
SECOND
79.5
88.1
86.9
88.2
87.2
84.8
85.2
SECOND
84.8
89.3
88.8
92.5
89.9
90.7
92.1
85.3
87.5
baseline
80.0
88.4
87.4
85.6
89.3
88.9
83.1
86.8
86.3
LLaVA-OneVision
79.7
87.4
86.4
86.7
89.0
88.7
84.3
86.7
86.4
Qwen2-0.5B
(SigL.PP-384)
SECOND
78.1
87.6
86.3
83.8
88.7
88.1
82.2
87.4
86.7
SECOND
79.7
87.9
89.3
89.1
83.3
87.8
87.2
baseline
70.3
82.0
79.6
77.0
84,0
82.8
74.5
81.0
79.7
Yi-VL (CLIP-448)
VCD
73.0
80.1
78.6
79.1
82.1
81.6
78.2
79.9
79.5
Yi-6B
SECOND
X
73.5
83.5
82.0
80.1
85.6
84.8
76.6
82.5
81.4
SECOND
83.4
84.5
84.3
87.7
86.3
86.5
83.3
82,8
82.9
Obviously, typical CD can be applied SECOND, utilizing the initial stage output amateur and the final expert. However, our focus is the multi-stage design of SECOND, which generates hierarchically structured outputs that progressively highlight the differences across stages.
5. Experiments
To evaluate SECOND, we implement SECOND on three re- cent VI.Ms. First, L.LVV--NXTT (Liu et al., 2024a), an enhanced version of LLaVA, incorporates one additional scaled-up features using CLIP vision model with resolu- tion of 336. We evaluate this model with both the Vicuna-7B and Mistral-7B LLMs. LLa Onevision (Li et al., 2024), similar to LLaVA-NeXX in applying a single level of scale- up but differs in employing SigLIP-384 as the vision encoder instead of CLIP-336. To validate the efficacy of SECOND on smaller LLMs, we conduct experiments using Qwen2- 0.5B. Unlike the previous two models, Yi-VL (Young et al., 2024) employs single-scale image features. We apply SEC- OND to this single-scale model to show its effectiveness even in models trained with single-scale features. Further- more, we incorporate VCD (Leng et al., 2024), a Contrastive Decoding framework that utilizes noised image, enabling comparative analysis on other decoding method. For the several hyperparameters in SECOND, we serve the optimal settings in Appendix C, further analyzing the hyperparame- ter sensitivity in Sec. 5.5.
Through patch selection across multiple stages, SECOND progressively integrates fine-grained visual information, re- sulting in increasingly refined outputs. The outputs gen- erated at each stage reflect the evolutionary progression from the initial to the final stage (Thm. 3.3), forming performance-ordere hierarchy. Leveraging these priori- tized multiple outputs, SECOND facilitates the application of multi-staged CD. This framework effectively amplifies the difference of visual information driven by scale interval and patch selection rather than simply contrasting the most amateur and expert (Sec. 5.4). To utilize the evolutionary trajectory of stage-wise outputs, propose multi-staged CD framework incorporating contrasting hyperparameters o, 3 and y defined within the range of 0 to as follows:
logitsSCOND logit expent
+ a alogit
logit amatour3)
5.1. Benchmarks and Evaluation Metricss
+ +(ogit ((oggit urnateur3
logit amateur2)
POPE POPE (Li et al., 2023) is widely adopted bench- mark that specializes in identifying perceptual hallucination by querying the presence of specific objects in given image through simple yes/no questions. It employs recall, accu- racy, and score as the primary evaluation metrics and includes 3k questions derived from well-known datasets such as MSCOCO (Lin al., 2014), A-OKVQA (Schwenk et al., 2022), and GQA (Hudson & Manning, 2019). In this study, evaluated the models using the popular split of the POPE benchmark.
+ (logit amateur?
logit
By applying this Multi-staged CD, SECOND amplifies the ef- fectiveness of patch selection at each stage. This enables the comparison between the inferences derived from sparse vi- sual information in the early stages and those obtained from progressively incorporated dense and fine-grained visual information. As a result, this approach facilitates significant performance improvements in object perception tasks.
6

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
Table Result of various benchmarks including VQAv2(lite), MMStar, and MMBench(lite). Inside the parentheses below each model
name indicates the type of vision model and its resolution.
Model
LLM
Method
CD
VQAv2 (lite)
MMStar
MMbench (lite)
baseline
76.4
37.3
75.8
VCD
72.9
38.1
74.2
Vicuna-7B
SECOND
X
76.5
37.5
78.0
VLAVA-Next (CLIP-336)
SECOND
77.5
38.6
80.0
baseline
X
72.0
32.4
74.2
VCD
70.1
34.0
71.2
Mistral-7B
SECOND
X
73.6
34.1
71.2
SECOND
74.5
36.2
70.5
baseline
X
74.6
38.9
73.5
I.L.VA-One
VCD
55.0
36.2
70.5
Qwen2-0.58
(SigLIP-384)
SECOND
73.6
39.6
72.7
SECOND
75.1
39.9
73.5
haseline
64.3
34.8
77.3
Yi-VL (CLIP-448)
VCD
61.9
34.4
79.5
Yi-6B
SECOND
63.6
37.4
82.6
SECOND
65.3
39.8
84.8
General Tasks For the general tasks, VQAv2 (Antol et al., 2015) serves as a benchmark for evaluating VLMSs ability generate answers for given quegeequeettonn pairs. VQAv2 measures performance through exact match evaluation, requiring models to provide answers in the form of yes/no, numerical, or open-ended responses. We evaluate the lite version consisting of 0.5k questions, which encompass wide range of categories, including object presence, at- tributes, and relationships. MMStar and MMBench (Chen al., 2024a; Liu et al.. 2025) provide comprehensive evalu- ations by categorizing VI.M tasks into primary domains: perception and reasoning. These benchmarks further divide main categories into several sub-categories, enabling de- tailed assessment of model performance across variety of tasks. MMStar comprises 1.5k questions, while MM- Bench's lite version includes 0.5k samples. These bench- marks collectively offer holistic view of VLMs' capabili- ties, ensuring balanced evaluation across both low-level perception and high-level reasoning tasks.
Original
Stage
Stage
Stage
Stage
Is there a tennis racker the image?
Q: What the surn of the digits on either end of the sign in the corner?
What part of statuc's outfit isa't made the material statuc?
Figure 4. Evolutionary attention refinement SECOND. Red areas represent high attention scores. SECOND gradually enhances visual attention, demonstrating robustness across diverse object scales.
5.2. Main results
ended questions. Furthermore, the performance improve- ments observed in MMStar and MMBench, which include both perception and reasoning tasks, highlight that SEC- OND'S approach does not negatively impact the reasoning capabilities of LLMs. These results also suggest that im- provements in perception performance can translate into benefits for higher-level tasks such as reasoning, underscor- ing the holistic advantages of SECOND.
Results on POPE As presented Tab. across the ex- perimental configurations of POPE, SECOND outperformed the baseline and VCD method in 11 out of 12 cases. These results provide compelling evidence that SECOND effec- tively mitigates perceptual hallucinations while substantially enhancing the object recognition capabilities of VLMs.
Results on General Tasks addition to general tasks, as illustrated in Tab. SECOND outperformed most basclines. The enhanced performance observed in VQAv2, which in- volves object perceptual queries in an open-ended format, indicates that SECOND is effective not only in single-word closed-answer tasks like POPE but also in handling open-
5.3. Robustness of SECOND
Leveraging multi-scale visual information that has not been seen during training, and executing multiple stages a sequential manner, requires that the algorithm operates ro- bustly. Therfore, demonstrate the robustness of SECOND

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
Table Effect of individual stages POPE MSCOCO henchmark performance and decoding time with LLaVA-NeXT
0.
M0N0ua Acveaas
0tt
Stages
POPE (fl)
Decoding Time
pare
Single Stage
Single Stape
14:33
Muili Stage Beat
23.35
Muki Stage Best
W.O. CD
w. CD
42-84-1 88-888--33806 168-336-672
87.6
89.3
1.4x
07.9
84-168-336-672
87.5
89.2
42
CD (Single Stage)
8-4
035
42
CDu (Stngle Stape)
168-336-672
87.5
88.4
0.7x
(a)
(b)
336-672
87.2
87.7
0.6x
Figure 5. Results POPE CD-a. (b) Results MM- Star, comparing single CD to the best of multi-stage CD.
Table Effect of diverse patch selection strategy on POPE MSCOCO benchmark performance with LLTVA-NeXT
Stages
Patch Selection
POPE (fI)
84-168-336--72
dynamic
87.5
84-168-336-672
reversed
70.6
84-168 84-168-336-677
fixed (30%)
78.0
MA00000
84-168 94-168-336-67 672
fixed (50%)
83.8
POUE AOAXYOA
F022 ODA
6.5
84-168-336-672
fixed (70%)
86.1
1884-168-336-672
all (100%)
87.2
(a)
(b)
84-168-336-67
all (100%)
87.1
168-336-672
all (100%)
87.1
Figure 6. Hyperparameter sensitivity analysis. (a) in patch se-
lection. (b) 3 and in multi-stage CD. The red line in (b) represents the performance of SECOND w.o. CD.
in Fig. 4. In the first example, although the side of the tennis racket is captured and thus poorly visible, the attention map is eventually corrected as fine-grained information added in later stages. In the second example, the small text on the upper-right wall initially receives no attention from the model. However, the stages progress, the relevant visual cues are gradually emphasized. Because relevant area is not attended on the first stage, Contrastive Decoding can even amplify the probability of generating the correct an- swer. the final example, the model initially over-focused on the shoulder area of the statue in the first stage. As the stages proceed, this unnecessarily strong attention weakens, demonstrating refinement of focus.
decoding time complexity (1.4x compared to 84). Therefore, the 84- 14-168-336-672 configuration is selected as the baseline, which balances performance and decoding efficiency.
Selection Strategy Analysis To validate our patch selec- tion mechanism and its theoretical basis for reducing halluci- nations, we perform an ablation study summarized in Tab. 4. The default dynamic selection strategy across four stages achieves the highest performance (87.5), demonstrating its adaptability in selecting salient regions. Using all patches at varying stages consistently results in lower scores, high- lighting the critical importance of selective patch utilization in optimizing performance and minimizing hallucinations. Fixed selection strategies (e.g. 30%, 50%, and 70%) further underscore this point, with the 70% fixed strategy achieving 86.1 but still falling short of the dynamic approach.
5.4. Ablation study
Impact of Stage Configuration In our multi-stage gen- eration process, we conduct ablation studies to systemati- cally analyze the influence of stage configuration, justify- ing the selection of our default setting. Tab. 3 provides a detailed analysis of how different stage configurations im- pact POPE benchmark performance and decoding time with LLaVA-NeXX Each column under Stages corresponds to the specific multi-scale resolutions utilized in the generation process. The results show that the performance without contrastive decoding (w.o. ed) is relatively stable across configurations, though notable performance gap exists be- tween the 14-1683336-67 and 168-336-672 settings when contrastive decoding is applied. While the inclusion of the 42 resolution slightly improves performance (0.1 in POPE), the effect not significant enough to justify the increased
Single vs. Multi Stage Contrastive Decoding To evalu- ate the advantages of multi- stage CD over single-stage, we compare their performance under varying configurations of the CD-a parameter, which is adjusted between 0 and 1. In the single-stage approach, stage logits are used as the primary outputs, while stage logits serve as the contrastive reference, following the adapted procedure described in Sec. 4.2. Fig. 5 shows that single-stage CD achieves com- parable performance to multi-stage CD in POPE, reaching the same peak value of 89.2 with optimal CD-a. However, for the more general mmstar metric, multi-stage CD out- performs single- stage CD, achieving a higher peak value of 38.8, demonstrating its robustness broader scenarios.
8

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
5.5. Hyperparameter Sensitivity
Acknowledgements
Patch Selection parameter In the patch selection pro- cess, introduce the hyperparameter A, which controls the number of regions selected within the visual attention as incremental patches for the next stage. As the value of increases, fewer regions are selected. Fig. 6(a) illustrates the sensitivity of SECOND to varying À values. This anal- ysis, combined with the preceding evaluation of selection strategies, highlights the importance of selecting patches at an appropriate ratio to achieve optimal performance. Prac- tically, we observe that À = 1.0 achieves consistently high performance across tasks (Tab. 6 in Appendix C); there- fore, we recommend using À = 1.0 as the default when deploying SECOND.
This work was supported in part by National Research Foundation of Korea (NRF) grant RKS-2025-0056076, RS- 2024-0041498]] Institute of Information & communica- tions Technology Planning & Evaluation (IITP) grant (IITP- 022--SS-002400039068. RS2021-1121 1113) and Seoul National University grant (Creative-Pioneering Researchers Program, Research Grant 0418-20240053) J. Do is with ASRI, Scoul National University.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Multi-stage Contrastive Decoding parameters the multi-stage Contrastive Decoding framework of SECOND, the number of hyperparameters scales with the number of stages. In the default 4-stage configuration of SECOND, three key hyperparameters, a, 3, and are used. As shown in Fig. 6(b), these parameters are explored by varying their values between and in increments of 0.1, resulting in 1,331 configurations. All configurations consistently show superior performance compared to the baseline LLaVA- NeXT, with 98.7% surpassing the non-Contrastive Decod- ing, which relies solely on patch selection.
References
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. Vqa: Visual question answering. In Proc. of Int'l Conf. on Computer Vision (ICCV), pp. 2425-2433, 2015.
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., and Shou, M. Z. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 2024.
6. Conclusion
In this paper, we propose SECOND (Selective and Con- trastive Decoding), novel framework that leverages multi- scale visual patch selection in Vision-Language Models (VLMs) and contrasts visual information across varying densities. Based on our analysis of perceptual hallucina- tions, SECOND demonstrates remarkable improvements in objuct-focused experiments while effectively mitigating hal- lucinations. These advancements enhance the foundational capabilities of VLMs for higher-level tasks and provide valu- able insights into the application of multi-scale approaches in VL.Ss. Despite these promising results, SECOND'S 's de- pendence on internal attention maps can fail when attention is poorly calibrated. Future work should therefore con- centrate on strengthening the core selection mechanism to ensure relevant patches are reliably identified.
Beyer, L... Izmailov, P., Kolesnikov, A., Caron, M., Korn- blith, S., Zhai, X., Minderer, M., Tschannen, M., Alab- dulmohsin, I., and Pavetic, F. Flexivit: One model for all patch sizes. In Proc. of Computer Vision and Pattern Recognition (CVPR), pp. 14496-14506, 2023.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y.. Lin, D., et al. Are we the right way for evaluating large vision-langue language models? arXiv preprint 2403:::2003233340 2024a.
Chen, Z., Zhao, Z., Luo, H., Yao, H., Li, B., and Zhou, J. Hale: Object hallucination reduction via adaptive focal- contrast decoding. arXiv preprint arXhv:2403.00423 2024b.
Cho, Y., Kim, K., Hwang, T., and Cho, Do you keep an eye on what ask? mitigating multimodal hallucination via attention-guidee ensemble decoding. In Proc. of Int' l Conf. on Learning Representations (ICLR), 2025.
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general- purpose vision- languaee models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.0673 2, 2023.

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q-, Zhang, J., Huang, F., and Zhang, S. Hallucina- tion augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 27036-27046, 2024.
Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A., and Soatto, S. Multi- modal hallucination control by visual information ground- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14303-14312, 2024a.
Kim, J., Kim, H., Kim, Y., and Ro, Y. M. Code: Con- trasting self-generated description to combat halluci- nation in large multi-modal models. arXiv preprint AxXi::24060019200 2024a.
Favero, A., Zancato, L., Trager, M., Choudhary, S., Per- cra, P.. Achille, A., Swaminathan, A., and Soatto, S. Multi-Modal Hallucination Control by Visual Informa- tion Grounding. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2024b.
Kim, S., Cho, B., Bae, S., Ahn. S., and Yun, S.-Y. VACode: Visual augmented contrastive decoding. In Trustworthy Multi-modal Foundation Models and Al Agents (TiFA), 2024b.
Guan, T., Wang, J., Lan, S., Chandra, R., Wu, Z., Davis, L., and Manocha, D. M3detr: supeettrpresnnnttinn,, multi- scale, mutual-relation 3d object detection with transform- ers. In Proc. of Winter Conf. on Application of Computer Vision (ECCV), pp. 772-782, 2022.
Lee, Y.-L., Tsai, Y.-H., and Chiu, W.-C. Delve into visual contrastive decoding for hallucination mitiga- tion of large vision-language models. arXiv preprint arX1v:2412.06775, 2024.
Guerreiro, N. M., Voita, E., and Martins, A. Looking for needle in a haystack: comprehensive study of hallu- cinations in neural machine translation. arXiv preprint arXiv:2208.05 35000, 2022.
Leng, S., Zhang, H., Chen, G., Li, Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proc. of Computer Vision and Pattern Recognition (CVPR), pp. 13872-13882, 2024.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proc. of Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, 2022a.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang. K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXi::2408.03326, 2024.
He, X., Zhou, Y., Zhao, J., Zhang, D., Yao, R., and Xue, Y. Swin transformer embedding unet for remote sensing image semantic segmentation. 60:1-15, 2022b.
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T.. Zettlemoyer, L., and Lewis, M. Con- trastive decoding: Open-ended text generation as opti- mization. arXiv preprint arXiv:2210. 15097, 2022.
Hu, H., Zhang, J., Zhao, M., and Sun, Z. Ciem: Contrastive instruction evaluation method for better instruction tuning. arXiv preprint ar arXiv:2309.023011 2023.
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.108 10355, 2023.
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J. Lin, D., Zhang. W., and Yu, N. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrosecction-allcatioo In Proc. of Computer Vision and Pattern Recognition (CVPR), pp.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
13418-13427, 2024.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
Liu, F., Lin, K., Li, L., Wang. J., Yacoob, Y., and Wang, L. Mitigating hallucination in large multi-modi models via robust instruction tuning. In Proc. of Ini'l Conf. on Learning Representations (ICLR). 2023.
Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., and Zhao, P. Self-introspective decoding: Alleviating hallucinations for large vision- Langugge models. In Proc. of Int'l Conf. on Learning Representations (ICLR), 2025.
Liu, H., Li, C.,, Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proc. of Computer Vision and Pattern Recognition (CVPR), pp. 26296 263306, 2024a.
10

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tuning. Proc. of Neural Information Processing Systems (NeurIPS), 36, 2024b.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang. J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 1rXiv:2409.1219, 2024a.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., al. Mmbench: Is your multi-modal model an all-around player? In Proc. of European Conf. on Computer Vision (ECCV), pp. 216-233, Springer, 2025.
Wang. X., Pan, J., Ding, L., and Biemann, C. Miti- gating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18711 2024b.
Park, Y., Lee, D., Choe, J., and Chang, B. Convis: Con- trastive decoding with hallucination visualization for mit- igating hallucinations in multimodal large language mod- els. Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.0445 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proc. of Int' l Conf. on Computer Vision (ICCV), pp. 11975-11986, 2023.
Purves, D., Augustine, G. J., Fitzpatrick, D., Hall, W. C.,, LaMantia, A.-S., McNamara, J. O., and White, L. E. (eds.). Neuroscience. Sinauer Associates, Sunderland, MA, 2 edition, 2001. URL https: httpps::/www... /www//www//o ncbi. nlm. nih. wwrih.gov/eee 2tto:/bbokk/NBB1107000
Zhang, H.,, Li, F., Zou, X., Liu, S., Li, C., Yang, J., and Zhang, L. A simple framework for open-vocabulary seg- mentation and detection. In Proc. of Iut'l Conf. Com- puter Vision (ICCV), pp. 1020-1031, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In Proc. of Int'l Conf. on Learning Representations (ICLR), pp. 8748 8763. PMLR, 2021.
Zhao, Z., Wang. B., Ouyang, L., Dong. X., Wang, J., and He, C. Beyond hallucinations: Enhancing Ivlms through hallucination--awanl direct preference optimization. arXiv preprint urXiv:2311 1. 16839, 2023.
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. A-okvqa: A benchmark for visual ques- tion answering using world knowledge. In European conference computer vision, pp. 146-162. Springer, 2022.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision- laiionl understanding with advanced large language models. arXiv preprint 2rX4v:2304.1035 2023.
Shamir, R. R., Duchin, Y., Kim, J., Sapiro, G., and Harel, N. Continuous dice coefficient: method for evaluating prob- abilistic segmentations. arXiv preprint arXiv: 1906. 1031, 2019.
Sun, Z., Shen, S., Cao. S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint ar arXiv:2309. 14525, 2023.
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proc. of Computer Vision and Pattern Recognition (CVPR), pp. 9568-9578, 2024.
Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. Show and tell: A neural image caption generator. In Proc. of Computer Vision and Pattern Recognition (CVPR), pp- 3156-3164, 2015.
Wan, D., Cho, J., Stengel-Eskin, E., and Bansal, M. Con- trastive region guidance: Improving grounding in vision- language models without training. In Proc. of European Conf. on Computer Vision (ECCV), pp. 198- 215, 2025.

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
A. Proof of Theorem 3.3
Theorem 3.3 Let Dice denote the Attention Dice coefficient at stage in a multi-stage framework. Multi-stage patch selection guarantees that the Attention Dice Coefficient at next stage, s is lower-bounded by the Dice coefficient at stage such that:
(5) Dice sit S Dice (s+1)
The updated attention score a (s+11 by selectively incremented patches from stage to stage (s + 1) is given by:
a (s+1) = a²()) + Ao.
Thus,
3 (a! + Aa,)gi
22
(e)
Dice (s+1)
(a + Da;) + Σ 91
1=1
Replace number of patches n with number of pixels N. Then,
(5) Aaj)gs
2 (aj
Dice
(++1)
N Σ [a + Day) + j=1
N Σ j=1
Claim: Attention Dice coefficient at next stage, + 1 is lower-bounded by the Dice coefficient at stage
Diceame S Diceam
(a+1) $
N
2
9j
2 Σ
(a
(a)
+ Aay)g)
==1
N
N
N
N
Σ
+
Σ 9j
Σ (a) + Aaa)
+ Σ gj
1=1
jml
5m1
Split domain within mask or not(~~ mask). Then,
2 Σ (5)
2 Σ (a, + Acx)gg
aj
9j
mask
mask
Σ mask
a +
Σ a -mask
+
Σ mask
gj
Σ (a) + Aaz) + Σ (a) + Daa)+ + E gj
mask
~mask
mask
Supposing selected patches are only on the mask with g == 1,
2 Σ
2 Σ (a)t) + Aaj)
mask
mask
Σ 0 mask
+ Σ maas
a
(*)
+ area(mask)
Σ (af5)) Aay)) + Σ (af))) + area(mask)
mask
smask
0 0 Σ (1 x
Σ (a(+) + area(mask)).
mask
~mask
12

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
B. Experiment down-sampled Vision Encoders
This section details the performance each stage when used independently. evaluate each stage with LLaVA-NeXT on POPE MSCOCO popular split and MMStar in Tab.
Table 5. Performance of down-sampled vision encoders in stand-alone selting
Stages
POPE (Acc.)
MMStar
84
51.2
25.9
168
74.4
34.5
336
84.9
37.4
672
88.3
34.0
C. Optimal Hyperparameter Settings in SECOND
This section details in the optimal settings of hyperparamerts in SECOND. Tab. 6 details in the optimal settings of patch selection hyperparameter λ, and Tab. 7 details in the multi-stage Contrastive Decoding hyperparameters, 8, and In Yi-VL, 7 is not utilized as the model is inherently designed for single-stage process and does not include Stage 4.
Table 6 Optimal settings of patch selection hyperparameter X.
Model
LLM
POPE
VQAv2
MMStar
MMBench
Vicuna-7B
0.5
7.0
LLxVA-NeXT
Mistral-7B
1.0
3.0
1.0
1.0
Qwen2-0.5B
1.0
1.0
1.0
1.5
Yi-VL
Yi-6B
2.0
4.0
1.0
3.0
Table Optimal settings multi-stage CD) hyperparameters a, 8, and 7--
Model
LLM
parameter
POPE
VQAv2
MMStar
MMBench
MSCOCO
A-OKVQA
GQA
0.7
0.3
1.0
0.1
1.0
0.4
Vicuna-7B
3
0.7
0.7
0.7
0.8
LLaVA-NeXT
7
1.0
0.4
1.0
0.1
0.4
0.6
0.5
0.9
0.9
0.1
0.1
0.1
Mistral-7B
8
0.5
1.0
1.0
0.3
0.3
0.4
7
0.4
0.4
0.6
0.1
0.4
0.4
0
0.4
0.5
0.5
0.2
0.1
0.1
Qwen2-0.5B
8
0.6
1.0
0.8
0.3
0.2
0.4
7
0.8
0.7
0.6
0.4
0.1
Yi-VL
Yi-6B
0
0.6
0.1
1.0
0
0,9
0.7
1.0
0.2
0.7
0.6
13

---

SECOND: Mitigating Perceptual Hallucination in VLMs via Selective and Contrastive Decoding
D. Patch Selection Algorithm
Algorithm Patch Selection Algorithm
Require:
Stages list stages, image feature image masks masks, instruction 2; patch selection hyperparameter X,
output Model response y.
Initialize image masks for each stage
2: for stage in stages do
3: if stage Stage then
4: masks|stage] ones()
// All-ones tensor for the first stage
5: else
6: mrasss(slage] <- zeros()
I/ All-zeros tensor for the subsequent stages // Process each stage, compute attentions, and update masks
7: end if
8: end for
9:
10: for stage in stages do
11: - (o x masks masks[stage]... 2))
12: if stage Final Stage then
13: break
// Stop if this is the last stage
14: end if
15:
Self + Get vision model's self-attention
CrossAttr < Get language model's cross-attention
18:
VisualAAtrr +- zeros()
I/ Initialize VisualAm to All-zeros tensor
19:
for weight in Cross do
20:
Visual + VisualAttn + (weight x SAA)
21:
end for
22:
H + get entropy Cetropy(Visuuii (Visual Attn)
expl ()
24: Pselect exp( -1)
Calculate patch selection percentage Psefect in Sec. 4.1
25: threshold +- topk( Visuul Atln, aoiisiallAti X Psetect) values]-1]
26: masks(stapy + 1] <- (Visual Attn threshold)
27:
end for
E. Hallucination probabilty on each stage
Fig. 7 (a) and (b) illustrates percentage of PHtll each stage with without hallucination.
Confidence with Ha Ilucinatioon Stages)
Confidenco Without Ha Hallucination Stages)
Stuge
Stage
Stage
Peromntage of Coses (50
60
Slage
Perountage of Cases (%6
Stage
Stage
Stage
Stage
Stage
40
20
20
0.5-0.6
0.6-0.7
0,7-0.8
0.800
0.9-1.0
0.5-0.6
0,6-0.7
0.7-0.8
6.8-0.9
0.8-1.0
Coufidence Range
Confidence Bange
(a)
(b)
Figure 7. (a) Percentage of PHat on each stage with hallucination. (b) Percentage of Pitat without hallucination.
14


Reviews:
[{"review_id": "R1T4Kkidl5", "reviewer": "Reviewer_KN5t", "summary": "This paper introduces SECOND, a training-free approach to mitigate perceptual hallucination in LVLM.  More specifically, it progressively refines (by patch selection) multi-scale visual information in an object-centric manner, and uses multi-stage contrastive decoding to reduce perceptual hallucinations.  Results show it outperforms baselines across diverse benchmarks.\n\n## update after rebuttal\n\nThe rebuttal has clarified my concerns. I am happy to maintain my original recommendation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KN5t,\n\nThanks for your valuable feedback! We provide point-by-point responses to address your concerns below.\n\n**W1: More tasks could be introduced to verify SECOND’s effectiveness on common VLM tasks, such as captioning, document understanding, infographics reasoning etc.**\n\nThank you for this valuable suggestion. To further substantiate the effectiveness of SECOND on a broader range of common VLM tasks, we have conducted additional experiments on MMMU_Pro, complementing the previously reported results on MMStar and MMBench.\nThese benchmarks collectively provide a comprehensive evaluation across a diverse set of task categories, including instance relation reasoning, diagram reasoning, calculation, and scene understanding—covering many of the task types you mentioned. The new result of MMMU Pro, presented below, offer additional evidence supporting the robustness and generalizability of SECOND across diverse VLM challenges.\nThis additional experiment has been added to Appendix of the main manuscript. We hope this additional evaluation addresses your concern.\n\n\n|  | CD | MMMU_Pro |\n| --- | :---: | :---: |\n| LLaVA-NeXT Vicuna 7B | X | 16.1 |\n| LLaVA-NeXT Vicuna 7B + SECOND | X | 16.7 |\n| LLaVA-NeXT Vicuna 7B + SECOND | O | 16.9 |\n| LLaVA-OneVision Qwen 0.5B | X | 14.5 |\n| LLaVA-OneVision Qwen 0.5B + SECOND | X | 15.0 |\n| LLaVA-OneVision Qwen 0.5B + SECOND | O | 15.0 |\n\n**W2: It’s unclear how efficient the method is, with additional multi-stage computation**\n\nThank you for highlighting this important point. To address concerns regarding the computational efficiency of SECOND, we have measured and compared the per-token generation time of our method against the baseline VCD and the method OPERA[1] (a widely used approach for mitigating VLM hallucination, proposed by Reviewer i6dn).\nThe results are summarized as follows:\n\n|  | VCD | SECOND(3-stages) | SECOND(4-stages) | OPERA |\n| --- | :---: | :---: | :---: | :---: |\n| LLaVA-NeXT Vicuna 7B (sec / token) | 0.41 | 1.28 | 1.80 | 5.00 |\n| Yi-VL 6B (sec / token) | 0.26 | 0.73 | - | 4.79 |\n\nThese measurements demonstrate that while SECOND introduces a modest increase in computation due to its multi-stage patch selection, the increase remains manageable. Importantly, this additional computational cost is well-justified by the consistent performance improvements on hallucination mitigation and general VLM performance across our experiments. We will include these results in the revised manuscript to provide a clearer picture of SECOND’s efficiency.\n\n[1] \"Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation.\" CVPR. 2024.", "claims_and_evidence": "The experiments, evaluations, analysis and theory proof support its claim.", "methods_and_evaluation_criteria": "The methods involve two core components: 1) adaptive multi-scale patch selection, 2) multi-stage contrastive decoding. Ablations show both are important in mitigating perceptual hallucinations.\n\nThe authors use the POPE benchmark to evaluate perceptual hallucination, and use VQAv2, MMStar, and MMBench for general tasks. However, more tasks could be introduced to verify SECOND’s effectiveness on common VLM tasks, such as captioning, document understanding, infographics reasoning etc.", "theoretical_claims": "The authors raise the hypothesis that a model’s ability to focus accurately on target objects significantly reduces hallucination risk, which was proved via Attention Dice Coefficient experiments. Then it proves that multi-stage patch selection increases the Attention Dice Coefficient, thereby reducing the probability of hallucination.", "experimental_designs_or_analyses": "The authors implement SECOND on three recent LVLMs: LLaVA-NeXT, LLaVA-OneVision, and Yi-VL; evaluate these models on the POPE benchmark for perceptual hallucination and VQAv2, MMStar, and MMBench for general tasks. Qualitative results are also presented to give readers insight.", "relation_to_broader_scientific_literature": "This paper successfully extends the concept of Contrastive Decoding from LLMs to LVLMs, which might be insightful for the community.\n\nThe theories and proofs presented in this paper also provide valuable insights for the community.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths\n- The ideal is novel and further strengthened by the theory proof.\n- The paper is well written and easy to read.\n- Theories are presented clearly.\n\nWeaknesses\n- It’s unclear how efficient the method is, with additional multi-stage computation.\n- As mentioned above, more common VLM tasks can be added to verify the effectiveness of the methods, such as in document instead of semantic understanding.", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nIPlLSmHkZ", "reviewer": "Reviewer_Lfoe", "summary": "In this paper, a decoding method for LVLMs named SECOND is proposed. SECOND consists of selective multi-scale feature integration and multi-stage contrastive decoding. The first method, selective multi-scale feature integration leverages multi-scale feature map with patch selection scheme, where important patches are progressively selected in high-resolution features based on lower resolution attention values. Multi-stage contrastive decoding contrasts outputs obtained with multi-scale features, thereby prioritizing outputs of relative ‘experts’ with better feature maps.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Lfoe,\n\nWe greatly appreciate your valuable feedback on our paper. We address the raised concerns and questions below.\n\n**W1: L153 (left column): The implication of the attention Dice coefficient could be briefly introduced for better clarity.**\n\nThank you for the insightful comment. To improve clarity, we have revised main manuscript by adding a brief explanatory sentence, “A higher Attention Dice coefficient indicates that an LVLM properly focuses on objects”. This addition will help readers better understand the meaning and importance of the Attention Dice Coefficient in our theorem.\n\n**W2: The paper states that “hallucinated responses tend to have higher Attention Dice scores.” However, in Fig. 2 (b), hallucinated responses appear to have lower scores compared to non-hallucinated responses. Is this a mistake in the paper, or have I misunderstood?**\n\nThank you for pointing this out. You are correct — there is a mistake in the original text. Hallucinated responses actually tend to have lower Attention Dice Coefficient, as correctly shown in Fig. 2 (b). We have corrected the sentence to reflect this and apologize for the confusion.\n\n**W3: L255 (left column): The explanation of how visual attention is generated and interpreted is not detailed enough. Additionally, the selection process is overly simplified. While Section D of the supplementary material provides further details, it would be better to introduce more information in the main paper and explicitly reference the exact section of the supplementary material.**\n\nWe are grateful for your constructive suggestion. To clarify, we describe the computation of attention in Section 3.2 (for calculating the Attention Dice Coefficient), and we adopt the same formulation in our methodology (Sec. 4).\nThe visual attention used to filter patches is derived from two sources:\n1.\tSelf-attention maps from the vision encoder of the LVLM, which operate over 2D spatial image patches.\n2.\tCross-attention maps from the LLM decoder, which capture the attention from generated textual tokens to the visual tokens (i.e., image patches).\nTo compute a unified attention signal that reflects both visual and textual modalities, we multiply these two attention maps element-wise. Since the cross-attention is defined over flattened visual tokens (i.e., in a 1D format), we first map the 1D token-based attention values back to their corresponding 2D spatial coordinates to align with the vision encoder’s attention layout. This alignment enables meaningful element-wise fusion of the two attention maps.\nThis fused attention allows us to effectively estimate the contribution of each image patch to the generated tokens, which we then use to guide patch filtering in our method. We will revise the manuscript to make this computation process more explicit and improve overall clarity.\n\n**W4: Essential References Not Discussed**\n\nWe sincerely thank the reviewer for highlighting this important references. We fully agree that recent decoding-based approaches—such as VASparse, VACoDe, ConVis, SID, and Ensemble Decoding (ED)—have made important contributions to hallucination mitigation in vision-language models (VLMs). To provide a more comprehensive and contextualized discussion, we have incorporated these references into the Sec. 2. Related Works of the revised manuscript. Furthermore, we have added explicit comparisons between each of these methods and our proposed SECOND approach in the main text, highlighting key distinctions in terms of decoding strategy, use of contrastive signals, reliance on external augmentations, and computational characteristics. We hope this addition strengthens the clarity of our contributions and the positioning of our work within the existing literature.", "claims_and_evidence": "The authors’ claims are supported with proper observations and analyses.", "methods_and_evaluation_criteria": "The proposed methods are sound. Also, experiments on POPE benchmark and various multimodal benchmarks supports the method.", "theoretical_claims": "The authors make some theoretical claims that are supported with observations and proofs.", "experimental_designs_or_analyses": "Proper ablation studies are provided, which validates the effectiveness of each method.", "relation_to_broader_scientific_literature": "This work provides a simple yet effective method to enhance LVLMs by effectively utilizing multi-scale features. Also, the observations provided in the paper help understanding problems of LVLMs.", "essential_references_not_discussed": "Since there are multiple works [1-6] utilizing contrastive decoding to mitigate hallucination problem of LVLMs, discussion with those works should be provided.\n\n[1] Huo et al., Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, ICLR 2025\n\n[2] Zhuang et al., VASparse: Towards Efficient Visual Hallucination Mitigation for Large Vision-Language Model via Visual-Aware Sparsification, CVPR 2025\n\n[3] Cho et al., Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding, ICLR 2025\n\n[4] Kim et al., VACoDe: Visual Augmented Contrastive Decoding, ICML Workshop 2024\n\n[5] Park et al., ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models, arXiv 2024\n\n[6] Lee et al., Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models, arXiv 2024", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Although the paper is generally well-written, there are some areas that could be improved:\n\n1. L153 (left column): The implication of the attention Dice coefficient could be briefly introduced for better clarity. For example, consider adding a sentence like: *“A higher attention Dice coefficient indicates that an LVLM properly focuses on objects.”*\n2. L163 (left column): The paper states that *“hallucinated responses tend to have higher Attention Dice scores.”* However, in Figure 2-(b), hallucinated responses appear to have **lower** scores compared to non-hallucinated responses. Is this a mistake in the paper, or have I misunderstood?\n3. L255 (left column): The explanation of how visual attention is generated and interpreted is not detailed enough. Additionally, the selection process is overly simplified. While Section D of the supplementary material provides further details, it would be better to introduce more information in the main paper and explicitly reference the exact section of the supplementary material.", "questions_for_authors": "Please kindly address concerns in other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wtQsSdiaAH", "reviewer": "Reviewer_Ekip", "summary": "This paper focuses on how to improve the hallucination of MLLMs. Firstly, the paper conducts some analysis of hallucinations in MLLMs, proposing two metrics, namely the Hallucination Probability and the Attention Dice Coefficient, and introducing the research motivation of needing to enhance the model's visual perception of objects. Subsequently, this paper presents a training-free method to alleviate the issues of MLLMs, named SECOND (Selective and Contrastive Decoding). This method first utilizes the model's self-attention to perform inference serially on multi-scale images, screening out the patches of the main objects of interest for the next stage of inference. Then, it fuses the inference logits of multiple stages and decodes them through contrastive decoding to obtain the final result. Finally, comparative experiments are carried out on the proposed method modules using three models, namely LLaVA-Next, LLaVA-OneVision, and Yi-VL. Additionally, ablation experiments are conducted on hyperparameters in multiple models to demonstrate the effectiveness of SECOND.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ekip,\n\nWe really appreciate your thorough review of our paper. We address the raised concerns and questions below.\n\n**W1: The formula definition of Hallucination probability proposed in Sec. 3 seems to lack support from relevant theoretical papers (not mentioned in the paper)**\n\nWe sincerely appreciate your feedback. The definition of Hallucination Probability in Sec. 3 builds on prior works [1, 2], where the probability of a valid sequence is defined as the product of token-level probabilities:\n\n\\\\(p(y|x,c) = \\prod^T_{t=1} p(y_t|y_{<t}, x, c), \\quad \\text{where } y_{<t} \\triangleq [y_0, ..., y_{t−1}]. \\\\)\n\nWe define Hallucination Probability as its complement, $1 - p(y|x,c)$, representing the likelihood of generating a hallucinated response. We have revised the manuscript to include these references and clarify the motivation.\n\n[1] \"Multi-modal hallucination control by visual information grounding.\", CVPR. 2024.\n\n[2] \"Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation.\" ACL. 2023.\n\n**W2: Only use Fig. 2 (a) It cannot be proven that 'Non aggregated responses preferentially exhibit lower hallucination probabilities'.**\n\nThank you for raising this important point. As you noted, Fig. 2(a) shows that $P_{\\text{Hal}}$ is typically lower for non-hallucinated responses, while hallucinated ones exhibit a more uniform distribution. We agree that this does not imply hallucinated responses always have higher $P_{\\text{Hal}}$.\nHowever, the converse holds empirically: responses with higher $P_{\\text{Hal}}$ are more likely to be hallucinated. For instance, in the [0.4, 0.5) range, the valid-to-hallucinated ratio is approximately 1:9, indicating a strong correlation. Additionally, Fig. 2(b) shows that higher $P_{\\text{Hal}}$ aligns with lower Attention Dice scores, suggesting weaker visual grounding.\nOur aim was to show that such high-risk responses can be mitigated by SECOND, which both reduces $P_{\\text{Hal}}$ and improves grounding. We will revise the manuscript to clarify this interpretation and better explain the rationale.\n\n**W3: a) the model part includes more advanced qwen2-vl and intervl2 b) the benchmark part includes MMUL/MMMU pro/MegaBenchmark, etc**\n\nThank you for your insightful comment. Due to the limited time during the rebuttal period, implementing SECOND on newly released models posed some challenges. Nevertheless, we have conducted additional experiments to compare SECOND with a broader range of more advanced models.\n|  | POPE (pop, f1) |\n| --- | :---: |\n| Qwen2-VL 7B | 87.9 |\n| InternVL2 4B | 87.3 |\n| InternVL2 8B | 86.7 |\n| InternVL2 26B | 87.8 |\n| Ivy VL 3B | 87.5 |\n| Ovis 8B | 88.6 |\n| LLaVA-NeXT Vicuna 7B + SECOND | **89.2** |\n| LLaVA-NeXT Mistral 7B + SECOND | **88.8** |\n\nAs shown above, SECOND outperforms advanced models on the object hallucination task.\nRegarding the benchmarks you mentioned, we would like to clarify that MMLU is primarily designed to evaluate LLM capabilities. For MegaBench, the length of the prompts is extremely large, and in some cases exceeds the max_token_length of models such as LLaVA-NeXT Vicuna. Therefore, we conducted additional experiments on MMMU-Pro instead. We kindly refer you to Reviewer KN5t W1 for further details.\n\n**W4: How is the visual attention used to filter patches calculated in the method, and to whom is the attention calculated**\n\nThank you. We revised the manuscript for clarity. For full details, please see our response to Reviewer Lfoe W3.\n\n**W5: Are patches from previous stages reused?**\n\nYes, visual patches are accumulated across stages. Each stage uses both newly selected and retained patches to refine grounding. While this was indicated in Fig. 1, Fig. 3 and Sec. 4.1, we have now clarified it further.\n\n**W6: seems unable to correct the already erroneous visual attention and introduces cumulative errors.**\n\nThank you for this insightful observation. We clarify that patch selection in SECOND is not restricted to a fixed or narrowing set of regions. As described in Sec. 4.1, the selection ratio ($P_{\\text{select}}$) is dynamically adjusted across stages based on changes in attention entropy. When entropy increases—signaling higher uncertainty—the selection ratio can also increase, allowing inclusion of previously unselected but relevant patches.\nOur Contrastive Decoding further ensures that newly added patches, even if missed earlier, can still influence the final output—helping correct early-stage attention errors. As shown in Fig. 4, some patches initially receive high attention but decrease as new patches are added. Also in the second row of Fig. 4, we observe the model gradually shifting its focus toward the top-right region, which initially had low attention.\nTo address cases of completely misaligned attention, we compare SECOND with a variant using all patches (no selection) in Tab. 4. SECOND outperforms this baseline, showing its robustness against cumulative attention errors.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "I think this work is tradition topic with a somewhat novel method.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. This paper provides a certain degree of analysis on the hallucination of MLLM, which may inspire some new research ideas.\n2. The method proposed in this paper is training-free, capable of conducting experiments on pre-trained models, and has low resource consumption.\nWeaknesses：\n1. It is very unreasonable in the theoretical analysis section of the paper: firstly, the formula definition of Hallucination probability proposed in Section 3 seems to lack support from relevant theoretical papers (not mentioned in the paper); Next, only use Fig 2 (a) It cannot be proven that 'Non aggregated responses preferentially exhibit lower hallucination probabilities'. Fig. 2 (a) shows two edge distributions, which can only prove that if it is known to be a Non exhaustive response, it has a higher probability of being a lower hallucination probability. Because when it is hallucinated responses, the values of these hallucination probabilities are nearly uniformly distributed.\n2. Due to the hallucination of MLLM not only existing in object-centric questions or tasks, but also in more general tasks, the experimental part lacks validation of more powerful models and common benchmarks, otherwise it is difficult to prove the generality of its method. Even if it cannot be validated, the author needs to explain the specific reasons: a) the model part includes more advanced qwen2-vl and intervl2 b) the benchmark part includes MMUL/MMMU pro/MegaBenchmark, etc", "other_comments_or_suggestions": "1. The details of the method section are unclear：a) How is the visual attention used to filter patches calculated in the method, and to whom is the attention calculated? b) Is the visual path input in the current stage all from the previous stage or only from the current stage", "questions_for_authors": "1. The problem of cumulative error in SECOND visual attention: SECOND uses multi-stage visual attention for patch selection, where the attention calculated in the current stage is used for the next stage of patch selection.This method seems unable to correct the already erroneous visual attention and introduces cumulative errors. For example, in the first stage, there is an error in the attention, that is, when the attention is completely focused on visual areas unrelated to the prompt, then the input patch in the next stage will have errors and cannot be corrected. So, how to improve this problem?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fuvw590WGb", "reviewer": "Reviewer_i6dn", "summary": "The paper addresses the problem of visual hallucinations in LVLMs by introducing a training-free framework called SECOND, which adaptively selects visual patches on multiple scales and applies Contrastive Decoding (CD) between the intermediate stage logits and the logits from the fine-grained expert. The paper provides theoretical motivation for the method and an empirical analysis on the hallucinations using the established benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer i6dn,\n\nWe greatly appreciate your valuable feedback on our paper. We address the raised concerns and questions below.\n\n**W1: Essential References Not Discussed**\n\nThank you for the suggestion. We have added the mentioned works to Sec. 2. In particular, we included OPERA in our computational cost comparison (see KN5t W2).\n\n**W2: It is unclear how attention is translated or mapped across different resolutions.**\n\nAs described in Sec. 4.1, we extract attention maps at multiple resolutions using different patch sizes. These are resized to a common resolution and summed to form a unified attention map that captures multi-scale cues.\n\n**W3: It is not clear how transitions between the stages are performed.**\n\nWe appreciate the feedback and would like to clarify. In SECOND, patch selection is not applied at the initial stage. The model first performs inference using tokens covering the full image, generating an attention map. Based on its attention entropy, the patch selection ratio for the next stage is determined.\nIn the second stage, high-resolution patches from selected regions are added for inference. This aligns with multi-scale strategies like LLaVA-OneVision, where features at different resolutions are combined. The updated attention map reflects both the initial full-image attention and the influence of newly added high-res patches.\nStage transitions are dynamic—SECOND may add patches in regions that gain attention over time. As shown in Fig. 4 (second row), a region with low initial attention becomes more prominent in later stages, demonstrating the adaptiveness of our selection process. (See also Reviewer Ekip W6.)\n\n**W4: How were hyperparameters determined in the experiments?**\n\nThank you for raising this important question.\nRegarding $\\lambda$, while each setting has its optimal value Tab. 6, we found that $\\lambda = 1.0$ consistently performs well across most conditions. This supports our recommendation of $\\lambda = 1.0$ as a robust default, further backed by the trend in Fig. 6(a), where performance peaks around this value.\nAs for $\\alpha$, $\\beta$, and $\\gamma$, we acknowledge the difficulty of manual tuning. Inspired by recent works [1], which explore adaptive parameter selection, we applied divergence based adaptive parameter selection. This dynamic adjustment allowed SECOND to perform close to manually-tuned baselines, achieving reasonable results on the benchmark POPE.\nWe believe this reduces hyperparameter sensitivity and improves the practicality of SECOND, and we plan to further develop this adaptive approach in future work.\n\n\\\\(\\Delta logit_{s} = α * (logit_{s} - logit_{s-1}),\\\\)\n\nwhere \\\\(α = 1 - D_{bd}(S||S-1),\\\\)\n\nand $D_{bd}(S||S-1)$ denotes the bounded divergence of token distribution between stage $s$ and $s-1$. (from [1])\n\n|  | POPE |\n| --- | :---: |\n| LLaVA-NeXT Vicuna 7B | 86.5 |\n| LLaVA-NeXT Vicuna 7B + SECOND w/ $D_{bd}$ CD | 88.2 |\n| LLaVA-NeXT Vicuna 7B + SECOND w/ Hard CD | **89.2** |\n\n[1] \"Code: Contrasting self-generated description to combat hallucination in large multi-modal models.\" NeurIPS 2024\n\n**W5: While experiments do show benefits of SECOND across different setups, there are still some cases where performance gains are marginal or even negative.**\n\nThank you for pointing out this important issue. While SECOND does not yield improvements uniformly across all models and benchmarks, it is important to highlight that its primary goal is to mitigate object hallucination in VLMs. In this regard, it shows strong effectiveness—outperforming baselines in 11 out of 12 settings on the POPE benchmark (as shown in Tab. 1), which directly targets perceptual hallucination.\nIt also consistently improves performance on object-centric tasks such as VQAv2 across all tested models, highlighting its benefit for fine-grained object understanding. Minor drops observed in general-purpose benchmarks (e.g., MMStar, MMBench) may result from untrained patch selection slightly disrupting attention patterns—an area we plan to explore further.\nDespite a few exceptions, the overall trend shows that SECOND offers reliable gains on object-level tasks. We have revised the manuscript to clarify both the strengths and limitations of our approach.\n\n**W6: How does computational complexity of SECOND compare to that of VCD?**\n\nPlease refer to our response to Reviewer KN5t W2, where we provide a detailed comparison of per-token generation time across these methods.\n\n**W7: Presentation of the method could be improved with more details about the method**\n\nWith responses to W2 and W3, we have revised and extended the explanations in Sec. 3 and Sec. 4 to improve the clarity and readability of our method.\n\n**W8: nit: L not defined in def 3.1**\n\nThank you for pointing this out. In Def. 3.1, L refers to the length of the generated sequence. We have clarified this in the revised manuscript to avoid confusion.", "claims_and_evidence": "The authors propose the SECOND method, which is a training-free framework to be integrated to VLMs to dynamically select more precise visual information from multi-scale patches to reduce the object hallucinations in VLMs. The authors provide an initial analysis on hallucinations which motivates the method, and evaluate SECOND on meaningful benchmarks. However, the authors could better show that SECOND is able to correctly select either broad or fine-grained visual information depending on the task.", "methods_and_evaluation_criteria": "While the chosen benchmarks do make sense, the proposed SECOND method is compared to only one baseline method but none of the following:\n\n- Paying more attention to image: A training-free method for alleviating hallucination in lvlms, ECCV 2024\n\n- Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training, ECCV 2024.\n\n- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024", "theoretical_claims": "In my view, the current version of the paper doesn't sufficiently explain the details of the proposed method:\n\n- it is unclear to me how attention is translated or mapped across different resolutions,\n\n- it is not clear how transitions between the stages are performed. In my understanding, at the end of the first stage we end up with a subset of the selected patches. In the second stage, we start with an image of higher resolution but I do not understand how the patches from the first stage are used here? Is the image always sliced to the same number of patches at each stage?", "experimental_designs_or_analyses": "- How were  \\alpha, \\beta, \\gamma, and \\lambda determined in the experiments? Table 6 and 7 show that these parameters are not robust to changes in the data and models. If I want to use your method, how do I know how to set these hyperparameters?\n\n- While experiments do show benefits of SECOND across different setups, there are still some cases where performance gains are marginal or even negative which makes me doubtful about its usefulness in practice. For example, in Table 1, baseline outperforms both VCD and SECOND on MSCOCO in row 3. Similar results are seen for Mistral-7B in Table 2. There is no discussion in the paper about this, despite the fact that computational cost is significantly increased. Is this a consequence of poor hyperparameter choice, specifics of the model or the benchmark? Please elaborate on these results.\n\n- Related to above, how does computational complexity of SECOND compare to that of VCD?", "relation_to_broader_scientific_literature": "Reducing object hallucinations in VLMs is an important topic and the paper introduces a novel approach combining CD and multiscale patch selection. This differentiates it from other proposed works which are typically proposing to contrast the output distributions of multimodal inputs with those of text-only inputs.", "essential_references_not_discussed": "As discussed above, the following relevant works are not mentioned nor compared to:\n\n- Paying more attention to image: A training-free method for alleviating hallucination in lvlms, ECCV 2024\n\n- Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training, ECCV 2024.\n\n- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024", "other_strengths_and_weaknesses": "See other comments", "other_comments_or_suggestions": "- Presentation of the method could be improved with more details about the method.\n\n- nit: L not defined in def 3.1", "questions_for_authors": "See the comments above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After the rebuttal, all reviewers have raised their rating to positive (1 Accept and 3 Weak Accept). The previous main concerns are about the presentation clarity. After the rebuttal, the authors have addressed these concerns. Thus, we recommend **Accept** for this submission. It is necessary to incorporate all these rebuttal discussions into the camera ready version.
Decision: Accept (poster)