Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models
Authors: Eun Woo Im
Muhammad Kashif Ali
Vivek Gupta
Venue: ICLR 2026 Poster (2026)
Keywords: large vision language models
large language models
contrastive decoding
multimodal learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal capabilities, but they inherit the tendency to hallucinate from their underlying language models. While visual contrastive decoding has been proposed to mitigate this issue, existing methods often apply generic visual augmentations that disregard the specific context provided by the text query, limiting their effectiveness. This study introduces a novel training-free decoding strategy that addresses these limitations, featuring two key contributions. First, a self-augmentation prompting strategy that leverages the intrinsic knowledge of the model to dynamically align semantics between the query and the visual augmentation. Second, an adaptive thresholding algorithm that adaptively adjusts next token candidate size based on the output sparsity, utilizing full information from the logit distribution. Extensive experiments across four LVLMs and seven benchmarks demonstrate that the proposed decoding significantly enhances factual consistency compared to state-of-the-art decoding methods. This work highlights the importance of integrating query-dependent augmentation and entropy-aware decoding for improving effective generation of LVLMs. The source code will be released upon acceptance.

Published as a conference paper at ICLR 2026
Self-Aug: QUERY AND ENTROPY ADAPTIVE DECODING FOR LARGE VISION-LANGUAGE MODELS
Eun Woo Im Muhammad Kashif Ali", Vivek Gupta' Arizona State University Miiedrich-Alexande--nniveeee Brlangen-Nürbberg (eunwooim, ygupt140)&asu edu, kashif www.maalleef... de
ABSTRACT
Large Vision-Language Models (LVLMs) have demonstrated remarkable multi- modal capabilities, but they inherit the tendency to hallucinate from their underlying language models. While visual contrastive decoding has been proposed to mitigate this issue, existing methods often apply generic visual augmentations that disre- gard the specific context provided by the text query. limiting their effectiveness. This study introduces novel training-free decoding strategy that addresses these limitations, featuring two key contributions. First, a self-augmentation prompting strategy that leverages the intrinsic knowledge of the model dynamically align semantics between the query and the visual augmentation. Second, an adaptive thresholding algorithm that adaptively adjusts next token candidate size based on the output sparsity, utilizing full information from the logit distribution. Ex- tensive experiments across five LVLMs and seven benchmarks demonstrate that the proposed decoding significantly enhances factual consistency compared to state-of-the-an decoding methods. This work highlights the importance of inte- grating query query-dependent augmentation and entropy- aware decoding for improving effective generation of LVLMs.
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable success in language comprehension, generation, and reasoning (Brown et al., 2020; Google, 2023; Touvron et al., 2023; Chiang et al., 2023; OpenAI, 2023). By integrating visual encoding and projection, Large Vision-Language Models (LVL.Ms) have extended these capabilities to multimodal applications such as visual perception and planning (Li al., 2022; Yu et al., 2022; Li et al., 2023a; Maaz al., 2023; Ye et al., 2023; Zhang et al., 2023a; Zhu et al., 2023; Huang et al., 2023). Despite their impressive performance, LVLMS inherit critical limitations from their foundational language models. One of the most significant issues is hallucination, phenomenon of generating plausible but factually incorrect or nonsensical outputs. This behavior is largely a byproduct of the auto-regressive training objective of the model, a process that incentivizes a reliance on spurious correlations over a precise understanding of underlying facts by maximizing token likelihood based on surface-level statistical patterns (Bender & Koller, 2020).
Advanced decoding methods can significantly enhance the factual consistency by shaping how token sequences are selected from output distributions at each generation step (Van der Poel et al., 2022: Favero et al., 2024). A prominent decoding strategy to reduce hallucination effect is Contrastive Decoding (CD) (Li et al., 2023c), a technique that improves factuality by contrasting the outputs of an expert model with those of a weaker, amateur counterpart (Zhang et al., 2023b; Chuang et al., 2023). Motivated by this principle, Visual Contrastive Decoding (VCD) (Leng et al., 2024) was introduced to improve the general perceptual capabilities of LVLMM by contrasting standard output with an amateur logit generated from an input image degraded by random noise. Subsequent research in VCD has primarily focused on determining which visual modifications or hidden states with experimental heuristics can maximize the sample variance while maintaining the semantics (Li et al., 2023b; Huang et al., 2024). However, these methods often overlook the critical role of the input text query, which specifies which aspects of an image are relevant to the user request. For instance, asking to identify an object in the image and solving a handwritten math problem require entirely different capabilities
"Corresooniigg author. Project page: https: //tpw//ww/www.//////////////.../ www.www...ithuu........... 10/ selffu

---

Published as a conference paper at ICLR 2026
and reasoning from the LVLM. While VACoDe (Kim et al., 2024b) addressed this by estimating the divergence between logit distributions among the predefined visual augmentation set at the first generation step in brute-force manner, there two fundamental limitations. First, first-token divergence is an empirical measure that does not always assure a favorable augmentation choice for the entire generation sequence. Second, its dependence on a single token renders it suitable for short, multiple-choice style answers but fundamentally limits its effectiveness for complex tasks requiring open-ended generation and multi-step reasoning.
Moreover, a challenge in contrastive decoding arises from the subtraction of the amateur logit from the expert logit (Li al., 2023c). This operation can cause undesired effects that amplify the scores of certain tokens; if the amateur model produces a negative logit value, it will have its final score erroneously increased (L et al., 2024). To mitigate this amplification effect, existing methods (Jin et al., 2024) truncate the vocabulary set based on threshold set proportionally to the maximum value of the expert logit distribution. However, while this approach effective at penalizing false positives, its reliance on single data point (i.e., the maximum logit) hinders it from utilizing the rich information encoded in the full logit distribution, such as model confidence.
These aforementioned limitations lead us to two main research questions. (1) How can the semantic intent of a text query guide the selection of a visual augmentation to elicit a maximally informative discrepancy for contrastive decoding? (2) Is there correlation between predictive confidence of the model and the plausibility of its next-token candidates? To address these questions, this study introduces Self-Aug, a novel decoding strategy that adaptively select which visual augmentation is best suited to be contextually relevant. Unlike prior works (Kim et al., 2024b), Self-Aug utilizes the intrinsic model knowledge determine an optimal visual modification out of the box. Furthermore, we introduce an improved thresholding algorithm, Sparsity Adaptive Truncation (SAT), to overcome the limitations of existing plausibility constraints. Where prior methods often fail to utilize full information from the logit, SAT dynamically determines threshold by utilizing the entire logit distribution as a proxy for the confidence of the output. The proposed method integrates seamlessly into any LVL.M without requiring any architectural modifications or additional training. Extensive experiments and analysis verify that the proposed methods significantly enhance factual consistency and reduce hallucinations across multiple models and benchmarks. The contributions of this study are summarized as follows:
1. This work introduces Self-Aug, prompting strategy that leverages parametric knowledge of the model to select visual augmentation that is semantically relevant to the textual query, thereby extracting a more informative discrepancy.
2. The proposed SAT improves the existing adaptive plausibility constraint by leveraging the entropy of the expert logit and dynamically sets a threshold of token implausibilities.
3. Extensive experiments validate the effectiveness of the proposed method across five LVLMs and seven benchmarks. The results demonstrate that Self-Aug significantly reduces halluci- nations while amplifying the relevance and informativeness the response.
PRELIMINARIES
Auto-regressive Generation of LVLMs Suppose that fo is LVLM (Gong et al., 2023; Maaz et al., 2023; Li et al., 2025a), parameterized by 0. The model operates on a vocabulary set V, and the set of all possible token sequences can be denoted by its Kleene closure, V' U1 V. where indicates the timestamp of the LVLM output. The function fo V x RAxwx3 + V auto- regressively generates a response from given text query I E V and visual input R At each timestep I, the LVLM computes a logit distribution over the vocabulary for the next token y/t, conditioned on the inputs (Ir, u) and the sequence of previously generated tokens y<a. This yields the probability distribution over the next token:
po x., Wee)) exp (logit, (,
The next token is then selected from this distribution according to a chosen decoding method. Decoding methods are broadly categorized into two families: deterministic search, including greedy
'Throughout this paper, we use the terms "augment" and "modify" interchangeably, consistent with their usage in the related literature.
2

---

Published as a conference paper at ICLR 2026
and beam search (Graves, 2012), and stochastic sampling, such as top-k, Nucleus (Holtzman et al., 2019), Mirostat (Basu et al., 2020), and typical (Meister et al., 2023) sampling.
Hallucination Ideally, the generated response y should be factually accurate, relevant to the query æ, and faithful to the visual content c. However, current LVLMs often fail to meet these criteria, suffering from critical issue known as hallucination (Rohrbach al., 2018). This phenomenon stems from multiple reasons, including imperfect learning and decoding (Ji et al., 2023). misalignment of vision and language modalities (Tong et al., 2024), and failure of understanding the context (Daunhawer et al., 2021). To address this issue, recent studies have suggested scaling the imput image resolution (Liu ct al., 2024b; Chen et al., 2024b), combining another inductive bias of visual encoders (Li ct al., 2025b), post-hoc rectifying (Zhou et al., 2023), self-correction after generation (Yin et al., 2024), and advanced decoding methods (Shi et al., 2024; Favero et al., 2024). Among those approaches, decoding- based methods are particularly promising since they enable real-time control, do not require additional training, and are compatible with other hallucination mitigation strategies.
Contrastive Decoding CD (Li et al., 2023c) tackled hallucination problems the NLP domain by contrasting the predictions of two different language models with different capacities. VCD (Leng et al., 2024) extended the idea of CD with vision modality and introduced the contrastive counterpart vf by degrading visual content with random noise to v. It sequentially treats the logit from v' as an output of the amateur model, sampling the next token from:
PCD u, v', = softmax ((1 + a) logit((ylv x) a logity: (ylv', (lo",)))) x)),
where a denotes an amplification parameter. Recent studies have focused on curating a better selection of the degradation to achieve maximal differentiation while preserving semantic integrity. For instance, cropping the patch which is likely to cause hallucinations (Chen et al., 2024a), caption substitute (Kim et al., 2024a), and visualization of the textual output (Park et al., 2025). While most VCD methods rely on a shared underlying principle of query-agnostic input modifications, VACoDe (Kim et al., 2024b) has introduced dynamic visual augmentation strategy. This approach attempts to be query-aware by using L2 distance between the expert and amateur logit distribution at the first token generation
function to select augmentation.
However, this reliance on first-generated token has fundamental limitations. The overall semantics of task are not universally guaranteed to be reflected by first-token divergence, which is empirical proxy. One example of failure when two logits have the same argmax but are distinct in terms of overall entropy. In this case, the query could not be invalidated because, despite the distortion, the model could still answer correctly, although the divergence remained large. This may be effective for short-answer and multiple-choice questions, but it can be ineffective for other scenarios including open-ended questions and multi-step reasoning.
3 Self-Aug: SELF-AUGMENTED VISUAL CONTRASTIVE DECODING
To address the preceding limitations, we introduce Self-Aug, a decoding method that identifies a query-specific visual augmentation to apply for visual contrastive decoding by utilizing the rich knowledge base of LVL.M (Li et al., 2024). Unlike prior methods that rely on simple heuristics, Self-Augleverages the world knowledge and common sense embedded in the LVLM to achieve a semantic alignment between the query and the selected augmentation. This approach enables the model to reason the underlying intent of a query and make a choice that elicits more targeted and informative discrepancy. Alg. and Fig. 1 outline the proposed method.
3.1 SELF-AUGMENTATION SELECTION
SAS Prompting Seff-Augmentatioon Selection (SAS) is a concept of meta-level classification task that aims to employ parametric knowledge of the LVLM to dynamically select the best task-optimal visual augmentation on the fly that amplifies the output divergence. This is achieved through structured SAS Prompt P, which comprises three key components. First, the prompt contains explicit definitions of each visual augmentation and corresponding effects, providing the model with the necessary operational knowledge. Second, to minimize the risk of post hoc rationalization, the prompt is structured to elicit reasoning before the final selection is made (Zelikman et al., 2024). Finally,
3

---

Published as a conference paper at ICLR 2026
Belf-Augmentation Selection Prompting
Sparsity Adaptive Truncation
Cholce:
angglive segmantation
Seleet modt semanically
Color inversion.
ereectee invelidatee
SMP
sucstion's promice
prweettic conrideet
coat?
Query:
Contrasted
image
Gyan pirk
Pod
sample(
What color is the coat?
Original
Red
Figure 1: Overview of the proposed augmentation choice process and sparsity adaptive truncation.
inspired by few-shot learning techniques (Brown et al., 2020; Patel et al., 2024), in-context learning (ICL) examples are included the prompt P to further condition the contextual knowledge (Alayrac et al,, 2022). The textual output is then processed by parsing function g(.) V - V x V", which separates the reasoning trace and final augmentation choice c. The contrasted image is obtained by feeding the and the final choice
predefined visual augmentation function A.
(r, c) = x)),, v' = A(c,v))
(3)
Subsequently, contrasted logit distribution is calculated from expert logit I = logity (yslty((// ( 2, yee) and amateur logit l' logit, ( (vrlA(c, 2,, Ver). The augmentation set is defined with random crop, random mask, noise addition, color inversion, horizontal flip, and vertical flip. Note that the generation configuration is set to greedy decoding for SAS Prompt P to ensure computational efficiency, determinism, and reproducibility. While further optimized prompting techniques (Manakul al., 2023) and multiple combinations of different augmentations can be deployed, we limit the scope to two prompting features and the aforementioned augmentation set in this work. Full prompt referred to the Appendix B.
3.2 RETHINKING ADAPTIVE PLAUSIBILITY CONSTRAINT
CD-based methods encourage the generation of implausible tokens since the output distribution from contrasted visual input s' still involves the underlying semantics of u (Li et al., 2023c). This can cause the two distributions to not cooperate properly, resulting in the reward of undesirable tokens. Adaptive Plausibility Constraint (APC) (Li et al., 2023c; Leng et al., 2024) addresses this challenge with controllable hyperparameter maximum probability of the new token, formulated as:
[0, 1], setting threshold proportional the logarithm of the
VAPC = {ve V po(sllv, (,, Met) 2 3- max (wrl,, x. }c))]- WEV
However, since this thresholding mechanism is based solely on the maximum logit value and the meaning of a logit value is relative to the other logits in the distribution, is a confidence-agnostin filter. Although this approach penalizes false positives by truncating the sample space, it becomes unreliable in low-confidence states, when the risk of discarding the correct token from the candidate set is high (Guo al., 2017; Wenkel et al., 2021). This deficiency arises because APC disregards the rich signal encoded in the full output distribution. The entropy of logit distribution provides more robust and holistic measure of model uncertainty which can be leveraged for more effective filtering of the candidate set.
Model uncertainty, characterized by the value of output entropy, a recognized correlate of model errors (Manakul et al., 2023). When the logit distribution is highly entropic, more lenient threshold required to create sufficiently inclusive candidate set and avoid erroneously discarding the context-relevant tokens. Conversely, in low-entropy scenarios where the model is confident (Tornetta, 2021) with a sparse output distribution, a more restrictive threshold is required retain pivotal tokens with high probability and to refine the candidate set by taking over the probability mass from filtered tokens (Li et al., 2023c). This inverse proportionality heuristic improves generation fidelity by minimizing the risk of sampling erroneous, probabilityy tokens the tail of the distribution.
To enable the confidence aware thresholding, the proposed method extends APC to SAT, a method that dynamically adjusts the plausibility constraint based on the sparsity of the output distribution. The method leverages the principle that a sparsity is inversely related to its uncertainty, quantified by Shannon Entropy H R (0, log, d] (Shannon, 1948), which maps probability distribution
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 Self-Aug: Self-Augment Visual Contrastive Decoding
Require: input image v, text query 2, LVLM fe, augmentation function A. SAS Prompt P, vocabu- lary set V. hyperparameter a.
1:
(P)
Identify augmentation from given Initiate t
2:
+ 0
3: while t T do
4:
< logita (Ue u. v. yct)
>. Set expert logit /
5:
I' + (Vel
lCp + A(c, U), 2, Vet))
Set amateur logit I'
6:
a.l
> Set contrasted logit D Set SAT parameter 3, from Eq.
7:
BE Hdecay (softmax(I)
8:
VSAT + Eye E V po [, v. V<r) 2 B5AT max (w v, æ, yct)} Set threshold
9:
[CDli] + -00
for all & VSAT
> Apply vocabulary truncation Token sample
Ye softmax((cD)
11:
++1+1
end while
13:
return {wo. ................................ VT -1
over its dimension d to its uncertainty, calculated as H()) = >m logg Pi- To implement an inversely proportional relationship where higher entropy yields smaller threshold, a decayed entropy function Haecay RM + (0, 0.5) is formulated compute the threshold value:
Haeway (p) = (-7
Σ
Ps logz Pt
(5)
where and 7 denote sigmoid function and scaling parameter, respectively. The choice of a sigmoidal decay is deliberate, as other decaying functions, such as exponential or polyno- mial (Provencher, 1976; Borichev & Tomilov, 2010), could be potentially considered, but they lack the versatility of a sigmoid. The curve of the sigmoid function is naturally bounded to (0, 1), and its lower plateau creates stable, consistent threshold for low confidence distributions, and precise controllability over the single steepness parameter y of the decay for mid-range entropy. Furthermore, by ensuring the threshold remains strictly less than 1, SAT prevents the candidate set from collapsing to single token, guaranteeing that the decoding process remains distinct from greedy decoding. The proposed SAT method introduces a dynamic threshold 35ΛT which calculated by incorporating the entropy of the logit distribution: 35AA Haccay (softmax (logity (yelu [, x, (ct))). The next-token candidate set, VSAT. is then constructed by filtering the vocabulary set with this adaptive threshold:
VSATT {ya V (ya VV, x. Vet) 2 ggaT max po (wvv, x; Ve+)). J. To exclude the implausible tokens, - oo is assigned to logit elements which are not involved in VSAT- Finally, the contrasted probability distribution is obtained by combining Equations 2 to 5:
lcD (yocll (u x, vC() = [1+ a.
if yt E VSAT otherwise.
(6)
-00
The token at timestamp t is sampled from PCD(V v,, v. Vet) softmax(ccv).
EXPERIMENTS
4.1 EXPERIMENTAL. SETTINGS
Benchmark and Model Selection Following standard practices in the literature (Leng et al., 2024; Kim et al., 2024b), three foundation model families are selected evaluate the effectiveness of Self-Aug: LLaVA-1.5-7BBI/ 7B (Dai et al., 2023) with vicuna-v1.1 (Chiang et al., 2023), and Qwen3-VL-SB (Bai et al., 2025). The evaluations are divided into two categories: discriminative and generative benchmarks. Discriminative benchmarks assess the factuality of visual recognition in the form of binary or multiple choice questions, while generative benchmarks evaluate broader capabilities by requiring open-ended responses and judge with proprietary models (Zheng et al., 2023; Gu et al., 2024; Ali et al., 2025). The selected discriminative benchmarks include POPE (Li et al., 2023d) constructed on MSCOCO (Lin
(Liu et al., 2024a), Qwen- Owen-VL-7B VL-7B (Bai et al., 2023). InstructB
5

---

Avg. A Published as a conference paper at ICLR 2026
Table Discriminative benchmark results on MME (Fu et al., 2024), MMVP (Tong al., 2024), and POPE (Li et al., 2023d) constructed on COCO (Lin et al., 2014), and A-OKVQA (Schwenk et al., 2022). Avg. A denotes averaged gain against Multinomial sampling across benchmarks.
POPE-MSCOCO
POPE-AOKVQA
Model
Method
Acc.1
F11
Acc.1
FI"
MME-P7
MMVP
Multinomial
82.07 81
80.48
1
79.86g3.26
1278.42
32,40
LLaVA-1.5-BB
VCD
83.66
82.55
80.51
81.11
1323.67 20.84
34.00
+10.86%
VACoDe
84.29
83,59
80.8644.97
81 81.87
1372.50
36.67
+9,52%
Self-Aug
82.93
83.57
$2.80
83.20
1431.30 15.88
36.00
+14.32%
Multinomial
83.86
81.02
20.97
80.79
1351.69
20:30
31,60 2.
LLaVA-LS--33B
VCD
83.86:1.72
82,68
1155
81.93+3.05
82.164.294
1372.77
31.60+4.81
+6.33%
VACoDe
84,86
84.17 11.68
82.34:33.00
83 0..0 1:3.02
112.79 1434.09
32. g3:22
+8.03%
Self-Aug
85.37
83.96
84.25
84.13
1462.18
18.22
34.80 :1:19
+11.99%
Multinomial
75.72.9.
72.07 1:0
76.64
74.68
1311.79 22344
17.33
VCD
27.98
75.42
78.85
77.73
1415.12421
21.33
+5.05%
Qwen-VL
VACoDe
78.35
76.08
28.98
78.02
1412.43
22.13
+7.49%
Self-Aug
77.58
78.23+2.00
76.78
1442,36
26.67
+6,69%
Multinomial
68.70
69, 69.34
65.5 65.52.33.00
68. 36.6
973 66
19.20
InstructHL.P JI
VCD
12.77 21.9
69 69.26
72.23.g2.00
#40.30 1079.39
18,93
+12.33%
VACoDe
73.29
74.26
70.01
73.4042.91
1090,88
21.87
+10.98%
Self Self-Aug
82.86 14
82.34
22.09
75.37
1198.53
16.13
+18.78%
Multinomial
87.93 s8.59
89, 99.022a.11
89,57 19.5
1725. 16
55.47=2272
VCD
88.76
88,22 88,22
89.00ga.14
89.59 89.59
1704. 770411
58.27
+1.00%
Qwen3- VBB.88
VACoDe
88.97 +0.05
88.4740.10
89.07
89.63
24116
59.87
+2.07%
Self-Aug
88.79
88.20g0.14
88 88.68gu.0
89 20.25
1726.77 23.66
60.50 20.64
+2.25%
Opet
Exper
Imagi
pending
A5g
Amarer
No
Amare
The
.mrge
31g
Fasttakion
Diper
26.14
Sapart
20.09
22.30
22.30
13500
18:33
2134
29.09
23.00
130
1030
Ameiiu
24-72
20.09
23.47
12.53
2535
15337
2173
23.41
1532
1750
58.00
33.41
20.55
25.09
22.50
33.9
2477
17.3
25.5
22.0
20.7
Query:
24:12
58.30
$3.99
24.77
25.5
2833
20.7
the blowing the flag
Figure 2: Qualitative examples Self-Aug on MM- Vet et 2023) and L.L.VA-Bend (Liu et al., 2023), and corresponding logit distributions and SAT thresholds by timestamp.
et al., 2014), and A-OKVQA (Schwenk et al., 2022) dataset, MME-Perception (MME-P) (Fu et al., 2024), and MMVP (Tong et al., 2024). MMHal-Bench (Sun et al., 2023), LLa (In-the-Wild) (Liu et al., 2023), MM-Vet (Yu et al., 2023) are selected for generative benchmarks. The ablation studies were focused on the LLaVA-1.5-7B the MME-P benchmark. This selection represents methodological choice, LLaVA-1.5 being one of the most widespread adoption within the open-source community, while MME-Perception offers the largest testbed among ones with diverse categories.
Implementation Details Unless explicitly stated otherwise, the CD hyperparameters set = 1, 0.1 for APC, and the SAT hyperparameter was set to ? = -0.5. Including automated judging of generative benchmarks, all APT calls to proprietary models gpt-4o-mini with temperature O for deterministic results and reproducibility. All main experiments were conducted over five runs, and ablation studies over three runs with different random seeds, with results reported as the average and standard deviation to account for the inherent randomness from the augmentation process and multinomial sampling.
made using OpenAl
4.2 EXPERIMENTAL RESULTS
Main Results Tab. and 2 summarize the averaged performance and standard deviations for all evaluated settings. The final column in each table, denoted as Avg. A, reports the average performance gain the multinomial sampling baseline for each method and combination. For this calculation, the accuracy score is used for POPE and the average score is used for MMHal- AMHal-Bench. For each configuration, the beff-peng method is highlighted in bold, and ties resolved in favor of

---

Published as a conference paper at ICLR 2026
Table 2: Generative benchmark results on LLa LLaVA-Bench (In-the-Wild) (Liu et al., 2023), MM-Vet (Yu et al., 2023), and MMHal-Bench Sun et al. (2023).
MMHal Bench
Model
Method
Avg. Score"
Hal. Rate,
MM-Vet"
LLaVA-Benchi
Avg.
Multinomial
2.27
0.6510.02
27.74
58.4812.17
VCD
2.32-0.00
0.65
31.14
69.0842.07
+2.82%
LLaVA-L.5-7B
VACoDe
2.32-00.09
0.64
29.88
69.1222.48
+6,14%
Self-Aug
2.55
0.59 1.59
31.14 10.14
69.2221.80
+6,97%
Multinomial
2.35-0.18
0.65
1..20
69,4822.78
VCD
2.37
0.64
35.00#1.61
73.62=1.40
+1.11%
LLaVA-1.5-13B
VACoDe
2.52-
0.61 40.03
34.18
74.56=1.62
+4.78%
Self-Aug
2.53_0.00
0.60
36.62 11333
76.24
+6.04%
Multinomial
2.21
0.50
1.770
35.98=1.50
VCD
2.17=0.08
0.51
34,04=1.21
38.78
+9.21%
Qwen-VL.
VACoDe
2.21_0.13 10.13
0,50 10.00
35.42
39.18
+10.47%
Seif-Aug
2.15
4.502
35,98
39.84=0.73
+17.09%
Multinomial
1.89-0.1
0.69s
23.06+0.69
53.24
VCD
1.99
0.704
28. 1.335
58,30 11.38
+4.99%
InstructBI.IP
VACoDe
2.03-0.07
0.68
27. 27.40+0.51
56.82
49.17%
Self-Aug
2.16-0.13
0.64
31.14
56.98
+17.08%
Multinomial
4.56_a.11
0.31
62.82
118. 118.50+2.36
VCD
4.52_0.16
0.32
63,48
19.76111.1
+0.44%
Qwen3- VL-BB
VACoDe
4.63-0.14
0.29
65,08 (1.8881110
121.06
+2.45%
Self-Aug
4.67=0.04
0.29 #0.01
64,50+11,3
121.88 +1.39
+2.62%
the method exhibiting lower variance across runs. Self-Aug achieves remarkable performance gains across both benchmark categories, up to 18.78% 8.788% relative to the multinomial sampling.
To further probe the effectiveness of Seff-Aug, token-level analysis was conducted verify how the proposed method mitigates hallucinations by examining the output logits of LLaVA-1.5-7B8 Fig. illustrates two examples of logit values of L.L. with Self-Aug on MM- Vet (Yu et al., 2023) and LLaVA-Bench (Liu et al., 2023). Amateur and Expert logit indicate the selected token with and without augmentation, and the final token, highlighted with gray, is the token that corresponds the argmax of the contrasted logit. Note that the applied augmentations are stylized for visual clarity.
These examples provide three important observations. (1) failure correction where the contrastive process between two logits successfully elevates the score for the correct Yes token, making the final answer. (2) The example on the right evidences hallucination penalty, where random noise triggered hallucination of generating _b bllue token from the amateur logit. threshold and be removed from the candidate set. (3) Adaptive nature of the SAT threshold (SAT is observed, with a higher threshold applied to common tokens (e.g., articles, prepositions) and a lower threshold applied to informative, findings highlight a clear validation of both core components Self-Aug, confirming that not only contextually relevant augmentation selection with model knowledge can effectively amplify the output divergence by invalidating the premise of the question, but also the efficacy of confidence-aware SAT.
The example to the left shows a case of
penalized through subtraction, causing its final score to fall below the SAT
confidenneeeece tokens (e.g., painting, red-boxed token). These
Computational Overhead The computa- tional cost of Self-Aug was evaluated by com- paring throughput (token/s) and latency (ms/to- ken) against other CD-based methods. The anal- ysis used the LLaVA-Bende with the LLa family on an NVIDIA A100 GPU. Detailed re- sults are presented Tab. 3. The superscript + on Self-Aug denotes the full prompt config- uration that includes reasoning and ICL, while denotes lightweight configuration without both components. The results show that the primary computational bottleneck for both adap- tive methods is the augmentation choice process. VACoDe is brute-orce that requires separate
Table 3: Decoding throughput (token/s) and la- tency (ms/token). Scale and # tokindicate model parameters and the number of generated tokens for multimodal query, respectively.
Decoding
Scale
# tok
tok/s?
ms/tok.
Score
7B
9914
18.50
54.06
69.08_2.07
VCD
13B
8785
14.01
71.38
73.62=:140
7B
8418
16.97
58.93
69.12222.4
VACoDe
13B
8568
13.03
76.76
74.56_1.62
7B
8346
17.39
57.50
69.20..2.12
Self-Aug
138
8793
11.37
87.92
73.82=11.6
7B
10163
15.08
66.32
69.22
Self-Au
13B
8805
11.33
88.26
76.24
7

---

Published as a conference paper at ICLR 2026
Model Choice Distribution Comparison
Table 4: Comparison between
100
static augmentations and adaptive selection strategies on MME-P using LLaVA-1.5-7B.
Choice Percentigh (N)
60
Augmentation
MME-P"
58%
40
Static
Noise
1351.7647.59
41%
Horizontal flip
1302.55 1.7.62
Vertical flip
1354,42416.18
LLaVA-Bench
VNE
MM Vet
MMilal Bench
MMVP
POPE
Random crop
1315.56+41.73
1771 LL-TA-7
Model
LLAVA-13
Orccko
Random mask
1302.50
Approration Chorce
Adaptive
VACoDe Self-Aug
1372.50
Figure 3: Distribution of self-augmentation choice across model size and benchmarks. Oracle indicates gpt-40-mini decisions.
1431.30
Oracle
1435.07 2:2.30
forward pass for each predefined augmentation, which includes the full set of visual tokens, result- ing in an overhead that scales linearly with the size of the augmentation set. On the other hand, Self-Aug demonstrates architectural advantage by requesting a single text-only generation pass, bypassing the main computational expenses of visual tokens, which constitute the majority of the input. This architectural feature enables a flexible trade-off between performance and latency, in that the cost-optimized prompting exhibits substantially higher efficiency with minimal impact performance. This confirms that Self-Aug provides a more scalable and controllable framework for query-aware contrastive decoding.
4.3 ABLATION STUDY AND ANALYSES
Augmentation Selection A detailed investigation of the augmentation choice made by the LLaVA- 1.5 family is presented in Fig. with different patterns by model capacities. For comparison, the choices from gpt-4o-mini are also included to provide a practical upper bound and will be denoted as the "Oracle" for notation convenience throughout the remainder of this paper. The results reveal that the distribution of selections varies significantly across different benchmarks. A notable contrast is found between the sparsest POPE and the most uniform MMVP. For POPE, random mask accounts for 87.6% of all selections. This strong preference arises because the queries related to object recognition in POPE are unified as Ts there a {object; in the image?, ? which are directly addressed by the definition of random mask as an occluding operation within the SAS Prompt. In contrast, the uniform distribution of MMVP reflects the diverse nature of the benchmark itself, which queries nine different visual pattern categories and thus applies wider range of augmentations. On the other hand, the infrequent selection of horizontal flip across all benchmarks direct result of the evaluated queries rarely testing for horizontal spatial relationships. These findings suggest broader principle that the set of predefined augmentations must be sufficiently diverse to match the complexity of the visual patterns in given task, while noting that the specific distribution of choices is also dependent the SAS Prompt design.
Model Capacity The impact of model scale on the quality of augmentation selection evaluated by comparing the VA-15- 7B and 13B models against the Oracle baseline using two primary metrics. First, selection accuracy was measured by calculating the agreement with the Oracle choice, where the 7B model achieved 65.15% and the 13B model achieved 66. 19%. Second, the quality of the reasoning trace for each choice was assessed by gpt-4o-mini on a scale of 0 to 10, with the 13B model producing higher quality justifications with an average score of 9.04 compared to 8.28 for the 7B model across benchmarks. The results from both metrics confirm that larger model capacity leads to improved augmentation selection and reasoning quality. Full prompt for reasoning assessment and detailed breakdown of these agreements are provided in the Appendix B.2 and E, respectively.
Comparison with Single Augmentation A comparison between static and adaptive visual aug- mentation strategies is presented in Tab. 4. Static strategies apply single, fixed augmentation across all inputs, while adaptive strategies utilize aware augmentations. There is a clear performance gap between the two approaches, underscoring the importance of context-optimal augmentation. The
8

---

Published as a conference paper at ICLR 2026
Performance Regarding Decaying Function
Table 5: Plausibility constraint thresholding with APC and SAT.
1400
TIME-Percepton
1350
Decoding
Thresholding
MME-P*
1300
VCD
APC (3 0.1)
1323.67 220.84
inversy-entropy
SAT
1395.17
1250
Ditroyy
APC
VACoDe
APC (3 = 0.1)
1372.50
1200
SAT
1414.21 226.85
1.0
Absolute Value of Gamma (lvl)
APC (3 = 0.1)
1345.46+
Self-Aug
Figure 4: Comparison of the normalized entropy and proposed inverse-entropy function by 7
SAT
1431.30
significant gap between the proposed method and the others underscores the importance of query- augmentation semantic alignment and architectural flexibility, opening the possibility of leveraging diverse knowledge sources from internal knowledge to external reasoning modules.
SAT Threshold The core inverse-entropy heuristic of SAT was validated through a compari- of the proposed Haucay against the APC baseline and normalized scaled entropy Hn
(- --E (p), (p)/ log2 [/)) The Hns function is direct- proeootpropottionall entropy function, t.e. implements the opposing rule to Hocsay. mapping high input entropy a more confined threshold. As visualized in Fig. 4, the results confirm a performance hierarchy: Hdecay consistently outperforms APC and Hns with more stable outputs. A further observation arises from the performance trend within each entropy based function with respect to the scaling parameter 7. A lower absolute value corresponds to more restrictive threshold for Hasay but more generous one for Hns- These findings not only imply mature thresholding is required to properly penalize false positives, but also provide strong empirical support for the inverse-entropy principle in the design of SAT.
Furthermore, the generalizability of SAT was evaluated through a direct comparison with the APC baseline. Both thresholding algorithms were applied to VCD, VACoDe, and Self-Aug, and the findings presented in Tab. 5. The results show that SAT consistently outperforms APC across all decoding configurations, achieving an average performance gain of 4.94%. This performance gain is attributed to the foundational difference in the usage of model confidence. The consistency of this improvement suggests that SAT is broadly applicable to other CD-based methods.
SAS Prompting The individual contributions of the operational knowledge, reasoning and ICL components within SAS Prompting were evaluated by selectively ab- lating each component from the full prompt. As shown in Tab. 6, the provision of operational knowledge is the most critical component for performance, whereas reasoning and ICL have a marginal impact on accuracy. However, note that the reasoning instruction is the most primary factor affecting computational latency, it requires the model to generate full text sequence for the justifica- tion. Removing this instruction reduces the generation requirement to fewer than ten tokens for the final choice.
Table 6: SAS Prompting with and with- our reasoning steps and ICI. OK denotes the operational knowledge.
OK
Reasoning
ICL
MME-P"
Baseline
1278 377..42+30.85
X
x
1312.33
1419.08 +
1428.0247.00
1428.63 231.85
1431.30
5 DISCUSSION
Failure Cases Self-Aug inherits the funda- mental assumptions CD. Suppose that we have a false positive index in both expert logit I and amateur logit It is assumed that l' has higher likelihood due to the disruption from visual augmentations.
Table Multinomial sampling results on Qwen3- VL-32B. High scores on induce failures.
Benchmark
lcp
i-th index (1ji] < ('(i))
MMVP
67.56
1. 67.78+0.38
MME-P
1798.1548.08
1786.4342.8 81
1789.9420.23
Then, the i-th value of the contrasted logit ICD will decrease since (1 aa)
(0) [[]]
[[] 1[(] always

---

Published as a conference paper at ICLR 2026
holds by the assumption [[t] < [[[ Conversely, for a true positive token index j, the method assumes the a lower likelihood on amateur logit (([)) > ([)). If remains as accurate as I, the contrastive formulation yields minimal performance gains because the two distributions
nearly identical. Therefore, the failure case in all CD-baaeed methods would be the case when the two assumptions invalidated (i.e, [[1] [[] and [[j] P.)) Tab. 7 demonstrates this phenomenon using the averaged scores of Qwen3-VL-32B across three separate runs. Determining and addressing such failure beyond the scope of this study, and deferred to future works.
I Limitations and Future Work The proposed method presents several branches for future work by addressing current limitations. First, the effectiveness of SAS Prompting depends on the reasoning and instruction-followinng ability of the base model. Less capable models might produce malformed outputs or poor augmentation choices. This dependency could be addressed in future work by developing more robust prompting methods (e.g.. Chain-of-Thhuugh (Wei et al., 2022)) utilizing smaller, specialized model for the selection task. Second, the current method limited to a predefined set of visual augmentations. While this set covers common scenarios, it may not contain the best augmentation for highly specialized visual reasoning tasks. A promising solution involves developing methods that can dynamically select from more diverse and larger library of transformations using external modules (e.g., object detector), enhancing the versatility. Finally, expanding the principle to the video domain by designing temporal-aware dynamic recalibration methods obtaining contrastive pairs would enable frame-consistent decoding. The direction aids understanding complex temporal contexts and facilitating downstream applications as highlighted in previous studies (Im et al., 2023; Ali et al., 2024).
6 CONCLUSION
This study introduces Self-Aug, novel decoding strategy designed to mitigate hallucinations in LVLMs. The proposed method aligns the semantics between query and visual augmentation by leveraging the flexible intrinsic reasoning of the model without relying on predefined heuristics. In addition, the proposed sparsity adaptive truncation introduces confidence-aware thresholding that dynamically adjusts candidate sets based on logit entropy, effectively penalizing false positives. Extensive experiments conducted across five LVLM and seven benchmarks demonstrated that Self- Aug consistently improves factual consistency practical computational efficiency. Beyond immediate performance gains, this study underlines the importance of the semantic coupling of query-augmentation with confidence-sensitiv decoding as principled approach for developing more robust multimodal generation.
existing decoding strategies while maintaining
ACKNOWLEDGMENTS
We gratefully acknowledge the Complex Data Reasoning and Analysis Lab at Arizona State University for their resources and computational support.
REFERENCES
Jean-Bapiie Captitatistte Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 35: 23716-23736, 2022.
Muhammad Kashif Ali, Eun Woo Im, Dongjin Kim, and Tae Hyun Kim. Harnessing meta-learning for improving full-frame video stabilization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12605-12614, 2024.
Muhammad Kashif Ali, Eun Woo Im, Dongjin Kim, Tae Hyun Kim, Vivek Gupta, Haonan Luo, and Tianrui Li. Harnessing meta-learning for controllable full-frame video stabilization. arXiv preprint arXiv:2508. 18859, 2025.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- lasion-language model with versatile abilities. arXiv preprint ar0iv:2308. /2966, 1(2):3, 2023.
10

---

Published as a conference paper at ICLR 2026
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 2025.
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity. arXiv preprint arXiv:2007.14966 2020.
Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning. form, and understanding in the age of data. In Anmual Meeting of the Association for Computational Linguistics (ACL), pp-
5185-5198, 2020.
Alexander Borichev and Yuri Tomilov. Optimal polynomial decay of functions and operator semi- groups. Mathematische Annalen, 347(2)4455 2010.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 13:1877-1901, 2020.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.0042, 2024a.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 62(12):220101, 2024b.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2(3):6 2023.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36: 49250-49267, 2023.
Iaant Daunhawer, Thomas M Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal vaes. arXiv preprint arXiv:2110.04122 2021.
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14303-14312 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL htttps://arxiiit 2306. 13394.
abs/
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04799 2023.
Google. Bard. https://barte google.goooooooooooooooooo boogle... com/, 2023.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv: 1211,711, 2012.
11

---

Published as a conference paper at ICLR 2026
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-juee judgee. arXiv preprint arXiv:2411.15594 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), pp. 1321-1330. PMLR, 2017.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. 2019.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocaiio In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13418-13427, 2024.
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems (NeurIPS). 36:72096-72109, 2023.
Eun Woo Im, Junsung Shin, Sungyong Baik, and Tae Hyun Kim. Deep variational bayesian modeling of haze degradation process. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 895-904, 2023.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, and Jun Zhao. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval- augmented language models. arXiv preprint arXiv:2402. 14409, 2024.
Junho Kim, Hyunjun Kim, Kim Yeonju, and Yong Man Ro. Code: Contrasting self seff-generated
description to combat hallucination in large multi- models. Advances Neural Information Processing Systems (NeurIPS), 11111335711133590 2024a.
Sihyeon Kim, Boryeong Cho, Sangmin Bae, Sumyeong Ahn, and Se- Young Yun. Vacode: Visual augmented contrastive decoding. arXiv preprint arXiv:2408.0553: 2024b.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decod- ing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition 2024.
(CVPR), pp.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025a.
Chaoyu Li, Eun Woo Im, and Pooyan Fazli. Vidhalluc: Evaluating temporal hallucinations in multimodal large language models for video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025b.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified lanquae understanding and generation. In International Conference on Machine Learning (ICML), pp. 12888-122 12900. PMLR, 2022.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language -angae pre- pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), pp. 19730-19742. PMLR, 2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-tee intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems (NeurIPS), 36:41451--1155 2023b.
12

---

Published as a conference paper at ICLR 2026
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Perey Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. 2023c.
Yaoyiran Li, Anna Korhonen, and Ivan Vulić. Self- Self-augmented in-context learning for unsupervised word translation. arXiv preprint arXiv:2402. /0024, 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprini arXiv:2305. 10355, 2023d.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36:34892-34916 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10296-263066 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024b.
Xinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen, and Jingkuan Song. Alleviating hallucinations in large vision-language models through hallucinatiooninduuee optimization. Advances Neural Information Processing Systems (NeurIPS), 37:122811-112111 1-122832, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08894 2023.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally typical sampling. Transac- tions of the Association for Computational Linguistics, 11:102-121, 2023.
OpenAl. ChatGPT. https: /ttp///aaa..cm/llggga/./oocaeale hhallo/ggchattt//.../ 2023.
2023.
Yeji Park, Deokyeong Lee, Junsuk Choe, and Buru Chang. Convis: Contrastive decoding with ballucination visualization for mitigating hallucinations in multimodal large language models. In Association for the Advancement of Artificial Intelligence (AAAI), volume 39, pp. 6434 6442, 2025.
Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng. Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic llaion-language negatives. Advances Neural Information Processing Systems (NeurIPS), 37:32731-32761 2024.
SW Provencher. A fourier method for the analysis of exponential decay curves. Biophysicul journal, 16(1):27-41, 1976.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint 1rXv::809.02256, 2018.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino. and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 146-162. Springer, 2022.
Claude E Shamnon. A mathematical theory communication. The Bell system technical journal, 27 (3):379-423, 1948.
13

---

Published as a conference paper at ICLR 2026
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 2: Short Papers), pp- 783-791, 2024.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 2.4i::2300.155223, 2023.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9568 9578,5778888 2024.
Gabriele N Tornetta Entropy methods for the confidence assessment of probabilistic classification models. arXiv preprint arXiv:2103./5157, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Liam Van der Poel, Ryan Cotterell, and Clara Meister. Mutual information alleviates hallucinations in abstractive summarization. arXiv preprint arXiv:2220.132222 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS), 35:24824-24837, 2022.
Simon Wenkel, Khaled Alhazmi, Tanel Liiv, Saud Airshoud, and Martin Simon. Confidence score: The forgotten dimension of object detection performance evaluation. Sensors, 21(13):4350, 2021.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4/78, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105, 2024.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems (NeurIPS), volume 1 1126, 2024.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858, 2023a.
Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint 1rXiv223/2.157/0, 2023b.
Lianmín Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS), 06:446595-466233 2023.
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large languaee models. arXiv preprint arXiv:2310.00755 2023.
14

---

Published as a conference paper at ICLR 2026
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing laiiual understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2026
Self-Aug: QUERY AND ENTROPY ADAPTIVE DECODING FOR LARGE VISION-LANGUAGE MODELS
APPENDIX
Due to space limitations in the main manuscript, provide supplementary materials in this appendix that elaborate on the proposed design, experimental settings, and visualizations. This includes the complete prompt design for Self- Seff-Augmentation Selection (SAS) and LLM-as-a-Judge for reasoning quality, additional qualitative examples, extended experimental results, and a detailed breakdown of the model and benchmark information.
A ADDITIONAL EXPERIMENTAL SETUP DETAILS
The visual augmentations were implemented top of the official VCD (Leng et al., 2024) source code with the following specific parameters. The color inversion operation was performed using the PyTorch torchvi s sion. trans forms. funct onall 1, invert function. For the random crop and random mask augmentations, a ratio of 2.0 was used, which corresponds to applying the operation randomly placed square patch with side lengths equal to half the original image dimensions. For the noise augmentation, diffusion noise step of 500 was applied from the official VCD random noise implementation.
B FULL PROMPT DESIGN
This section provides the verbatim prompts used for both the self-augmmentation selection and the subsequent reasoning quality assessment. The full SAS Prompt, which leverages in in-context learning and reasoning to achieve optimal guery-augmentation semantic alignment, is presented first. This followed by the prompt used to instruct the LL-M-as--- LM-as-a-Judge for the evaluation of the SAS reasoning trace against the Oracle. The individual effects the reasoning and in in-context learning components within the SAS Prompt are quantified in the ablation study section of the main manuscript.
B.1 SAS PROMPTING
You are an expert data augmentation analyst. Your task is to select the single most semantically disruptive image augmentation that most effectively invalidates the question's premise prevents confident answer. Provide clear reason explaining why the augmentation is chosen, then your final choice.
## Augmentations and Their Effects ## Vertical flip: Flips image top-to-bottom. Disrupts questions about "above", "below", "under" reading orientation.
Color inversion: Replaces each color with its complement. Disrupts questions relying on accurate color identification.
Random crop: Removes random parts of the image. Disrupts questions requiring global context peripheral objects.
Random mask: Occludes portions of image. Disrupts object presence, count, or attribute recogni- tion.
Noise: Adds visual distortion. Disrupts questions requiring small details, texture, text clarity, Horizontal flip: Flips the image left-to-right. Disrupts questions about left/right positioning and left-to-right text reading.
## Examples ##
Question: "Is the mirror above the TV?" Reason: The question focuses vertical positioning. Ver- tical flip reverses top and bottom, making "above" mean "below." invalidating the question. Other augmentations don't affect vertical relationships. Choice: vertical flip
16

---

Published as a conference paper at ICLR 2026
Question: "Is this photo taken indoors?" Reason: The question requires identifying specific envi- ronmental context. Random crop may exclude key background elements like trees, invalidating the question. Flips, color inversion, noise, and random mask don't directly affect scene context. Choice: random crop
Question: "Are there any green beans in the image?" Reason: The question requires identifying specific color. Color inversion changes green to its complement, invalidating the question. Flips, noise, random mask, random crop don't target color directly. Choice: color inversion
Question: "How many people are in the image?" Reason: The question requires counting visible people. Random mask can completely obscure one or more people, making the exact count impossible. Noise obscures details but typically doesn't 't hide entire objects, allowing approximate counting. Flips and color inversion don't ' affect object visihility count. Choice: random mask
laptop?" Reason: The question relies on horizontal positioning. Horizontal flip reverses left and right, making "right" mean "left", invalidating the question. Other augmentations don't t target horizontal positions. Choice: horizontal flip
Question: "Is the cat on the right side
Question: "Does this artwork exist in the form of painting?" Reason: The question requires identifying the texture of the artwork. Noise obscures fine details, making hard to identify the medium. Other augmentations don't target texture details. Choice: noise
## Your Answer ##
multiple augmentations could disrupt the question, select the whose effect is most direct and unambiguous. You must choose one of given augmentations following the "Reason:" and "Choice:" format. Question: "[text)"
B.2 LLM-AS-A-JUDGE PROMPT FOR REASONING QUALITY
Your task evaluate a candidate model's 's response against an expert-provided reference solution. The goal is to select the image augmentation that most effectively disrupts the premise of given question.
## Evaluation Rubric (Integer Scale 0-10) ##
(Excellent): The candidate's choice highly effective and the reasoning clear, logically sound, and directly supports the choice. The response is of reference quality.
7-9 (Good): The choice is effective and the reasoning is logical, but may be slightly less specific or insightful than the reference.
4--6 (Acceptable): The choice is plausible but not optimal. The reasoning is generic, weak, or contains minor flaws.
- 1-3 (Poor): The choice is ineffective and the reasoning is flawed or irrelevant.
0 (Very Poor): The choice and reasoning are completely incorrect or nonsensical.
## Reference Example ##
Question:
"How many people are in the image?"
Reference Reason: "The question requires counting visible people. Random mask can completely obscure one more people, making the exact count impossible."
Reference Choice: "random mask"
Candidate Reason: "Random crop might cut some people out frame."
Candidate Choice: "random_crop"
Evaluation: Score: 7, Reason: The candidate's 's choice is valid strategy for disrupting a counting task, but it is less direct than the reference. The reasoning is correct but lacks specificity.
## Task
Question: "[question)"
Reference Reason: "(oracle reason)" }"
Reference Choice: "(oracle choice)"
Candidate Reason: "I reasooo "(model_choiee
Candidate Choice:
Evaluation:
17

---

Published as a conference paper at ICLR 2026
C
MODEL AND BENCHMARK DETAILS
Model Families
LLLVA-1.5 (Liu et al., 2023) is a powerful open-source LVLM that establishes the ef- fectiveness of visual instruct tuning for creating general purposs visual assistants. Its architecture is characterized by its simplicity, connecting pretrained CLIP vision encoder Vicuna LLM using a single Multi-Layer Perceptron projection layer. The LLaVA-1.5 version improved upon the original by incorporating a more capable LLM and scaling the instruction-following data.
Qwen- VL (Bai et al., 2023) is series of highly performant, versatile vision-language models based on the Qwen language model family. A key feature of the Qwen-Vi architecture is its support for multiple languages, the ability to process multi-image inputs, and its strong capabilities in fine-grained visual understanding, including text recognition and object localization.
InstructBLIP (Dai et al., 2023) is vision-language instruction tuning framework designed to enhance zero-shot generalization across a diverse set of tasks, Its central innovation is the use of an instruction-awe features from the image encoder that are specifically relevant to the given text instruction, enabling more targeted and effective multimodal reasoning.
Query Transformer. This module trained extract visual
Qwen3-VL (Bai al., 2025) is the latest evolution in the Qwen vision-language series, engineered to overcome the limitations of fixed-ree reed-resolution processing and enhance visual agent capabilities. Its architecture employs Naive Dynamic Resolution, which processes images at their native aspect ratios by converting them into variable-length token sequences, effectively eliminating the loss of detail caused by resizing or padding. Combined with Mul- timodal Rotary Positional Embeddings (M-RoPE) that decompose positional information into temporal, height, and width components, the model achieves state-of-the-ar perfor- mance in comprehending high-resolution documents, long-context videos, and executing complex tool-use use instructions.
Discriminative Benchmarks
MME (Fu et al., 2024) is benchmark that provides granular evaluation of multimodal tasks, spanning 10 perception and 4 cognition categories. The performance is measured on binary yes or no questions using an accuracy-based MME score. Following the standard practice (Leng et al., 2024; Kim et al., 2024b), we consider the perception category for the experiments.
MMVP (Tong al., 2024) is designed to evaluate a model's understanding of fine-grained visual details. It achieves this by using 300 CLIP-blind image pairs, where models must capture subtle differences to perform paired classification accurately. These image pairs cover nine distinct visual patterns: orientation and direction, feature presence, state and condition, quantity and count, positional and relational context, color and appearance, structural and physical characteristics, text, and viewpoint and perspective. The evaluation follows multiple-choice format, where final model responses are mapped to the answer options using GPT-4 as automated judge.
POPE (Li et al., 2023d) serves as dominant benchmark for assessing object hallucination by testing models with three distinct types of negative questions. These categories include queries about random non-existent objects, popular objects that are frequent in the dataset but absent from the image, and adversarial objects selected their high co-occurrence. The dataset contains 9,000 question- pairs built from 500 images, each evaluated against multiple questions across the three categorics.
Generative Benchmarks
LLLVVA-eena (In-the-Wild) (Liu et al., 2023) is benchmark to evaluate the ability of Large Vision Language Models (LVLMs) to handle complex tasks and adapt to new domains. It features 24 images and 60 queries, which collapse into three categories: conversation,
18

---

Published as a conference paper at ICLR 2026
detailed description, and complex reasoning. The evaluation is conducted using GPT-4V as a judge to rate both the model response and a reference answer. The final performance is reported as score ratio, calculated by dividing the total score of the reference answer.
(Sun et al., 2023) evaluates and penalizes hallucinations across diverse set of reasoning types. is composed 96 image-question pairs that cover eight distinct categories, including object attributes, comparison, and spatial relations. Evaluation is performed using GPT-4V as an automated judge to assess the severity of hallucination the generated response. The responses scored scale from 0 to where higher score indicates greater facutal consistency.
MMHal-
MM-Vet (Yu et al., 2023) evaluates an LVLM to integrate multiple multimodal capabilities for complex reasoning. The benchmark defines six fundamental multimodal abilities: recognition, knowledge, OCR, spatial awareness, language generation, and mathematics. A key feature of Me- is its focus compositional tasks, where these six core abilities combined to create 16 distinct capability integrations. The dataset itself is composed 200 images and 218 questions, each requiring specific combination of these integrated skills,
D ADDITIONAL QUALITATIVE RESULTS
To provide granular understanding of the behavior of the method, this section presents additional qualitative results from both discriminative and generative benchmarks. Each example provides a comprehensive analysis that includes the reasoning trace for the chosen augmentation, stylized visualization of the augmentation, the logit values for the expert, amateur, and contrasted distributions, and the corresponding Sparsity Adaptive Truncation (SAT) threshold. For improved visualization clarity, common punctuation tokens such as commas and periods have been omitted from the presented logit distributions.
Expert token
Expert token
Left
Amaleur token
b
Amaleur token
Left
MILLER MOTORCARS
token
8
Final token
Right
Expert logit
20.88
Expert logit
22.62
20.77
22.38
Amateur logit
20.89
Amateur logit
22.89
20.50
22.03
20.88
22.56
Query:
Final logit
Query:
Final logit
How many spots are the animal in the image? (a) (b)
20.50
Is the sports the picture facing left or right?
22.72
0.361
0.253
The question requires counting the number animal. Flandom mask completely obscure the animal, making impossible to count the spots. Noise and flips don't affect visibility of the animal the count.
Reason:
determining direction the sports car facing. Hortzontal flip reverses left and right. making 1etf" moan "right" and vice versa, invalidating the question. Other augmentatiins don't target horizontal positions.
Reason:
The question
Figure A.1: Qualitative results on MMVP (Tong et al., 2024).
19

---

Published as a conference paper at ICLR 2026
Expert token
No
topan
token
_No
Amateur token
No
Final token
Yes
26.
27.56
Expert logit
Expert logit
27.56
25.33
27.36
28 84
Amatour logit
Amaleur logit
28.64
23.1 19
25.53
24.95
26.48
Query: Is there red scarf
Finallogit
24.95
Fogil
28.48
Query:
27.47
the image?
Are there two horses in this image?
0.380
Reason:
question requires identilying
Reason: question requires kdantilying the presence horses. Random can completely abscure
object. Color inversion changes
makiing exact presence impossible. noise and flps don't affect the numbor of horses or their visibiltt.
complement, making impossible determine there is red scarl in the image. Flips, noise, random mask, and random crop don't direcity affect color identificaion.
Figure A.2: Qualitative results MME (Fu al., 2024).
Espat Juhan
Tte
Jeghight
.61
Araaoor towm
_Nghight
.507
-55
.rage
17.00
15.25
23.14
19.73
24.11
24.00
21.52
25.17
18.53
2150
15.23
Deperlogt
17.00
18:50
73.14
18:75
34:11
94.00
91.55
35.17
18:3
21.0
16:6
Anomuringg
10.32
24.00
24.00
21.44
2135
15.22
17:80
18:26
23.00
10:52
24.00
26.90
21.44
25.00
1853
23.6
15.22
Query:
Fhaelogtt
17.11
1832
23.22
19:28
24.22
24.00
21.52
22.00
18.22
23.65
15.20
Which of these highlighted otties the
0.190
0.341
3.67
0.342
3.500
0.584
0.986
0.665
0.351
0
6.48
0.966
Resson: The ip queedion wegu nes ddwinilynng the reket pos fon cille.. Nokie obscunes Ehe map and makou 8 diliko delemmine the didances belanee the diec Other bugmentat ons don't affect the no othee positione of difee.
Eapentrion
_To
J6
.oott
Icssdd
.B
20
Amate
Fingl Julann
Thel
8
.e65iice
3c33d
30
36
35
<5m
Eqpart ligit
21.53
25.9
21.09
19.95
22.7
30.13
25.66
27.78
21.87
25.15
21.8
25:00
22:99
23.59
20.12
22.34
20.99
21.15
22.00
22.00
19.44
22.00
27.11
22.28
25.55
2400
22.17
logle
21.05
32.08
25.00
18:11
22.00
27.31
20.28
29.37
21.32
27:11
29.47
17.15
21.39
2034
22.00
28.45
22.06
25.00
27.34
2581
75.33
right
21.52
1744
26.19
20.4
23.89
18.45
22.88
21.34
12.02
the shampoo?
0.408
Reason: question requress donnthyng position restive anothorr Randomocop the contoot inlationshooo maldin impossblee poaitir. Figua color rivaraion nales nandom dirucilyy ralalion postaarnng
Figure A.3: Qualitative results MM-Vet (Yu al., 2023).
20

---

Published as a conference paper at ICLR 2026
Exaetizion
The
.113
.20
.3
p900
<
Fut Iilin
-Te
-0009
-4
110
.001
15
29:00
1801
-30
1200
150
133
96.66
90.2
31.41
56:18
96.76
26.11
2605
91.03
37.50
22.44
22.44
26.19
26:78
26,11
26.75
20.73
27.58
23.00
21.44
12.44
50.19
20.70
26.11
26.35
26.39
13.21
22.82
5.
27.5
27.38
Araaaar logi
26:06
21.00
32.34
18.64
36.19
55.56
50.00
7667
06.00
96.87
30.29
18.07
18.84
31.66
27.18
Query:
20.86
20.07
92.77
19.08
34.36
58.00
58:88
96.20
20.19
37.30
30.05
3134
81
2968
What colors of shirts right in the image?
Peolingt
20.04
20:47
27.77
20.00
23.35
23.07
20.06
20.00
26.70
2012
27:00
2331
24:33
31.54
27.6
The randomadoo crop-doth color parception.
ohints, Colorimeesionn each
inpossiblee desemmnt
ArcanurtSiee
The
35
26
.81
335
33
</t
27.91
29.41
55.54
56.12
26.7
26.35
31.03
34:33
19
19.65
22.02
26.77
Expertogh:
27.51
23.41
23.34
25.12
25.72
23.25
25.35
34.2
1325
1325
25.20
28.77
28.08
1750
17.20
26.72
22.50
Tial lgg
13:88
18.02
What are calors parachutee In the sky?
12.8
22.50
0.87
0382
2.188
0.09
0272
Reasor: The q juedton reculiee lyeng specilio. oolooo oolor inneiion dhangee eschoolkr 12 entert, making impoeeite deternnn teorgholoooon Flls. ritee, random mast, and tandom crop dur' direcly whocl ralor perceptiion
Figure A.4: Qualitative results on MMHal-Bench (Sun et al., 2023).
21

---

Published as a conference paper at ICLR 2026
Confusion Matrices: LLaVA-7B vs. GPT-4o-mini
LLaVA-Bench
MME
herontel Ri.
0
Pevioorttl fip
0
17
Oracle (GPT (GPT-Ao-mini)
104
(GPT
readeru
21
seden, sww.
35
14
935
radon.raak
56
45
45
Shaetton
wetset
wetse
Prediction
Prediction
MM-Vet
MMHHa- Bench
hwricenel
4
herfoontal..0 fib
Oracio (SPT-4o-minI
(GPT-40-mi
raakin
14
87
sedeen
14
14
vittellcc Ao
vertoal 1t
makeo Jever
Aesional
seddes meak
Prediction
Prediction
MMVP
POPE
Inveries
28
14
16
T-to-min))
Oracie (GPT (GPT-4o-min))
nobe
12
nolee
13
66
randen,
346
2013
19
verioal.b
rendee meak
me
noston
Prediction
Prediction
Figure A.5: Confusion matrix for LLAVA-1.5 7B model.
22

---

Published as a conference paper at ICLR 2026
Confusion Matrices: LL3VA-13B GPT-4o-mini
LLaVA-Bench
MME
eeer Invendes
OWT Ievendes
57
178
Oracio (GPT- -ir-Aomminn))
neee
163
16
20
13
840
meder
verteet
25
vdder hatternal by
nowe
Invvetton
sondom mare
testiel
Iesterett
Prediction
Prediction
MM-Vet
HMMHall-Bench
horibaraal
Oraeto (GPT-4ominn)
nele
20
Oracle
medor mark
S7
54
Notionill Pl
reniemt mark
Noteoaal
Prediction
Prediction
POPE
Nortontal.5b Ra
horooral
Gracle (GPT 40--440-min)
(GPT
reddem cre
15
sedee
verisal
reteee mare
reriume
Prediction
Prediction
Figure A.6: Confusion matrix for LLAVA-1.5 13B model.
23

---

Published as a conference paper at ICLR 2026
Table A.I: Comparison of model performance against the gpt-4o-mini oracle. Agreement measures the accuracy percentage (%) of augmentation choices, and Judge Score estimates the quality rating of the model reasoning on a 10 scale by gpt-4o-mini.
Model
Metric
VL--enhench
MME
MM-Vet
MMHal
MMVP
POPE
Average
Agreement (%)
56.67
72.56
61.93
54.17
52.00
87.57
64.15
LLaVA-7B
Judge Score
7.97
8.59
7.98
7.79
7.69
9.63
8.28
Agreement (%)
51.67 8.63
69.77
40.37
63.54
72.67
99.10
66.19
LLaVA-138
Judge Score
9.12
8.55
9.08
8.86
9.99
9.04
E DETAILED COMPARISON AGAINST ORACLE
In the main script, experiments were conducted evaluate the impact the model scale on the quality of augmentation choice and reasoning. The agreement of each model's choice and the quality of its reasoning trace were measured against the Oracle, with the results summarized Tab. A.I. These results confirm that larger model capacity generally leads to better auery-augmentttion semantic alignment and higher reasoning quality.
A more granular analysis using the confusion matrices in Fig. A.5-A.6, reveals complex, task- dependent relationship. On uniform benchmarks such as POPE, the alignment between the 13B model and the Oracle is nearly optimal. In contrast, the 13B model exhibits a predictive bias, frequently selecting random crop when the Oracle chooses the functionally similar random mask. Note that this disagreement is not critical failure, but rather choice between two functionally similar occlusion-base augmentations.
complex benchmarks such as MM-Vet,
This finding highlights key strength of the proposed method. The fact that strong downstream per- formance is achieved without requiring a perfect, Oracle-level selection confirms that the framework highly effective leveraging the competent, albeit imperfect, reasoning of different model scales to significantly improve factual consistency.
24


Reviews:
[{"review_id": "oWGD0ZOHi2", "reviewer": "Reviewer_cNnA", "summary": "This paper addresses the issue of hallucination in Large Vision-Language Models (LVLMs) to improve their generation quality. Previous works have employed contrastive decoding (CD) to enhance the output of Large Language Models (LLMs), an approach later extended to the visual domain as Visual Contrastive Decoding (VCD). Although promising, existing VCD methods often rely on generic visual augmentations that are agnostic to the specific context of the text query, limiting their effectiveness. To address this, this paper introduces a training-free decoding strategy with two key innovations: first, a self-augmentation prompting technique that leverages the model's intrinsic knowledge to dynamically align semantics between the query and visual content; and second, an adaptive thresholding algorithm that adjusts the candidate token set based on output sparsity, making full use of the logit distribution. Extensive experiments validate the method's effectiveness across four LVLMs and seven benchmarks.", "questions": "Please refer to my weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well-organized with good writting. Importantly, the authors provide extensive experiments to validate the effectiveness of their method.", "weaknesses": "**Novelty Assessment**: As I am not deeply familiar with the contrastive decoding literature, I cannot authoritatively assess the novelty of the proposed method against all prior works in Visual Contrastive Decoding (VCD). The idea appears novel based on my reading, but a more thorough discussion of related work would help situate this contribution for a broader audience. I look forward to the discussion with other reviewers and the authors' response on this point.\n\n**Model Generalization**: The experimental validation, while extensive, relies on a set of LVLMs that does not include several recent state-of-the-art models (e.g., InternVL3/3.5, Qwen-VL2/3). To strengthen the claim of generalizability, it would be valuable to see if the proposed SAVCD method's benefits hold on these newer architectures.\n\nNote to the AC and Reviewers: Given my limited background in this specific sub-field, I will rely heavily on the discussion and author response to finalize my assessment, and I defer to the experts on the finer points of the related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uN7GRP5d7L", "reviewer": "Reviewer_1xaB", "summary": "This paper proposes Self-augmented Visual Contrastive Decoding, a training-free decoding strategy designed to mitigate hallucination problem in Large Vision-Language Models (LVLMs). SAVCD consists of two main components. First, it dynamically selects a query specific visual augmentation that produces maximally informative discrepancies between expert and amateur logits. Second, it applies contrastive decoding based on the self-augmented selection-based augmented image to penalize tokens that are likely to cause hallucination, without requiring additional training. The method significantly reduces hallucinations compared to existing baselines and achieves high computational efficiency", "questions": "Please refer to the Weaknesses part.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths are summarized as follows:\n\n(1) Conceptual Advancement: Query-Aware model-driven self-augmentation to reduce the hallucination effect of VLMs.\n\n(2) Practical and Efficient mitigation without retraining method. \n\n(3) Comprehensive evaluation and well-written to follow.", "weaknesses": "(1) The performance of SAVCD remains dependent on the underlying LVLM, limiting its robustness when transferred to different architectures or weaker base models. It would strengthen the paper to include expedriments on multiple LVLMs with varying parmeter scales to show cross-model consistency. \n\n(2) The range of available visual augmentation is still constrained to a predfined candidate set, even though the method dynamically selects among them. \n\n(3) VaCoDe Misrepresentation; While the paper highlights a major distinction fro mVaCoDe, the description partially misrepresents VaCoDe's mechanism. Specifically, VaCoDe also selects augmentations dynamically from a fixed set based on output divergence, rather than using purely fixed perturbations. It would strengthen that clarifying the comparison in the main text between VaCoDe and SAVCD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIA23lFBHH", "reviewer": "Reviewer_msRD", "summary": "This paper proposes a training-free decoding strategy named SAVCD, aimed at mitigating hallucinations in Large Vision-Language Models (LVLMs). The method features two core contributions: 1) a Self-Augmentation Selection (SAS) prompting strategy that leverages the model's own knowledge to dynamically select the most relevant visual augmentation for constructing a meaningful \"amateur\" model in contrastive decoding; and 2) a Sparsity Adaptive Truncation (SAT) algorithm that dynamically adjusts the threshold for candidate tokens based on the entropy of the output logit distribution to improve factual consistency. Experimental results demonstrate that the proposed method outperforms existing decoding strategies across several benchmarks and models.", "questions": "1.  Regarding the robustness of SAS selection: What is the accuracy of the model's augmentation selection in the first step? Are there cases where the model selects a suboptimal or incorrect augmentation, leading to worse results than the baseline? Could the authors provide an analysis of such failure cases?\n2.  Regarding the limitations of the SAT algorithm: The core of SAT is using entropy to measure model uncertainty. However, models can sometimes produce a completely wrong hallucination with very high confidence (i.e., very low entropy). In such \"confidently wrong\" scenarios, could SAT's stricter threshold inadvertently lock in this error, making it harder to correct?\n3.  Regarding the generality of the prompt: The SAS prompt appears quite elaborate. How much would the performance degrade if this prompt were simplified (e.g., providing only the names and simple descriptions of augmentations, without the ICL examples)? This would help in understanding to what extent the performance gain comes from the method itself versus the elaborate prompt engineering.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The paper presents a complete and well-designed decoding framework. The experimental section is thorough, covering multiple models and benchmarks, and the results consistently show that the proposed SAVCD method outperforms baselines in improving factuality. This demonstrates its practical utility as a plug-and-play tool.\n2.  The core idea of having the model self-select the most appropriate visual augmentation via prompt engineering to construct an \"amateur\" perspective for contrastive decoding is highly novel and interesting. It explores the possibility of using a model's own reasoning capabilities to guide its decoding process, offering a new perspective on mitigating LVLM hallucinations.\n3.  As a purely inference-time strategy, SAVCD requires no additional training or architectural modifications. This gives it strong generality and a low barrier to adoption, making it an attractive feature for the community.", "weaknesses": "The main weakness of this paper lies in its methodology, which is built on an assumption that may not always hold, and the solution relies to some extent on elaborate engineering tricks, which may affect its robustness and generality.\n\n1.  **Questionable Core Methodological Assumption**: The core of SAVCD, the SAS prompt, relies on a critical assumption: that a model prone to hallucination and reasoning failures can accurately interpret a complex prompt to select an optimal corrective strategy for itself. This introduces a degree of circular reasoning. If the model's foundational reasoning is strong enough to perform this selection task perfectly, the initial problem of hallucination might be less severe. This design makes the method's success partially dependent on the very problem it aims to solve.\n2.  **Over-reliance on Prompt Engineering**: The method's effectiveness is heavily dependent on a meticulously crafted, complex prompt that includes extensive descriptions and examples. Such approaches are often brittle, being sensitive to changes in wording, structure, or even model versions. Furthermore, its scalability is limited; for instance, introducing new augmentation types would require manually modifying and debugging this complex prompt. This makes the method feel more like a highly-tuned heuristic for a specific set of tasks and models rather than a general, robust principle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D4u4O58Xcz", "reviewer": "Reviewer_poDY", "summary": "This paper tackles hallucination in Large Vision-Language Models (LVLMs) by improving visual contrastive decoding (VCD). It introduces SAVCD: (1)a self-augmentation selection (SAS) prompt that asks the LVLM to pick a semantically-disruptive augmentation for the current query, and (2) Sparsity Adaptive Truncation (SAT) — a thresholding rule that uses the entropy of the expert logit distribution to set the plausibility cutoff. At each step, SAVCD contrasts the expert and amateur logits, then removes tokens below the SAT threshold before sampling. Experiments on multiple LVLM families and 7 benchmarks show consistent gains over prior VCD methods.", "questions": "I would appreciate it if the authors can share any information about applying it to newer models, if they had done experiments on that before. This would help me determine if this method has value for today's VLM development.\n\nI am giving a slightly higher score, as I hope authors can address my concerns.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The technical soundness of SAS and SAT is solid. Authors have conducted quantitative experiments and intuitive explanations to demonstrate its effectiveness.\n2. The experiments, though tested on old models, can be rated as solid.\n3. The paper is well-written: clear and easy to follow. The figures are carefully polished and informative.", "weaknesses": "1. The contributions sound incremental, given that VCD has already been widely researched. SAS and SAT make sense intuitively and experimentally, but might not be enough contribution for a top tier conference. However, I am not an expert on LLM Decoding, so I will carefully refer to other reviewers' comments on novelty and contribution before making my final decision.\n2. The SAS Augmentation Choice seems arbitrary to me, as \"Noise, Horizontal Flip, Vertical Flip, Random Crop, Random Mask and Color Inversion\" only presents a subset of the image's properties. For instance, when asked about size, reachability, OCR or spatial relationship, chances are that these Augmentation choices will not work.\n3. The experimental results don't show robust generalization across datasets and models: for instance, SAVCD in Qwen-VL or POPE-MSCOCO consistently underperform other methods. \n4. The experiments are conducted on rather old models like LLaVA-1.5, Qwen-VL, which may cast doubt on its effectiveness on its effect on contemporary models. The claim could be strengthened with an experiment on Qwen2.5-VL.\n\n(minor) For Table 1: LLaVA-1.5-7B (POPE-MSCOCO) Acc. SOTA should be VACoDe, not SAVCD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)