VSCoDe: Visual-Augmentation Selection for Contrastive Decoding
Authors: Sihyeon Kim
Boryeong Cho
Sangmin Bae
Sumyeong Ahn
Se-Young Yun
Venue: Accepted by TMLR ()

Abstract: Despite the impressive performance of recent Large Vision-Language Models (LVLMs), these models often produce inaccurate responses. To address this issue, previous studies have aimed to reduce hallucinations by using contrastive decoding (CD) with modified images, such as cropping objects related to query or adding noise, thereby contrasting with the original image. However, these methods have several limitations. First, employing fixed visual augmentation, such as adding noise, is a simple approach but too rigid to contrast on various queries. Conversely, using semantics in queries or images by leveraging external models can adaptively generate contrastive images, but it entails significant additional costs. To address these shortcomings, we explore using pre-defined visual augmentations to enable flexible adaptation to each query without relying on external models. We observe that each query achieves different contrasts through different visual augmentations. Based on this, we propose a novel method called VSCoDe, Visual-Augmentation Selection for Contrastive Decoding, which adaptively selects augmentations using a proposed distance metric to identify those with higher contrast. Our empirical evaluations demonstrate that VSCoDe outperforms previous methods and enhances the quality of various vision-language tasks without additional training or reliance on external models.

Published in Transactions on Machine Learning Research (09/2025)
VSCoDe: Visual-Augmentation Selection for Contrastive De- coding
Sihyeon Kim" KAIST AI
sihk@kaist.oo oc.kr
Boryeong Cho' KAIST AI
vennhammkkaist.cc.kkk
Sangmin Bae
bemn@e22@kaist..oc.
KAIST AI
Sumyeong Ahn KENTECH
sumjeongohn&kkenteccoookkk
Se- Young Yun KAIST AT
yunseyoung@kaiis..co oc.
Reviewed on OpenReview: https: o/penrevee, net/ forum? id= CQSyPc9W7Y
Abstract
Despite the impressive performance recent Large Vision-Language Models (LVLMs), these models often produce inaccurate responses. To address this issue, previous studies have aimed to reduce hallucinations by using contrastive decoding (CD) with modified images, such cropping objects related to query or adding noise, thereby contrasting with the original image. However, these methods have several limitations. First, employing fixed visual augmentation, such as adding noise, is simple approach but too rigid to contrast on various queries. Conversely, using semantics in queries or images by leveraging external models can adaptively generate contrastive images, but it entails significant additional costs. To address these shortcomings, we explore using pre-defined visual augmentations to enable flexible adaptation to each query without relying on external models. We observe that each query achieves different contrasts through different visual augmentations. Based this, propose novel method called VSCoDe, Visual-augmentation Selection for Contrastive Decoding, which adaptively selects augmentations using proposed distance metric to identify those with higher contrast. Our empirical evaluations demonstrate that VSCoDe outperforms previous methods and enhances the quality of various vision-language lasks without additional training or reliance on external models.
1 Introduction
Pre-trained Large Vision- Language Models (LVLMs) (Liu et al., 2024a; Ye et al., 2023; Zhu et al., 2023; Dai et al., 2024; Li al., 2022; 2023a; Radford et al., 2021) have gained prominence due their ability to understand multiple data formats, especially vision and language, simultancously. These models have demonstrated exceptional performance in various tasks such as zero-shot image classification (Radford et al., 2021; Yao et al., 2021), image-text retrieval (Yao et al., 2021; Li et al., 2022), visual question answering (Dai al., 2024; Liu et al., 2024a), and image captioning (Li al., 2022; 2023a). Unlike carlier encoder-basod models like CLIP (Radford al., 2021), most recent largo-scale VLMs, such as LLaVA (Liu al., 2024a),
Authors contribute equally
i Corresponding Authors

---

Published in Transactions on Machine Learning Research (09/2025)
Question
Applying Single Augmentation
Contrastive Decoding
Where is the cat?
Color
Flip
Crop
Noise
PCB color
Forig
cotor
Image
left
Answer:
right
Incorrect
left right
left
left right
Edge
on
left right
right
Fortg
cotor
crop
CD kerop
X
GD "Rois
X
CD
Answer: left
Incorrect
Augmentation Selection
Distance between lorig and Laug
Flip
gip
loris E lyip
Vanilla
left
Answer: right
Single Aug
0:12
0.54
0.33
0.09
0.14
Lpip
right
Correct
VSCoDe
Figure 1: CD with single augmentation may not yield correct answers for all types of questions, it may lail to modily the key visual feature related to the semantics of the query. For example, the visual feature of the position-related query is affected by flip augmentation, whereas color augmentation does not. Therefore, it is essential to apply the appropriate augmentation for each query when applying CD. VSCoDe selects the augmentation with the largest distance D(-) we define to identify the augmentation that modifies the key feature of the image relative to the question. VSCoDe successfully selects the augmentation, enabling CD to produce the correct answer.
MPlugOWL (Ye et al., 2023), MiniGPT-4 (Zhu et al., 2023), and InstructBLIP (Dai et al., 2024), utilize autoregressive language decoders to expand their functionality, allowing them to cover more complex tasks.
However, language decoders sometimes produce incorrect outputs, phenomenon called hallucination. Among various methodologies (Wei et al., 2022; Rose et al., 2023; Shao et al., 2024), promising approach is contrastive decoding (CD) (Li et al., 2023b), which generates final answers by examining original outputs through contrastive outputs. More precisely, CD works in two stages: (1) generating output distributions given original and contrastive prompts each, and (2) subtracting the two output distributions to reduce the likelihood of hallucinated tokens. The effectiveness of CD relies on how well the contrastive prompt is generated and facilitates the contrastive predictions effectively. While creating contrast in language models is relatively straightforward by replacing original words with their opposites random words (Kim et al.; Wang et al., 2024)- vision-language models require more deliberate approach as no clearly defined strategy exists for generating contrastive images.
Several works have been performed on the generation of contrastive images, but they are limited to various semantics considerations. VCD (Leng et al., 2023) adds Gaussian noise to the image, but does not consider the semantics of the query. On the other hand, some studies, such as HALC (Chen al., 2024) and CRG (Wan et al., 2024), have attempted to understand semanties generate bbject-manipulated images. However, they are limited to object-level semantics and involve the additional cost of using external models for object detection. Therefore, we pose the question, "How can we leverage pre-defined cheap visual augmentation operations while incorporaling semantic understanding?"
To answer our question, we first empirically observe that the augmentation required to generate contrast. varios depending on the query's semantics. For instance, as shown in Figure 1, when the question pertains to position, using position-related augmentations such Flip creates significant contrast. Consequently, selecting augmentations that induce more contrastive predictions for each query can make CD more effective. Building on this intuition, we hypothesize that augmentations producing greater distances between the logits of the original and augmented images facilitate contrasting predictions.
Based on our findings, we propose VSCoDe, a novel method for automatically selecting augmentation for CD, which generates appropriate augmentation based on the semantic of query without requiring additional training or external model. As illustrated in Figure 1, VSCoDe involves three steps: (1) given multiple candidate augmentations, provide various types of augmented images to VLMSs and generate single token output
2

---

Published in Transactions on Machine Learning Research (09/2025)
each image, (2) measure the distance D(-) between output distributions of the original and augmented images, and (3) select the most contrasting output with the maximum distance D(-) to achieve the final CD result.
Our contributions can be summarized as follows:
We explore the effect of visual augmentation on LVLMs, Our findings indicate that each augmentation has a distinct impact on the given question, altering the output distribution of VLMs and subsoquently affecting the response. It highlights the importance of using appropriate augmentation depending on the query.
Based on the findings, we introduce an algorithm called VSCoDe that selects contrastive augmentation to empower CD capability without additional training or using external models. Through distance D(-), VSCoDe automatically determines the appropriate visual augmentation for given query semantics, thereby achieving a higher level of CD effect.
We extensively evaluate the performance of VSCoDe across various tasks, including Visual Question Answering (VQA) the MME (Fu et al., 2024), MMBench (Liu et al., 2024b), VQAv2 (Goyal et al., 2017), and POPE (Li et. al., 2023c), well as capt ioning tasks on the MSCOCO (Lin et al., 2015) dataset using LVLMs VSCoDe shows improved performance on each task, and in particular, on the MME benchmark with the LLa 7B model, VSCoDe outperforms both single augmentation approaches and previous CD-based methods, yielding a 1.82x larger performance gain compared to previous methods.
2 Preliminaries
Here, we provide a concise summary of background information to aid in understanding this research. We specifically discuss LVLMs, visual data augmentation, and contrastive decoding.
Generative LVLMs. LVLMs are among the most prominent multi-modality models. They process pairs of imput image e and text quusstqusti q, denoted as (v,4), and gencrate answers by utilizing the visual information within v. In this paper, we primarily focus on generative LVLMs, rather than CLIP-like (Radford et al., 2021) models. These generative LVLMs produce tokens one at a time sequence similar to LLMs. The mathematical expression for this process is:
Yt p(u U, q, Mce).
Here, p(-) represents the softmax of the output of the vocabulary set, and yer denotes the tokens generated up to but not including the timestamp I. Like LLMs, IVLMs are also prone to hallucination (Li et al., 2023c; Liu et al., 2023a; Tong et al., 2024), where the model erroneously assigns higher probabilities to tokens that do not factually exist in the provided image.
Visual Augmentation (VA). VA consists of long-ostablished techniques that modify visual data to produce desired images for computer vision research, such as enhancing sharpness, adjusting color jitters, and more. While some augmentation techniques, like mixup (Zhang et al., 2017) or CutMix (Yun et al., 2019), require combining more than two images, our discussion focuses on single-image augmentation operations for simplicity. We focus on a specifie framework:
J' = O'E(())))
where (v)) represents a manipulation on the image and an augmentation within the set A. In this paper, we employ the augmentations A =ccollrr, flip, random crap, random erase, sharp, edge, noise). Examples are illustrated in Figure 2. The descriptions of the augmentations are: (1) color: color inversion, (2) flip: horizontal flip followed by vertical flip, (3) crop: cropping a random part of the image, (4) erase: randomly erasing part of the image, (5) sharp: adjusting image sharpness, (6) edge: extracting edge textures, and (7) noise: adding diffusion noise. Note that use the default noise setting from VCD (Leng et al., 2023).
Contrastive Decoding (CD). The CD approach was first introduced in the language domain (Li et al., 2023b). It operates by generating two outputs using two different models: an expert model that produces the original outputs and an amateur model that generates contrastive outputs. The CD is then performed based the contradictions between them. This idea has also been explored in the vision-language domain,
3

---

Published in Transactions on Machine Learning Research (09/2025)
(a) Original
(b) Color
(c) Flip
(d) Crop
(e) Erase
(f) Sharp
(g) Edge
(h) Noise
Figure 2: Examples of visual augmentations utilized in this paper.
three zippers in the picture?
Q: is there red bicycle with white handlebars the image?
Original
Crop
Color
Original
Crop
Color
Yos
0.61
Yos 0.03
Yos
3.45
Yos
Yes
0.37
Yes
0.01
No
No
(a) Count-type query
(b) Color-type query
Figure 3: The examples show output probabilities from the MME benchmark using the LLaVA-1.5 7B model. The influence of augmentation varies depending on the query type. For instance, in (a) with a count-type question, a cropped image generates contrastive distribution, whereas a color-changed image does not. Conversely, in (b), color-related question is affected by color augmentation.
where manipulated images are used to induce contrastivenoss and help remove unrelated information, such as hallucinations. Instead employing two separale models, the VLM setting uses original and manipulated images as two distinct inputs make the contrast. The process operates as follows:
PCD g. O) (1 a)SOFTMAX (aaeeaxx(f/lle (()+=))))Oxxx((lllo).)
(F
(1)
where f(-) is the output logit obtained from VLM, and a as hyperparameter for the strength of contrastiveness. To amplify the inherent hallucinations in VLMs, VCD (Leng et al., 2023) introduced noise to the image, HALC (Chen et al., 2024) applied cropping, and CRG (Wan et al., 2024) employed object-wise crasing. Another approach, ICD (Wang et al., 2024), involves modifying the text prompt to create confusion in the model's outputs. Subsequently, they subtracted the logits of the hallucinated image from the logits of the original image.
3 VSCoDe: Visual-Augmentatte Selection for Contrastive Decoding
This section explores the impact VAs on LVLMs, focusing specifically on CD. essence, we demonstrate that certain VAs cause either contrast or persistence on key visual feature, implying that the output distribution of the augmented image either varies stays consistent with the distribution of the original image for the given query. Furthermore, distance metric we propose, which relies on the difference between probability distributions. Building on these insights, we present novel algorithm named VSCoDe, which selects contrastive augmentation among multiple candidate augmentations for the given query.
detail our discovery that contrastive augmentation
be identified using the
3.1 Query-Dependens Augmentation Effect
We begin by demonstrating that augmentations affect LVLM outputs differently, depending the information required by the query from the image. This difference occurs because augmentation that modifies crucial feature needed for the query results in shift in the LVLM's output. Conversoly, when the essential fcature is left unchanged, the output remains stable. We define these cases as contrastive augmentation and persistent augmentation, respectively. Figure illustrates the output distribution from differently augmented images, where the bar represents probabilities computed with the softmax function of logits. In the first example, the crop augmentation is a contrastive augmentation, altering the key feature for a count-related question, leading to significant change in probability distribution. Conversely, color augmentation is classified
4

---

Published in Transactions on Machine Learning Research (09/2025)
Figure 4: Both (a) and (b) show Gain measured on the MME benchmark using all augmentations in A. (a) Larger distance D(-) yields larger Gain. (b) Choosing augmentation with the largest D(-)
Table 1: The values indicate the net change from CD per category, with blue and yellow marking the best and worst results. No augmentation performs best across all categories, and effectiveness varies. Results are averaged over five seeds the MME benchmark using LLa VA-1.5 7B.
is most effective in achieving high Gain.
(a) Gain by distance D
(b) Gain by aug.
Category
Aug
Ext
Cnt
Pos
Clr
Pst
Cel
Sen
Lmnk
Art
OCR
Dist
Noise
-2
-15
37
-2
Postive Gain (88)
Edge
Color
-2
11
-6
10
60
76
42
82
68
-15
00-
Color Flip
Crop
13
-4
-11
33
32
49
15
11
-14
1.24
Nolse & Random
Edge
-4
18
0
0
68
91
58
93
27
-13
70-
8
Crop
0.72
Flip
-1
8
2
36
35
31
53
10
-5
60-
Postive Gan
Erase
-4
26
-14
Gai
Sharpness
Sharpness
3
-17
50
0./2 uae 120
Distance
Gain
persistent augmentation, having a similar probability distribution. This effect is similarly observed in the second example.
This effect results in varied performance on the IVLM. To demonstrate the distinct impact of each aug- mentation on different queries, we evaluated CD performance with each augmentation applied to the MME benchmark. The perception lasks in the MME benchmark are calegorized into 10 groups, including color, count, and existence. Observing performance in each category enables us clearly assess the impact of each augmentation on specific question types. Table 1 shows the difference between the number of samples corrected by CD and those that became incorrect due to CD across categories for each augmentation. Positive values, especially higher ones, indicate effective augmentations that likely act contrastively for the respective query type, whereas negative values suggest that an augmentation may not be suitable for that query type. For instance, changing the color of the image acts contrastively for color-related queries by leading to incorrect predictions, while remains consistent with position-related queries, as positional information is unchanged. This underscores that no single augmentation is universally effective, emphasizing the importance of selecting the optimal augmentation for each question.
For effective CD, contrastive augmentation should be applied to guide the LVLM toward generating incorrect answers, thereby increasing the logit for the correct token while decrcasing the logit for the wrong token after logit subtraction. As the contrastiveness of each augmentation depends on the visual information required by each question, selecting an appropriate augmentation is crucial. To address this, we propose an automatic augmentation selection strategy that identifies the optimal contrastive augmentation for each question.
3.2 Maximizing Contrast: Selecting Visual Augmentation with the Largest Distance
The remaining challenge is to select a suitable visual augmentation that yields the most contrastive outputs relative to inference on the original image. In this section, we outline the indicators for effective rastrvvvive augmentation. Intuitively, applying CD should lead (1) an increase in logit for the correct token and (2) a decrease for the incorrect token. Based the intuition, we hypothesize that the augmentation resulting in the most different output distribution can serve as contrastive augmentation. To measure the distribution difference (--) between the output distributions from original and augmented images, we use Lg-norm.
D(v, O) = (SFFTAX((((( -OTTAX((()/()))))
Note that we mainly use Lz-norm hereinafter, and report analysis of various metrics in Appendix A.
To assess the effectiveness of CD, we use the logit increase of the ground truth token achieved through CD as a key indicator. This measure, referred to as Gain, is calculated as follows:
Gain(v, q, 0) P(yCT Ov)). p(UTT Ooriginat (v) g))
(3)
5

---

Published in Transactions on Machine Learning Research (09/2025)
where yeT is the ground truth token and Ooriginal is a transformation that returns the original image. To verify our hypothesis - bigger distance D()) con have the most contrustiveness D-) on the MME benchmark using augmentations from A across 5 seed runs. As shown in Figure 4a, as D(-) increases, the Gain gets bigger, most resulting in positive Gain values, which supports our hypothesis. Thus, choosing the augmentation with the largest D(-) can be an effective strategy for maximixing the Gain. To test. this approach, we select the augmentation that has the biggest. D--) on each augmentation and average their Gain scores and percentage of positive Gain scores. As shown in Figure 4b, we confirm that the augmentation with the greatest (--) yields the highest average increase in the Gain score. Notably, this increase surpasses that of any single augmentation, validating the effectiveness of a D(.) augmentation selection strategy. This finding supports using the augmentation with the largest D(-) as the contrastive augmentation o for each query 4-
measure the Gain and
Algorithm VSCoDe: Visual-Augmented Contrastive Decoding
Input: Image and question pair (v, q); target sequence length T, Augmentation set A, The number of of Visual augmentations N, Distance function D(.), Amplification coefficient for CD, plausibility constraint.
parameter for t = I...I do
if t = 1 then
> Determine contrast angmentation for the entire decoding process 2: < [velv, 4, veee) and ž,; + (e)))o0 Yo E
D Generate logits
Pt < SOFTMAX(2) 2)) and pt.s f- SOFTMAX(2,.) 24,),, Vo A
> Compute probability
- arg max,c.... (pt.....))
& Select the most constrastive augmentation
else
2t < f(ule, 4, vet) and Hn, f(v ((y.)Os(r); g, Pt - SOFTMAX (z+) and Pr.o <- (F..)
Generate logits
Compute probability
end if
PusCoDe, = (1 + a.) Pt a Pr,o
Compute VSCoDe probability > Candidate Set
Veand (1<() + {ye E EV: pi(uelu 9, yet) 2 8 max (wllee, 9.34)))
Pvscoba, (1) = 0, if y & (yer)
Discard nol-candidate words
V1 (P
Sampling next word
end for
3.3 Proposed Algorithm
Based on the above observations, we propose VSCoDe lo automatically select an appropriate augmentation for each query by utilizing the D()). The entire procedure summarized in Algorithm 1. In the initial decoding phase with the given question, adaptively select contrastive augmentation by choosing the augmentation with the maximum D(-). This produces similar effect to applying CD for every token while significantly reducing the computational cost. The chosen augmentation Ô is then used for the remainder of the decoding process. Once the contrastive augmentation is determined, LVLM calculates the probability of tokens PvscaDe.t using Eq. (1). Subsequently, among the whole vocabulary V, the candidate vocabulary set Vcand E V is defined to select a more reliable token following the original CD algorithm (Li el al., 2023b). This process is repeated iteratively to generate the output words yt-
We use two scenarios to define candidate augmentations: All and Coresel. All uses all the augmentations in the candidate set A. However, due to the characteristics of data distributions, certain augmentations may not yield meaningful contrast effects and may be effectively replaced with alternative ones. In this case, excluding these augmentations may work to eliminate noisy augmentations. So we introduce the Coresel strategy that leverages validation to choose a subset of augmentations A' C A to use only the more effective augmentations.
3.4 Coreset Strategy
Removing noisy augmentations via acceptance threshold т. Using the D(-), we aim to seloct a VA that shows high- perforrrfeee improvement when applied to CD. However, there may exist cases where certain VAs may not be appropriate contrastive augmentation for a specific task. In this case, these VAs contribute
6

---

Published in Transactions on Machine Learning Research (09/2025)
less to performance improvement than other VAs average and can sometimes become noise that prevents other VAs from being chosen as contrast. To determine these, We introduce the acceptance threshold т, a simple baseline that eliminates the noise VAs. To determine the suitability of VAs for the target task, we utilize the LVLM's first token generation distance by VSCoDe for cach VA in the sample sub-dataset. Let G be the number of times that VA, selected as cont VA among a total of M VAs. For the N data samples and acceptance threshold т, candidate VAs with 6 < r M are treated as unsuitable for this task and removed. We used the t 0.5 as a baseline throughout the main experiments.
4 Experiments
4.1 Experimental Settings
Datasets and evaluation metrics. We conduct experiments on Visual Question Answering (VQA) tasks and captioning tasks. VQA task evaluates how well LVLM generates robust and correct answers to various questions, and the captioning lask measures how well the LVLM generates captions given the image. We use MME (Fu al., 2024), MMBench (Liu et al., 2024b), VQAv2 (Goyal et al., 2017), and POPE (Li et al., 2023c) benchmarks for the VQA task, and each datasct consists of imago-question pairs. For the captioning lask, evaluate MSCOCO (Lin et al., 2015).
MME is LVLM evaluation dataset with granular question categories, including 10 categories from the perception lasks and 4 from the cognition lasks. The labels consist of 'Yes' or 'No,' and performance is measured by MME score, which is derived from accuracy. In this paper, we evaluate the perception tasks as our method focuses on observation ability.
MMBench is dataset of image-question pairs from 20 categories to validate how skillfuuly LVLM performs on various vision-language tasks with given option labels. For evaluation, we incorporate CircularEval, which rotates the positions of the possible option labels in a circular manner.
AQA+2 is a dataset containing open-ended questions paired with images. This allows for proper evaluation of how expertly the model can utilize the given visual information rather than simply using the learned language priors of the decoder. We randomly select 30K samples from the VQAv2 evaluation dataset to validate our method.
POPE benchmark is proposed to evaluate the hallucination of LVLMs. It asked whether the given object exists in the image or not. The fake object is generated from three sampling strategies: random, popular, and adversarial. Images are from three sources: MSCOCO (Lin et al., 2015), A-OKVQA (Schwenk et al., 2022), and GQA (Hudson Manning, 2019), and we report the average performance across the sampling strategies on each image source.
MSCOCO dataset provides 328K images for varions tasks, such as object detection, or captioning. We use 500 random images from the validation set to generate image captions and evaluate with CHAIR (Rohrbach et al., 2018) metric, which is designed evaluate hallucination in image captioning models. It assesses how accurately generated captions reflect the visual content of an image without including irrelevant details. CHAIR_ (I,S} metrics cach assess alignment with the image and semantic plausibility.
For the reliability of the results, we report performance using the average of the results of 5 different seed runs for MME and MMBench, and a single run for VQAv2.
Models. We evaluate the performance of VSCoDe using state-of-the-art LVLMs, mainly on LLa VA-1.5 (Liu et al., 2023b). Specifically, we use pre-trained LLa with Vieuna (Chiang al., 2023) 7B and 13B as the language decoder. Additional experiments on various IVLMs are provided in Appendix B.
CD methods. We use augmentations in Figure set A. For the baseline, we use vanilla setting that do not use CD, and CD with each single augmentation. To evaluate the effectiveness of VSCoDe, we compare with VCD (Leng et al., 2023), ICD (Wang et al., 2024), CRG (Wan et al., 2024) and HALC (Chen al., 2024). Note that single noise addition augmentation is equivalent method the VCD. For VSCoDe, we
7

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: MME score on perception tasks using the LLaVA-1.5 7B. Q-Dep denotes that the method varies the contrast based on the given query, and Ext specifies whether an additional external model is employed. Select denotes that the choice of augmentations selected by Metric, instead of fixed single augmentation. Note that All uses set of seven augmentation types, including noise, but excluding text augmentation, while Coresel utilizes subset of four augmentations (Color, Edge, Crop, Flipp) selected by the acceptance threshold T. The best and second-bes performances are reported using bold and underline formatting, respectively.
Augmentation
Category
Method
Type Q-Dep Ext Select Metric Existence
Count
Position
Color
Posters
Celebrity
Scetie Landmack Artwork
OCR
Total
Vanilla
180.00
112.00
117.67
147.00
121.02
110.24
148.80
129.15
100.35
97.00
1272. 22, 28
Color
X 180.00
129.33
127.33
152.67
132.45
128.06
148.30
138.35
109.75
101.00
1347.24 21
Edge
176.00
133.00
126.67
141.00
133.61
133.35
150.
142.15
110.70
103.50
1350.668 08..066..6...
Single CD
Sharp
X
184.00
113.33
127.00
163.67
126.46
111.50
150.80
132.15
112.00
102.00
1323.6016.51
Crop
x
183.00
122.00
129.67
151.67
129.86
128.33
146.80
134.70
110.95
101.50
1338.50ga3. 50.
Erase
x
x
X
182.00
118.33
119.33
153.67
126.12
115.21
119.96
131.85
112.90
98.50
1310.89=
Flip
x
x
X
178.00
124.67
129.67
152.33
132.24
125.24
147.30
137.05
111.25
107.00
1344,75.s1.0
VCD
Noise
x
x
X
186.00
115.00
122.33
154.33
127 76
125.17
149.85
138.30
110.90
93.
1323.4 21227/0
ICD
Text
x
183.00
122.67
120.33
165.33
128. 16
112.82
150.65
130.90
113.10
98.
1325. 1325.47= 10.300
CHG
Erase
185.00
116.00
115.33
125.67
144.49
1332.88
158.40
135.95
101.80
1310.022-20.19
Random
/
Rand
182.00
125.67
127:33
153.00
130.82
123.94
148.90
134.80
110.80
102.50
1339.76ga2.19
/
Dist
179.00
137.33
129.67
146.67
133.95
131.88 88
147.40
142.00
112.50
108
1368.89.200
VSCaDe
Correet
/
Dist
179.00
138.33
129.67
150.67
132.31
130.88
145.00
139.75
110.25
108.50
13641.6
experiment on both all and coresel strategies. The augmentations used by coreset differs depending on the datasets. For example, the MME benchmark, the LLaVA-7B model utilizes 4 angmentations: color, edge, crop, and flip. The implementation details for each method can be found in Appendix E.
Implementation details. We choose a = 1.0 and 0.1 for the main experiment. Additionally, we use T= 1.0 and p = 1.0 for the sampling strategy, which employs the softmax distribution for the next token generation. Additionally, we conducted experiments under various decoding settings, and the corresponding ablation studies can be found in Appendix C.
4.2 Experiment Results
Result on each category. Table 2 presents the MME score of CD with various augmentations and different methods applied the MME dataset across different perception categories, using the VLa-A-1.5 7B model. The baseline performance with individual augmentations shows an overall increase in the total MME score. This suggests that augmentations likely help create contrast in the visual features for CD. For example, in questions involving the identification of celebrities or landmarks, humans can still answer accurately even if the color of the image is altered. However, LVLMs may struggle to recognize objects when color, implying that color information is one of the important feature to recognize the object. Still, each augmentation is effective within spocific categories, underscoring the need to choose suitable augmentations for optimal results.
Comparing with baselines, VSCoDe results the best performance. When using single augmentation for CD of LVLMs, it is challenging to gain distinguished performance across all types of questions. In contrast, VSCoDe, which selects the most suitable visual augmentation based on the given question, demonstrates significant performance improvements across various question categories compared to single augmentation approaches. Furthermore, it surpasses both VCD and ICD, emphasizing the critical role of selecting appropriate augmentations. Note that we exclude HALC from the baseline as it fails to generate proper output on VL- 1.5 7B model. In Appendix D, we report the performance of LA-1.5 13B and analysis on the selected augmentations on each categories by VSCoDe.
Result on performance and cost comparison Table 3 presents the evaluation scores for all datasets MMBench, VQAv2, POPE, BLEU, and CHAIR) used in the experiments on LLaVA-1.5 7B. Additionally, it includes the time and memory costs, which were measured during the captioning task. Although HALC records high performance in captioning task, it consumes big cost for using external model. In contrast,
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table Performance on the various benchmarks and computational costs for different methods for LLa VA-155 7B. (M), (A), and (G) denote the MSCOCO, A- A-OKVQA, and GQA datasets for POPE. Detailed descriptions of each experiment are provided in Appendix E.
A Augmentation
Score
Cost
Method!
Type Q-Dep Ext Select Metrie
BLEU
CHAIIS
CHAIR,
POPE (M)
POPE (A)
POPE (G)
MME
MMBenchh
VQAw2
Time
Mem
Vanilla
40.13
8.00
5.47
53.63
80.41
80.85
1272.22
71.46
06.48
1.00
1.00
VCD
Noise
K
x
47.98
6.40
4.24
53.40
81.41
81.53
1323.44
72.03
89.19
1.98
1.00
ICD
5.38
53.72
81.23
81.12
1325.47
72.05
70.27
1.97
HALC
Segnent
16.87
20.40
6.75
11114.1
64:27
1.45
TALC!
Segnent
/
x
50.24
5.80
4.31
85.00
7.02
1.44
CRG
Fruse
x
42.70
5:20
3.37
85.28
82.27
82.25
1310.02
13.07
70.25
2.64
1.19
Random
Ail
x
x
Rand
19.02
5.75
3.91
83.87
81.80
81.81
1339.76
72.49
09.41
1.96
1.00
All
x
/
Dist
61.37
4,80
3.21
84.4
83.06
82.75
1968.89
74.32
71.02
2.45
1.04
VSCoDe
/
Coreael
x
Dist
51.97
4.60
3.10
84.63
82.75
82.84
1364.36
74.27
70.94
2.15
1.03
VSCoDe shows best performance with slight increase on the cost. Detailed setting for measuring the cost is described in Appendix E.
In the process of applying VSCoDe, our model requires inference as the number of VAs used in the first step only, and each subsequent generation step requires twice token generation stages, which is the same with other CD methods. Table 3 illustrates the cost on MSCOCO captioning task. They denoted cost compared to the vanilla baseline, which does not utilize CD. In the case of HATC and CRG, since they use an external object detector, the amount of memory used and the time taken in the decoding process are longer. In particular, in the case of HALC, the basic reproduction process takes a long time because of generating very long interpretations. Although optimized HALC generates short answers, it took times more time than the original due the complexity of the method itsolf. For CRG, we include the time cost. for object detection while use provided external data. VSCoDe does not use an external model or dataset, and especially using coreset, the time and memory cost is similar to that of single CD methods.
respective
While HALC and CRG show high performance on some tasks, the time and memory cost of using an external model is expensive. In addition, the performance of the two mothods can be affected depending on which object detector is used. In the case of CRG, the MSCOCO dataset was used in the pre-training process of DLONndingDINO-BB the object detector, resulting in high performance on POPE of MSCOCO dataset.
Results on various tasks. As shown in Figure 5, VSCoDe demonstrates superior performance across both VQA and captioning lasks. The figure shows that VSCoDe consistently outperforms other CD methods across diverse VQA task categorics and answer for- mats, including benchmarks such as MME, MMBench, VQAv2, and POPE. Furthermore, it generates signif- icantly fewer hallucinated answers, multiple performance metrics on the captioning task and the POPE benchmark. In particular, higher BLEU scores indicate that the generated captions more accu- rately describe the given image, while lower CHAIR scores demonstrate a reduction in hallucinated content.
Vanilla Random
VCD
ICD
CRG
VSCoDe (All)
VSCoDe (Coreset)
MME
Captioning (CHAIR_I)
MMBench
evidenced by
Captioning (CHAIR_S)
VQAv2
8.0
F6.5
40.1
6.0
Captioning (BLEU)
01.3
POPE (MSCOCO)
4.3 Discussion
POPE (GQA)
POPE (A-OKVQA)
A Analysis of the validity of CD selection by dis- tance. Table shows that choosing appropriate visual augmentation for each question is essential to maximize contrastiveness for an effective CD. In this section, we
Figure 5: Performance of the LLaVA-7B model on various VQA and captioning tasks. In most tasks, VSCoDe shows the best performance.
analyze the effectiveness of using distance D(.) for augmentation selection. Figure 6 shows the proportion of

---

Published in Transactions on Machine Learning Research (09/2025)
Noise
Color
Crop
Edge
O Flip
Erase
Sharp
60
Chosen Percenlage
20-
existence
count
position
color
posters
celebrity
scene
landmark
artwork
OCR
Category
Figure G: The percentage of each augmentation being selected based the distance D() within each calegory MME benchmark.
(a) LL 7B
(b) InstructBLIP 7B
1370
t-0.00
1260
t-0.02
0.30
1=0.38
1365
1250
1.30
MME Score
r=1.21
1360
11:14
1240
1230
LLaVA-155 78
InstructSUI 78
-266
Best MME
1-277
Best MME
1350
1220
2
Number of Coreset Augmentations
Number Careset Augmentations
Figure 7: Ablation study on the effect of acceptance threshold J in the coreset strategy across LLA 7B and InstructBLIP 7B. Each point on the curves indicates the threshold that determines the number of augmentations selected into VSCoDe, corresponding to the value shown on the r-axis. The F yiclding the highest performance denoted as Best MME.
single augmentations selected by VSCoDe for each category of MME when distance D(-) is used. Accordingly, D(-) can effectively select visual augmentations that significantly increase contrast for each category. For example, the 'color' category has a high proportion of color augmentation, which distorts the image's color, and the 'position' category has a considerable proportion of flip augmentation, which distorts the position information of the given image. Furthermore, D(-) gocs beyond simply using single augmentation for queries of specific calegory. According to Table 2, for many categories, VSCoDe outperforms each augmentation. This means that distance improves effective augmentation selecting for CD single category, beyond simply using single high-contrassivennss augmentation. More detailed analysis can be found at Appendix D.
Effect of coreset strategy. As shown Table and Figure 5, although coreset uses less number of augmentations than all, it performs better in many cases. This indicates that our coreset strategy, which removes noisy augmentations, is effective and underscores the importance of selecting only the most impactful augmentations for better results. This method not only enhances both efficiency and effectiveness but also serves useful tool for choosing augmentations from multiple options. Table 4 presents the selection ratio of augmentations by VSCoDe across dilferent datasets. According to the strategy, when 7 augmenlation are used in the set, each must have a selection ratio of at least 7.14% to be included. This demonstrates that the most impactful augmentation types vary depending on the data. distribution. For instance, the captioning task, which requires accurate image description, augmentations like the crop- which removes the background regions of image have higher impact it, while the effects of noise and color relatively diminished.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: Selection percentage (%) of VSCoDe across datasets and augmentation types. Can- didates excluded by the coreset strategy threshold 7 are highlighted in gray.
Table 5: Comparison of augmentation selection on the first token and full tokens the captioning task. Dist (F) denotes selecting CD augmentation based on the first token, whereas Dist (E) re-sclects CD augmentation at every token.
Augmentation Types
Augmeatttion
Score
Cost
Dataset
Nosin
Color
Grop
Edige
Flip
Fisse:
Storrp
Matlod
Typw
Q-Dap
Seles
Metric
BLEU
CHAIR,
CHAIR,
Time
Mem
MME
5,42
17.00
16.00
40.16
15.72
4,36
Vanilla
40.33lg1.1
8.00
5.47g
1.00
1.00
VQAr2
MMBBench
5.04
7.38
13.84
18.77
16:45
36.27
21.51
12.16
5.82
5.22
1,09
Random
Rand
49.02a1.x
5.75asus
1,66
1,00
POPE(MSCOCO)
POPE AOKVQA)
8.03
15.43
15.03
17:40
38.63
14.07
15.03
5.63
5.43
1.74
95050
Connard
All
V
/
V
Dist (P)
Diat (P)
21.97
31.37aus
4.80
4.00gg
3.2
3:10gm an
x2.45
x2.15
x1.04
x1.08
POPE(GQA)
3,00
15.00
18.27
02.28
13.77
5.93
1.80
Diet (R)
51.75gm
4.20ga.cx
2.83ga is
x3.31
x1.11
Captioning
1.90
6.80
28.20
43.00
14.00
6.00
0.40
VS0sDe-F
Conoect
Dist (8)
4.36an
2.832
x1.37
Table 6: MME performance of VSCoDe with different candidate combinations. We evaluate the performance of the candidate set A by excluding each candidate one by one.
Augmentation
Category
Method
Type
QeDep Select
Metric Existemcc
Count
Position
Color
Posters
Culebeity
Scree
Lamdmark
Artwork
OCR
Tosal
Vanilla
180.00
112:00
117.67
147.00
121.02 00
110.21
148 80
129.15
109.35
97.00
1327. 55
Color
x
182.00
134.00
129.33
160.00
142.86
142.24
154.60
143.40
112.60
113.50
1414.53
Single CD
Crop
K
187.00
110.33
138.33
147.87
149.80
146.65
156.70
146.65
105.75
103.50
1392.30
Flip
X
X
189.00
122.00
129.00
155-00
143.61
132.12
151.45
133.90
100.55
115 00
1374.82
A (rstenerrpp)
/
186.00
116.67
132.33
160.00
150:27
149.82
155.70
153.35
108.75
1420.90
VSCaDe
A Hosd
Dist
181.00
198.33
136.33
161.67
145.10
141.41
150.10
141.00
113.55
108 00
1416..00 50
Dist
184.00
116.00
133.33
150.67
148.57
147.94
155.55
151.50
107.80 .0
103 50
1398.86
Apston.rsp.popg
133.33
153.70
100.20
1426..2
Effect of acceptance threshold т on the coreset strategy. We conduct an ablation study on the acceptance threshold T to investigate its role in the Coreset. Strategy. As described in Section 3.4, a lower I allows a greater number of augmentations to participate in VSCoDe, which inereases computational cost while improving performance. In contrast, higher т filter out less effective augmentations and retains only those with more substantial contributions, thus reducing memory and time requirements. In particular, augmentations excluded at higher 7 values tend to provide larger gains when included in VSCoDe. In contrast, those admitted only at lower thresholds contribute marginal improvements or can even degrade performance. As shown in Figure 7, this trade-off is consistently observed in both LLaVA-1.5 7B and InstructBI.IP 7B. Interestingly, in the case of InstructBLIP, using the Coreset with an appropriate 7 can even outperform the total augmentation set, highlighting that efficiency and accuracy gains can be achieved simultancously. In general, these results demonstrate that carefully tuned within the Coreset Strategy not only improves computational efficiency but also provides robustness in performance in VSCoDe.
Selecting augmentation on the first token. VSCoDe mainly uses the first token to select the augmentation. Sclecting the augmentation on every token improves performance, but it comes from an additional cost. In this case, the memory cost is severe but can reduce time by running in batch-wise manner. Without batch-wise operation, the time cost increases linearly depending candidates. We name this strategy that CD on every token as VSCoDe-E and report the result in Table 5 on captioning task using the MSCOCO benchmark. While it needs about 130% more time and memory, it shows about 115% more performance gain in CHAIR metrics compared to VSCoDe. Using contrastive augmentation on each token may be more helpful to mitigate hallucination.
the number of augmentation
Analysis of the combination of visual augmentations. We evaluate different combinations of aug- mentations to estimate the impact of cach augmentation. For simplicity, we limit the augmentation set to (color, flip, random random crop] for this ion. Table 6 shows the effect of using all augmentation candidates in the set and the impact of excluding each one individually. According to the results, VSCoDe performance using all three augmentations, color, crop and flip, shows higher performance than other sub-combinations. Specifically, when color or flip augmentation removed from the augmentation set, performance in the color or position categories significantly decreases. Considering each augmentation has a different contrastive effect,
11

---

Published in Transactions on Machine Learning Research (09/2025)
Which corner doesn't t have any fruits?
Original
Noise
Color
Edge
Crop
lorig
fyrup
farig e
Lytp
top-left
0.35
0.39
0.31
0.37
0.43
0.19
0.34
top-right
0.30
0.27
0.20
0.19
0.28
0.08
0.56
bottom-left
0.28
0.32
0.40
0.34
0.23
bottom-right
0.07
0.07
2
0.003
0.08
D 0.026
0.10
0.015
0.06
D 0.009
0.06
0.223
0..03
what's
the relationship
these
arcatures?
Original
Noise
Color
Edge
Crop
Flip
Lorig
Erlpp
lorig
edge
symbiatic
0.66
0.74
0.93
0.95
0.89
0.83
0.14
predatory
0.32
0.24
0.058
0.04
0.11
0.16
0.85
competitive 0.01
0.01
0.00
0.00
0.00
0.00
0.00
parasitic 0.01
0.001
0.00
0.173
0.208
0.00
0.061
0.00
Figure 8: Two qualitative examples of VSCoDe on MMBench benchmark. On both cases, VSCoDe successfully corrects the wrong prediction from the original image by selecting the augmentation with biggest distance D(-).
the results confirm that providing an appropriate combination of VAs for A to cover all types of queries can provide proper contrast for a given task.
Qualitative study. In this section, discuss examples of applying VSCoDe MMBench with LLAVA-131 illustrated in Figure 8. The first example shows that the original image yields incorrect prediction, identifying empty space as 'top-left'. To adjust the logit of the ground truth token through CD, the augmented image must have a low logit for the ground truth token. Only the flipped image introduces positional contrast, resulting in a low logit for the token and enabling a successful CD. On the second example, the incorrect answer already holds high probability, indicating that strong contrastiveness is needed correct the prediction. Although all augmentations increase the logit of the wrong answer token, adding noise to the image produces only minor difference, which keeps the prediction incorrect. In this situation, maximizing contrastiveness is essential to achieve the correct answer. Therefore, selecting the augmentation with bigger D(-) is effective in reaching the correct answer. The CD results from unselected augmentations can be found in Appendix F.
5 Related Works
Contrastive decoding. CD (Li et al., 2023b) was introduced in the NLP domain using two differently sized language models. It leverages contrastive output by subtracting the small model's probability from the larger model'ss to retain the strengths of the large model while eliminating the weaknesses that are evident in the small model. There are variants like DOLA (Chuang et al., 2023) which utilizes contrast in layer-level outputs and Instructive Decoding (Kim et al.) uses two contrastive instructions generate an output opposite to the original output.
Recently, similar approaches have been applied in LVLMs, utilizing contrastive images to guide the model in generating accurate text, mainly focusing reducing hallucination in LVLM (Li et al., 2023c; Liu et al., 2023a; Tong et al., 2024). VCD (Leng et al., 2023) demonstrates that adding noise to the image can elevate the hallucination inherent in LVLMs, subsequently applying CD to manage the hallucination. Another work CRG (Wan et al., 2024) employs a black bounding box from external data to conceal the object relevant the question, amplifying hallucination, while HALC (Chen et al., 2024) uses multiple different cropped images from the detection model and explores multiple pairs of cropped images to find pairs that amplify the information in the cropped image. These works address methods to manage hallucination in LVLMs
12

---

Published in Transactions on Machine Learning Research (09/2025)
using single type of augmentation, which has limitations in generating enough contrast for various types of questions. Unlike previous studies, VSCoDe explores multiple augmentations and selects the most effective one to answer the question. Moreover, it does not require additional training or an external model, providing direct perturbation to the image.
Visual augmentation. In the computer vision domain, visual angmentation has been employed to increase the diversity of sample data, thereby helping to overcome the challenges associated with acquiring large training datasets and mitigating overfitting issues in environments with limited samples. Traditional augmentations include changes in color, cropping, and flipping. Additionally, there are more advanced tochniques such as erasing (Kunar Singh Jae Lee, 2017; DeVries & Taylor, 2017; Zhong al., 2020), and other techniques such as mixup (Zhang al., 2017) and CutMix (Yun et al., 2019). Furthermore, the automatic application of multiple augmentations has been explored (Cubuk al., 2019; Lim al., 2019).
Some studies in LVLMs employ VA to achieve the desired output in various methods. FGVP (Yang al., 2024) adds blur to the background of the image, leaving the main object clear to emphasize it. To focus on each object in the image, (Chen et al., 2023; Surís et al., 2023; Lin et al., 2024) use multiple cropped images, each focusing on a single object to generate the desired output, while (Kim et al., 2023) uses impainting to erase objects to measure the correlation between objects.
6 Conclusion
this paper, we present VSCoDe, a method that leverages multiple augmentations by adaptively selecting the most contrastive augmentation for effective CD. Our initial analysis revealed that the level of contrast generated by augmentations varies according to the query, so choosing an appropriate augmentation critical for enhancing CD. Building this insight, we developed VSCoDe, algorithm that selects augmentations based the largest distance D(-) metric. Experimental results demonstrate that VSCoDe surpasses existing methods across various tasks, highlighting the importance of using appropriate augmentations.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Limitation. Our method selects the appropriate contrastive augmentation among augmentation candidates. No matter how well VSCoDe works and the appropriate augmentation is chosen for the given task, if there is no sufficient contrastive augmentation for the task among the candidates, it is difficult to expect a significant performance gain.
Future Work. For future work, exploring methods to determine task-specific augmentation candidates could be a valuable direction. Additionally, identifying subset of highly influential augmentations or leveraging augmentation sets proposed in other works could be a promising direction. Furthermore, investigating the impact of text augmentations in conjunction with visual augmentations and integrating the two approaches could further improve the results.
Acknowledgement
This research was supported by the Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSTT) (No. 2022-0-00871, Development of AT Autonomy and Knowlodge Enhancement for AI Agent Collaboration, 50%) and the Ministry of Trade Industry & Energy (MOTIE, Korea), under the project titled " "An Unbreachable Multilayer AT-based Security Mechanism that Contimuously Adapts and Evolves in Dynamic Conditions" (Project Number: RS-2025- 02653102, 45%) and Institute for Information & communications Tochnology Planning & Evaluation(IITP) grant funded by the Korea (epermmen((MSST]) (RS-2019-[1190075, Artificial Intelligence Graduate School Program(KA 5%).
14

---

Published in Transactions on Machine Learning Research (09/2025)
References
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403. 200e24030002,, 2024.
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for wewledde-basee visual reasoning. arXiv preprint 23111 05226, 2023.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:/ ptttps:/1mmys..oo org/blggg 9028-03-30-vicuna/
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprinl
2023.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 113 123, 2019.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-langune models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. Xivi preprint Xrv: 04550, 2017.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Yash Goyal, Tejas Khot, Douglas Summors-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp- 6325 6334. IEEE Computer Society, 2017. doi: 1109/CVPR. 2017. .70.
URL https: /t////ii.org/10////
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial--sreength natural language processing in python. 2020. doi: 10.5281/2endd..221 1212303.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. URL bttps://arxxr wttp:///.wrrraa///....... 09506.
Jac Myung Kim, A Koepke, Cordelia Schmid, and Zeynop Akata. Exposing and mitigating spurious correlations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 2584 2594, 2023.
Tachyeon Kim, Joonkee Kim, Gihun Lee, and Se Se-Yo Yun. Instructive decoding: Instruction-tumed large language models are self-refiner from noisy instructions.
Krishna Kumar Singh and Yong Jac Lec. Hide-and-scek: Forcing a network to be meticulous for weakly- supervised object and action localization. In Proceedings of the TEEE international conference on compuler vision, pp. 3524-3533 2017.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311. 16922, 2023.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp- 12888 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730 19742. PMLR, 2023a.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. Contrastive decoding: Open-ended text generation optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 12286 12312, Toronto, IaaiiisssVoooii
Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1 acl-lon....nnn...... long. 687. wrtp:/23..r/02222... acl-long.600 687.
URL
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint orXiv:2355.03555 2023c.
Sungbin Lim, Modoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Aduances in
Neural Information Processing Systems, 32, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
URL attps:///arae http:///aaaivvvvo/////////e////e 14tp://145.00000 0312.
Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 36, 2024.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference Learning Representations, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Haotian Liu, Chunyuan Li, Qingvang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint ar arXiv:2303.0544 2023c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (ods.), Proceedings Brussels,
of the 2018 Conference on Empérical Methods in Natural Language Processing, pp. Belgium, Octoben-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18-111
URL
Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan Hc, Alex Mei, Yujic Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings. Xiv preprint or arXiv:2305.02311 2023.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022. URL 2206.01718.
org/otgg//ooo abs/
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. arXiv preprint XXiu:2403.16999 2024.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprinl orXim:293300811 08128, 2023.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXim:2401.06900 2024.
David Wan, Jaemin Cho, Elias Stengel-Fskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-languaae models without training. arXiv preprint arXiv:2403. 02325, 2024.
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision- language models with instruction contrastive docoding. arXiv preprint arXiv:2403. 18715, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-offthought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 -44877, 2022.
Lingleng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. Advances in Newral Information Processing Systems, 36, 2024.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fino-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint ar Xiv:22304. 14178, 2023.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEFE international conference computer vision, pp. 6023 133-6032,21 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint vrXiv:1710.00441 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001- 2020.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. Xiv preprint ar arXiv:2304.1055 2023.
17

---

Published in Transactions on Machine Learning Research (09/2025)
-Supplementary Material-
VSCoDe: Visual-Augmentatie Selection for Contrastive Decoding
A
Ablation on Distance metric D(-)
this section, we examine comprehensive several additional ablation experiments that are considerable in the environment in which VSCoDe applicd. Based on these ablation results, we expect VSCoDe to have universally high robustness and be able to perform various tasks, models, and inferences.
We perform experiments using several common distance measures to define function D(-) that VSCoDe uses lo select which VA will produce high contrast. The experiment is performed in the MME dalaset using the LLaVA-1.5 13B model. Also, we use the average softmax Gain directly to check the effect. In detail, Gain on the correct answer label obtained when applying the distance measure candidate D; used in the experiment and the VAs used Figure 2 to VSCoDe for all samples. order to control the variables of VAs that contain randomness, each experiment performs a total of experiments with different seeds on the entire MME dataset and then measures Gain through the average.
Softmax Gain
0.04
0.02
L1
L2
13
Cos
KL
EM
Figure 9: Average softmax gain by different distance metrics.
Figure 9 shows the result of affectness of different D(-) metrics. In this experiment, we L1, L2 L3, Loos Cosine similarity, Kullback-Leibler Divergence (KT. divergence), and Farth Mover's distance (EM distance) candidates. The x-axis of the results in Figure 9 means the candidate distance names used, and the y-axis means the average softmax gain improved compared to regular decoding obtained through VSCoDe when each distance is used as a measurement. From the results, we can check that L1, T2, and Lı norms show high performance improvement almost no diffcrence overall. This mcans that any of these can be used in the algorithm as distance function at a similar level. However, in the case of Los and KL divergence, it can be seen that the actual performance improvement much smaller compared to others. These show very low-performance improvement compared to the L2 distance, which used in the main experiment, meaning they are improper measurements for estimating the expected contrast of VAs. The other two distances, cosine similarity, and EM distance, performed higher than KL divergence but did not perform higher than Tz norm for the entire MME dataset. Based on this result, we empirically confirmed that using Le norm as our main D(-) for VSCoDe is a meaningful standard through experiments with these distance measures and the results shown throughout our main experiments.
B Analysis of Different Model Types and Sizes
We showed that VSCoDe proper for general LVLMs and has a significant effect on performance by experi- menting with five different models LLaVA-1.5, LLa VA-OneVisin, InstructBLIP, Qwen-VL, and Qwen2-VI on various types of datasets at the Section In this ablation, we conduct an experiment using LLaVA-1.5 7B, 13B and BnStructBI.IP 7B, 13B to check the effect of the model size VSCoDe. MME dataset is used for
18

---

Published in Transactions on Machine Learning Research (09/2025)
Table MME performance by different models.
Augmentation
Model
Method
Type
Q-Dep
Select
Metric
L.aVA-1.5
LLaVA-Qmeeisle 7B
InstructBLIP
Qwen
Qwen2- VL
Vanilla
1272.22
1441.59
1155.26
1355.32
1481.82
Color
8
x
1347.24
1528.28
1224.26
1422.69
1513.15
Edge
8
x
1350.68
1547.04
1221.15
1393.32
1552. 14
Sharp
8
X
1323.60
1496.46
1177.84
1395. 14
1516.84
Single
Crop
x
X
1338.50
1517.61
1194.13
1396.83
1512.10
Eruse
x
X
1310.89
1509.56
1195.27
1385.33
1501.28
Plip
x
X
1344.75
1530.90
1222.34
1425.20
1541.49
VCD
Noise
x
X
1323.44
1531.79
1218.90
1406.15
1519.61
All
/
Dist
1368.89
1579.34
1406.05
1547.07
VSCoDe
Coreset
Dist
1574.25
1254.16
1426.48
Table 8: MME performance by different model sizes.
Augmentation
Model
Method
Type
Q-Dep
Select
Metric
LLa15-1 13B
InstructBLIP 13B
Vanilla
1327.53
1151.45
Color
x
X
1414.53
1237.7 71
Edge
1424.20
1220.63
Sharp
x
X
1361.74
1164.32
Single
Crop
X
X
1384.65
1205.55
Erase
1365.91
1185.32
Flip
X
X
1374.62
1213.61
VCD
Noise
1354.34
1208.44
All
/
Dist
1443,06
1248.30
VSCoDe
Coreset
Dist
1443.14
1256.09
this experiment. We measured the performance for the perception category and the total performance for each model.
Table and Table 8 show the performance of VSCoDe each model and size for the MME dataset. From the result, when each VA is used in VSCoDe is robust to the type and size of the model and shows tendency to be dependent on the given task. Throughout the experimental results, the single VA edge and color show very high performance. On the other hand, we can see that single VA sharp and erase have slight performance gain than others. For different models, the performance gain shown by each VA shows an overall similar trend, and there is a higher performance improvement compared the original regular decoding.
confirm even if the model size and model used are different, the softmax gain obtained
Furthermore, for different model sizes, there is significant performance gain when applying our algorithm VSCoDe. VSCoDe using all of the VAs specified in Figure shows higher performance improvement than using each single VA. This indicates that, regardless of model and size, each application has the highest performance the entire perception category and total performance.
C Analysis of Different Sampling Strategies
We perform analysis studics on different sampling strategics to see how VSCoDe is affected by sampling methods other than basic regular decoding. In this experiment, 4 sampling techniques are applied: (1) Top P sampling (specifically, p = 0.9), (2) Top K sampling (specifically, & 50), (3) Temperature sampling (specifically, T 0.7/1.55) Top sampling a method which the only token candidates the distribution on cumulative probability can be selected as the next token. This has the effect of preventing noise samples with too low probability to be extracted from candidates. Top K sampling uses only the top & candidates from the
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table 9: MME performance by different sampling strategies for LLaVA-1.5 13B.
Augmentation
Sampling Strategy
Method
Type
Q-Dep
Select
Metric
Top P (p=0.9)
Top K (k=50, T-0.7)
Temp (T-0.7)
Temp (7-
Vanilla
1352.87
1399.33
1403.99
1169.71
Color
1405.90
1443.27
1445.19
1349.20 20
Edge
1434.14
1433.86
1420.64
1364.72
Sharp
X
1381 .00
1413.88
1416.63
1294.08
Single CT)
Crop
X
1391.01
1413.09
1422.15
1342,43
Erase
X
1374-47
1404.27
1399.67
1315.08
Flip
1404.76
1426.97
1425.54
1340.88
VCD
Noise
X
X
1370.47
1425.60
1429.52
1316.95
All
Dist
1462.67
1456.03
1454.32
1389.03
VSCaDe
Correct
Dist
1462.58
1457.10
1458.73
1377.47
highest probability for sampling. In temperature sampling, temperature scaling is applied to the softmax to calculate the next token logits. When temperature T is low, the possibility of selecting a high-probability candidate group increases, and the possibility choosing low-probability candidates decreases. has the effect of increasing the probability of more static responses. Conversely, when the temperature T large, the chance of choosing among the nigh-probability candidates decreases, and the low-probability candidates increases. It has the effect of increasing the possibility of making more diverse responses.
Table 9 show the experiment result of VSCoDe with different sampling strategies. From the table, that VSCoDe gives us high performance in various types of samplings. This is not only for regular decoding, but it also shows higher performance compared to single VA in the Top P sampling and Top K sampling. A notable observation that VSCoDe shows high performance in both cases where the temperature scale gets higher or lower. In the case of high temperature, the model has higher probability of generation more diverse, and the explanations and representations are getting richer. However, in this case, there is a potential problem that the entire output is inaccurate while in generation. In particular, if specific information for given image must be utilized rather than using inherent prior knowledge, there is a possibility that incorrect output may lose correlation with visual information on LVLMs. Our results show that using VSCoDe in this situation can be expected to have the effect of concentrating the model to intentionally utilize visual information by contrastive decoding the output through contrast VA. As can be seen from the results, in situation where the temperature scale large, CD through VA produces more siguificant performance gain. Additionally, the magnitude of contrastiveness produced by each VA different in the task so we considerable performance difference between single VA CDs. In this situation, VSCoDe, which automatically selects and applies the appropriate VA for given task, can be used more appropriately and robustly to the given scenario. Furthermore, it shows that VSCoDe has the highest performance improvement.
check
Our algorithm can also be used low now-temperature scale scenarios, which increases the sampling possibility of high-probability token being chosen as the next token. In this scenario, the original model's high logits become more extensive than usual by temperature scaling, increasing the probability of being selected the next token. When the correct answer logit does not have high value, the possibility of being chosen the next token is crucially dropped. For a low-temperature scale, once the model starts generation with incorrect token, it is more likely to continue generating incorrect responses. As mentioned in CD, in the case of high confidence in high logit sampling methods in a generation, a wrong token selection can siguificantly impact the quality of future responses. In this situation, using VSCoDe can increase the likelihood that low correct answer token will be selected as the correct answer through CD using contrast VA. As a result, it shows high robustness against the temperature sampling scale and increases the likelihood of providing an appropriate response.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Table 10: MME score on perception tasks using the LLaVA-1.5 13B. The best and second-best performances are reported using bold and underline formatting, respectively. As with LLaVA-1.5 7B, VSCoDe shows strong performance on the 13B model.
Augmentation
Category
Method
Type Q-Dop Ext Select Metric
Existence
Count
Position
Color
Posters
Celebrity
Scene
Landnark Artwork
OCH
Total
Vanilla
182.00
125.33
110.33
154.67
128.57
123.00
153.05
131.30
108.30
111.00
1:327
1966.
Color
X
182.00
134.00
129.5 33
160.00
142.86
142.24
154.00
143.40
112.60
113.50
(414.53ag..
Edge
x
X
X
185.00
146.00
125.00
157.67
111.70
1/2.21
152.95
139.50
113.15
121.00
14221.2 2011220:22.. 22.0
Single CD
Sharp
X
182.00
113:33
130.00
156.33
136.46
130.78
156.90
137.10
109.85
109.00
1331 170.3
Crop
183.00
127
124.33
150.00
143.13
139.24
153.05
141.70
108.70
114.50
13884. 665+24.7
Erase
185.00
126.67
116.33
144.67
147.55
128.29
156.60
132.85
110.95
117.00
13365.1 22.5
Flip
183.00
122.00
129.00
155.00
143.61
132.12
151.45
133.90
109.55
1 15.00
1337..22+.4.9
VCD
Noise
185.00
122.33
125.00
151.67
137.66
133.12
151.15
139.10
110.85
98.50
1354.334:24.
ICD
Text
182.00
124.00
127.33
159.00
134.63
126.47
156.00
137.00
110.45
103.00
13359.881.0
CRG
Erase
187.00
109.67
129.33
147.66
147.35
1/40.66
162.15
153.35
105.15
116.00
1398 233:2.46
Random
Ait
V
Rand
183.00
118.33
135.00
153.67
141.29
133.06
151.45
140.95
110.40
106.00
1373.15,aa.o
All
Dist:
184.00
138.65
67
134.00
167.00
146.80
144.29
149.35
145.30
114.65
119.00
1443.00g...
VSCoDo
Conract
/
/
Diat
183.00
140.33
132,00
165.33
146.46
1443.71
149.80
1445.05
114.45
123.00
14443.14g9
D Analysis of the Selected Augmentations
In this section, we discuss how often each augmentation is selected by VSCoDe based on the maximum distance D within each category of the MME benchmark using LLa 13B. The corresponding MME benchmark performance of using each augmentation and VSCoDe for CD is reported in Table 10, and the selection ratio of each augmentation by categories can be seen at Figure 6. In most categories, edge is predominantly selected, which aligns with its high performance, as shown in the table. Moreover, flip and color are mostly selected in the position and color calegories, respectively, which is intuilive since flip and color augmentations alter position- and color-related information. Noise and sharp augmentations, which show low performance in the table, are selected less frequently, as there are other more effective augmentations available.
E Experiment Details
E.1
Experimental Setting
Here, we provide detailed description about the setting and results of the experiments with an additional bascline and computational cost. We compare with VCD (Leng et al., 2023) ICD (Wang et al., 2024), HALC (Chen et al., 2024), and CRG (Wan et al., 2024) and follow the setting of original paper for the experiments. VCD (Teng et al., 2023) is a method that applies contrastive decoding to an image manipulated by adding diffuuson noise. It adds Gaussian diffusion noise to given image with total of 500 steps. ICD uses contrastive query that adds 'You are a confused object detector' to the head of the given query text.
HALC (Chen et al., 2024) is method that uses an additional object detector to contrastively decode an adaptive sample for a given query image called Adaptive Focal-Contrast Grounding, We used the official HALC code to reproduce results with the default setting using GroundingDINO as object detector (Liu et al., 2023c). It provides the codes for MME, MSCOCO captioning, and POPE COCO, which we used to reproduce the results. However, this process, although the reproduced performance of HALC was similar to the values reported in the HALC paper, the scores were significantly lower compared to our baselines. In addition, HALC uses unique evaluation metric called OPOPE, which was unavailable for conducting an accurate comparison.
Therefore, we conducted various types of prompt engineering to bring out the polential of the HALC method and achieve a similar level. However, HALC did not perform well in prompt instruction following. Using the default prompt texts we applied to other basclines or common prompts used in VLM resulted in poor performance and generated output sentences with unwanted lengthy interpretations. Consequently, we obtained captioning output using a prompt 'provide short. description for this photo without interpretations!
21

---

Published in Transactions on Machine Learning Research (09/2025)
which was almost the only prompt that could force the output to answer with short caption. The result of the performance obtained was recorded as HALC" at Table 3.
We provide CRG (Wan et al., 2024) for an additional baseline, which is method that uses an image with the corresponding region of interest (RoI) removed from the image given in the query for contrastive decoding. If no specific Rol given, CRG extracts noun phrases from the given query text and gets the Rol of extracted phrases by using the object detector. Following the original paper, we use spaCy (Honnibal et al,, 2020) for the noun phrase detector and obtained bounding boxes by using GrouddngDINO-B (Liu et al., 2023c) with threshold of 0.3. In this process, filtered noun phrases that do not represent objects ("the image", " "the type", "the photo", etc.) in the prompt of MME or POPE benchmark to prevent incorrect Rol extraction. For the MSCOCO captioning task, we use bounding boxes provided by the annotation of the dataset.
In this paper, all reports of our experiment used LVLM models that can run on a single 48 GB NVIDIA RTX A6000.
F More Case Studies
We present additional examples of MMBench using the LLaVA-1.5 13B model. The figures illustrate the probability distribution of outputs from original and augmented images, along with the probabilities after applying CD. The yellow bar represents the option with the highest probability for the original image, the green bar shows the top option for the augmented image with the maximum distance D(-), and the blue bar indicates the option with the highest. probability which is correctly aligned with the ground truth after applying CD.
Figure 10c and Figure 10d demonstrate that augmentations with maximum D(-) effectively manipulate the image to produce incorrect predictions. Note that due to hallucination, the model retains a non-negligible probability for the original prediction. CD mitigates this hallucination, enabling the model to generate correct outputs. Conversely, as shown in Figure 10c, Figure 10f, and Figure 10b, augmentations with maximum D amplify the incorrect predictions of the original image, facilitating successful CD. Figure 10g highlights that, although the augmented images yield high confidence for the same option as the original image, which corresponds to the ground truth option, CD leverages only reliable tokens with subsequent probabilities Voand, as explained in Section 3.3, to produce trustworthy answers. In this case, only option D considered as Voand since other options have low logit values. Finally, Figure 10a illustrates a failure case where the output distribution of the original image is uniformly spread across options, causing ambiguity in determining the direction of image manipulation that facilitates the incorrect output. Although the probability of correctly predicted output turning into an incorrect one is only 3,5%, this challenge remains in future research.
22

---

Published in Transactions on Machine Learning Research (09/2025)
How bananas there the image?
A.
C.
Original
Noise
Color
Edge
Crop
Flip
Grap
Sharp
0:30
0.00
18
0.75
0.27
0:28
0:07
DH0.225
0.55
DH0.055
0.00
D-0
0.33
0H0
020
DH00000
6.2
D-0.03
0.57
Dos
01
0.t5
0004
025
0.22
021
6:33
0.25
5.A4
07
0113
0.33
(a) Question: "How many bananas are there in the image?"
Roughly how picture occupied by the the picture?
B. less than 40%
D. 0.8
Original
Noice
Color
Edge
Crop
Fli
Crop
Sharp
16:02
0400044
02
D44 071
1062
040
02
0401222
10.01
240
19
040054
10.02
D40 Des
0.28
0.57
0.05
0.08
40
12033
1000
10
10
0.00
003
051
0.
06
0.79
0.89
0.28
0.66
0.00
C.0
0.00
0.00
0.00
D003
(b) Question: "Roughly how much the picture occupied by the cat?"
Which corner docan't t have fruits?
top-left
bottom-left
bottom-right
top-right
Original
Noise
Color 31
Edge
Crop
Flee
Crop
Sharp 0:28
0:88
0:07
0:07
06
008
0.6
0.77
0 06
0.28
C:44
0.22
0.34
0:08
0.32
0.27
(c) Question: "Which corner doesn't have any fruits?"
23

---

Published in Transactions on Machine Learning Research (09/202))
Which the main topic the image?
A. A toy bear and toy dog
toy beer and a toy chicken
C. bear and cat
Original
Noise
Color
Edge
Crop
File
Crop
Sharp
0.38
0.34
0.13
0.08
0082
0.23
10
14.23
Da3
BODr
1952
100
10.00
0.56
0.2
10.02
0.24
1943
0.17
0.00
5.00
0.12
10.00
10.00
(d) Question: "Which is the main topic of the image?"
In nature, what's the relstionship between these creatures?
A. Predatory relationships
Competitive relationships
C. Parasitic relationshiip
Original
Noise
Color
Edge
Crop
Flip
Grop
Sharp
01
0..00
0.00
10.00
0.00
0.01
0.01
0.01
0e0.015
0.00
De0 152
10.00
D=0.173
0.000
D=0.108
0.00
D=0.061
10.01
D=0.024
0.01
De0.008
0.88
0.74
0.94
35
0.89
0.83
0.54
0,70
80
0:86
68
0.57
0.24
0.38
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0 00
0.00
0.00
0.00
0.00
0.57
0.20
0.14
0.32
0.43
0 76
0 62
(e) Question: "In nature, what's the relationship between these two creatures?""
Where the sheep?
8. The sheep the front of the car
The sheep on the right the car
D. The sheep is on the left of the car
Original
Noise
Color
Crop
Fib
Crop
Sharp
0.00
0322
0:21
0.14
B0D4
023
D-O
D-O
1002
1004
JD02
D-0,07
CA3
033
022
0.07
0.2
6MM
0.13
031
50
0.29
032
0.7
0.15
0.2
0.10
D:03
(0:22
0.7
(L) Question: "Where the sheep?"
24

---

Published in Transactions on Machine Learning Research (09/2025)
Where the giraffe looated in the picture?
A. right
C. bottom
Original
Noise
Color
Edge
Crop
Flip
Crap
Sharp
D0
19.2
040 002
10.04
Do0 368
10
Do0 92a
00
0-0-403
80
D41,009
10:02
Oo0 005
1600
Do0
19.23
10.04
1:35
BB 0:07
(4:23
10
0.00
0.00
0.00
DV03
0100
010
0.00
0.00
0.20
0.00
0.00
0.00
0.00
0.00
(g) Question: "Where is the giraffe located in the picture?"
Figure 10: Additional case studies utilizing VSCoDe.
25


Reviews:
[{"review_id": "EVub8BkCkM", "reviewer": "Reviewer_f7iM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis work adaptively selects task-specific visual augmentations to improve the effectiveness of contrastive decoding. By applying simple image manipulations, it effectively corrects the output of large vision-language models without requiring additional model modifications.\nWeaknesses:\n1. Limited novelty: The proposed method appears to be a combination of existing techniques， which applies known visual augmentations within a contrastive decoding framework. This integration may limit the originality of the contribution.\n2. Potential Limitations of Augmentation: In some cases, visual augmentations for contrastive decoding seem to negatively impact model performance. For example, in Table 1, all augmentation strategies appear to degrade performance on OCR tasks. Similarly, in Table 2, performance on Scene tasks is worse than that of Vanilla decoding method.\n3. Randomness in augmentations: Some augmentations, such as cropping, may randomly retain the entire task-relevant objects, which is not necessarily beneficial for contrastive decoding. In addition, flipping can reinforce symmetrical location biases in position-sensitive tasks, which are not necessarily severe hallucinations in the original output.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an adaptive method for selecting augmented images for contrastive decoding(CD) across a range of vision-language tasks. By identifying and utilizing the most effective visual augmentations, the approach enhances the performance of large vision-language models(LVLMs) without additional training or reliance on external models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Apply augmentation strategies with certain randomness multiple times to enhance stability—for example, perform cropping or erasing several times and average the results. For flipping, you can also try using different rotation angles.(Suggestions for Improving the quality)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8hIJEgYdF7", "reviewer": "Reviewer_oq7g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. The author propose a Query-Dependent Augmentation Effect section, aiming in demonstrating that various augmentations affect LVLM outputs differently. The result in Table 1 effectively and quantitively shows the impact of various augmentation.\n\n2. The main method is Maximizing Contrast: Selecting Visual Augmentation with the Largest Distance which aims in selecting the visual augmentation that yields the most contrastive outputs relative to inference on the original image. The distance measurement is through measuring distribution difference between the output distributions from original and augmented images.\n\n\n\n## Weakness\n1. In demostrating various visual augmentations, would the combination of different methods yields a better performance? It seems like that author only focus on invidual augmentation.\n2. In exploring the distance measurement for the method design, the author select through measureing the output distribution. However, the more intuitive way is through the semantic distance (such as using the CLIP visual encoder to do the distance selection). This distance measurement should be considered.\n3. Based on the motivation experiments and main experiments, the most effective visual augmentation normally has a connection with the question. If so, directly using the question-augmentation matching such as using text encoder would directly select the most effective augmentation?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "language decoders sometimes produce incorrect outputs, a phenomenon called hallucination. \n1. The author aims in digging the impact of different visual augmentation to the visual perception process which is convincing.\n2. Based on the motivation/oberservation in line 1, the author propose a novel strategy in VSCoDe that selects contrastive augmentation to empower CD capability without additional training or using external models.\n3. The result shows the advancement of the strategy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Same as above", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "79gjKj6dxb", "reviewer": "Reviewer_PTRk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The authors propose a training-free approach to improve the quality of the generation of language models with visual capabilities.\n- The presentation of the work is curated and detailed, explaining the parts clearly.\n- The authors tested different models to confirm the validity of their approach.\n\nWeaknesses:\n- Unclear why the authors apply the distance measures directly to the normalized logits (i.e., after softmax). As also confirmed in the Appendix, doing so results in very little change (i.e., at most a difference of 0.04), regardless of the distance metric used (i.e., variation between 0.04 and 0.01, with L1, L2, and L3 performing identically). In addition, it seems like the authors ablated the distance measure on the MME benchmark, one of the benchmarks they use to evaluate their method.\n- Unclear why the authors report results mixed between llava-1.5-7b and 13b on the main paper (e.g., Tab 1 is on 7b, Fig 4 is unclear, Tab 2 is on 13b, and Tab 3 is on 7b again). Moreover, it is unclear why the authors present results only for llava 1.5 in the main paper and keep the results for instructblip and qwen in the Appendix. Last, all these model families have more recent versions, e.g., qwen2-vl, or llava-1.6 (also llava-onevision).\n- Unclear how the authors set the hparams for their method, i.e., (alpha, beta, T, p, and tau). Also, the work is missing an ablation on the tau hparams, to show how performance changes with different thresholds on the coreset strategy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors aim to improve the quality of the responses of Large Vision Language Models at test time with contrastive decoding. Their approach consists of selecting the most appropriate augmentations to apply to the input image to contrast the generation against its unmodified version, reducing the likelihood of decoding incorrect tokens and increasing the chance for the correct ones. For this, they propose the VSCoDe method, which identifies the most suitable augmentations for the problem via a distance metric and applies the augmentations to calibrate the generation, effectively improving the output quality, as demonstrated by extensive experiments on different benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should primarily focus on adding missing details regarding the weaknesses highlighted in the above box. Specifically:\n- Explain why they apply the distance measure on the normalized logits, potentially showing how performance changes when applied directly on raw outputs.\n- Confirm if they tuned the distance metric on the MME benchmark, potentially justifying their decision.\n- Explain why the main paper reports a mixture of results between llava-1.5-7b and 13b, potentially uniforming the results using a single version.\n- Explain why the authors did not test more recent models, e.g., qwen2-vl, or llava-onevision.\n- Explain how the authors set the hparams of their method.\n- Demonstrate with an ablation the performance of the coreset strategy with different tau.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)