CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models
Authors: Neeraj Anand
Samyak Jha
Udbhav Bamba
Rahul Rahaman
Venue: Accepted by TMLR ()

Abstract: Despite the rapid success of Large Vision-Language Models (LVLMs), a persistent challenge is their tendency to generate hallucinated content, undermining reliability in real-world use. Existing training-free methods address hallucinations but face two limitations: (i) they rely on narrow assumptions about hallucination sources, and (ii) their effectiveness declines toward the end of generation, where hallucinations are most likely to occur. A common strategy is to build hallucinated models by completely or partially removing visual tokens and contrasting them with the original model. Yet, this alone proves insufficient, since visual information still propagates into generated text. Building on this insight, we propose a novel hallucinated model that captures hallucination effects by selectively removing key text tokens. We further introduce Generalized Contrastive Decoding, which integrates multiple hallucinated models to represent diverse hallucination sources. Together, these ideas form CRoPS, a training-free hallucination mitigation framework that improves CHAIR scores by 20% and achieves consistent gains across six benchmarks and three LVLM families, outperforming state-of-the-art training-free methods.

Published in Transactions on Machine Learning Research (1/2026)
CRoPS: A Training-Free Hallucination Mitigation Frame- work for Vision-Language Models
Neeraj ; Samyak Jha', Udbhav Bamba', Rahul Rahaman Indian Institute of Technology (ISM), Dhanbad, India
Tronsmute Al
National University of Singapore
Reviewed on OpenRevicw:
Abstract
Despite the rapid success of Large Vision-Tanguage Models (LVT.ss), a persistent challenge their tendency to generate hallucinated content, undermining reliability in real-world use. Existing training-free methods address hallucinations but face two limitations: (i) they rely on narrow assumptions about hallucination sources, and (ii) their effectiveness declines toward the end of generation, where hallucinations are most likely to occur. A common strategy is to build hallucinated models by completely or partially removing visual tokens and contrasting them with the original model. Yet. this alone proves insufficient, since visual information still propagates into generated text. Building on this insight, we propose a novel hallucinated model that captures hallucination effects by selectively removing key text tokens. We further introduce Generalized Contrastive Decoding, which integrates multiple hallucinated models to represent diverse hallucination sources. Together, these ideas form CRoPS, training-free hallucination mitigation framework that improves CHAIR scores by 20% and achieves consistent gains across six benchmarks and three LVLM families, outperforming state-of-the-at training-fiee free methods.'
1 Introduction
Recent advances in large vision-language models (LVLMs) have demonstrated remarkable multi-modal ca- pabilities (Liu et al.) 2023c Dai et al. 2023 Their ability to comprehend both images and language has enabled a wide range of applications. However, similar to Large Language Models (LLMs), LVLMs are prone to "hallucinations" generating convincing responses that lack precision, which can lead to mislcading information (Rani et al. 2024 Li el. al.
2016). This limitation poses a significant, challenge to their reliability
trustworthy AT assistants in real-world applications (Wang et al. 2023b Liu et al. 2023a).
Recent works aiming to mitigate hallucinations can be categorized into two regimes: training- bosed methods
[Biten et al. 2022 Kim et al.] 2023 Rohrbach et al. 2018). and training freen methods (Wei et al. 2022 Li
ct al. 2023 Long et al. 2024 Wang et al.) 2024b Favero et al. 2024). In this study, we are inferested in a very specific group of training- free approaches that utilize a common framework of Contrastive Decoding (CD) (Li et al. 2023). CD, a training-free approach, proposes to negate the hallucinations from LLM outputs by contrasting (subtracting) outputs of heavily hallucinated models, essentially reducing the problem to identi- fying and using hallucinated models to contrast with. These methods often suffer from several shortcomings. Firstly, by design, the hallucinated models used for contrasting cease to perform well as the generation of output tokens (Figure 3). making them effective primarily in the early stages of single hallucinatod model does not sufficiently represent all the sources of hallucination, c.g., hallucination due lo uninformative visual tokens and hallucinations from the bias produced by the training data.
a
* CCorreppondnng Author: meersjanapndinsttqmmmll.com
1 Code is available at https///hhhcccco//mmmaaac/aaaaca www.a//owononotooooooaoaaaaaaeee

---

Published in Transactions on Machine Learning Research (1/2026)
In this work, we start with an analysis of the existing methods (Favero et al. 2024 Huo et al. 2024 and show, by means of both empirical analysis and theoretical computations, how existing methods struggle to remove hallucination as the LVLM generates more tokens. We then propose novel hallucinated model that overcomes this challenge by going beyond the removal full or partial visual tokens and considering textual inputs for removal. Our proposed model motivated by what our analysis shows, that the later stages of the generation, previously generated output tokens carry equal, if not more, importance than visual tokens.
Next, we empirically show how existing contrastive decoding methods fail to address different kinds of sources of hallucination. To mitigate this issue, generalize the concept of contrastive decoding and extend it to accommodate multiple models to contrast with rather than single model. Our proposed Generalized Contrustive Decoding, allows us to conjoin our newly proposed hallucinated model with existing works Huo et al. (002)), to devise our novel, training-free free hallucination mitigation method CRoPS (fig. ! We evaluate and compare our proposed CRoPS against competing methods in a wide range of generative and discriminative benchmarks. We also show that CRoPS outperforms all methods across three different choice of architectures.
Contribution: Below summarize our main contributions. In this work,
We provide detailed analysis of how existing methods perform poorly in the later stages of the LVLM token generation. This also enables us to devise a novel hallucinated model by moving beyond visual tokens to identify and remove textual inputs from prompt and past generated tokens.
We empirically show how the individual methods only tackle specific source of hallucination and fail to remove other sources. To this end, we formulate Generalized Contrastive Decoding, by extending the scope of contrastive decoding to allow multiple models to be contrasted with.
Finally combine our novel text-deficit hallucinated model with imagc-deficit hallucinated model under the novel generalized contrastive decoding framework and propose CRoPS. Our proposed method significantly outperforms baseline, and brings consistent improvement over competing meth- ods across a wide range of tasks, datascts, and LVLM architectures.
2
Related Works
With the progress in LLMs, recent studics have explored LVLMs by integrating visual encoders into pre- trained LLMs. These models have shown some advanced multi-modal capabilities. However, they suffer from hallucinations (Rohrbach et al.) 2018 Zhang et al. 2024 Guan et al. 2024 Wu et al.] 2024b), which restrict their real-world applications (Wang et al. 2022bb Liu et al.) 2023a). There are several causes of hallucinations, including lack of understanding of world knowledge, overfitting to specific training data patterns, and insufficient common sense reasoning. In LLMs, hallucinations typically occur when generated responses contradict real-world knowledge common sense. In contrast, for VLMs, the main concern is whether the generated response conflicts with the provided image.
Researchers have explored several methods to mitigate this issue, which can be broadly categorized into the following two groups:
Trasning-based Approaches. Training- based methods involve either retraining VLMs with curated datasets or using auxiliary models to supervise revise generations. These approaches include instruc- tion fine-tuning on hallueinationnaaware datasels (Lee et. al. 2022 Gunjal et. al. 2024 Zhao et al., 2024 Jiang et al.) 2024 Yu et al.] 2024 Yue et al. 2024] and post-hoc training of revisor networks that analyze the model outputs and correct hallcinations using auxiliary networks (Manakul et al. 2023 Zhou et al.
2024 Yin et al.) 2024 Chen et al. 2024 Wu et al. 2024a Feng et al.) 2024). While these techniques can be effective, they require extensive computational resources and careful dataset. design.
Training-free Approaches. Training-fre free approaches, in contrast, modify inference-time decoding attention patterns without additional training or supervision. They include decoding-time modifications
2

---

Published in Transactions on Machine Learning Research (1/2026)
Image (vision)
Vision tokens V
Original outputs
Pt
SSM LLMo (V. X, y-c)
Full vision Full text
Least important vision
Visually hallucinated
D
vis-hal
= dSM LLMg (V, N,Y,yay
Prompt Instruction (text)
Text tokens X, y<i)
Least important vision
Full text
Please describe the image
Visuo-textually hallucinated
Past Outputs (text)
Least important text X, y
P p.
== OSM o LLM X
The image depicts man
No vision
Least important text
log CRaPS = (1+a*++a +a! logp," oris " lo risbal
0/ log Vhotet-hal
Figure 1: Overview of CRoPS framework. CRoPS combines two hallucinated models: one that removes visual tokens to capture vision-related hallucinations, and another that removes key textual tokens to capture text-related hallucinations. Their outputs are then integrated through generalized contrastive decoding framework to roduce hallucinations in LVLMs.
(Huo et al.
2024
Wang et al. 2023b Favero et al. 2024 Li et al. 2023 Chuang et al. 2024 Liu et al.
2024b
Leng ct al.
2024 Wang et al. 2024b
2024b Zhu et al. 2024 Huang et al. 2024 Kim Kim et al.
Woo al. 2024 Zheng et al. 2024 el al. 2025 and atlention adjustment mechanisms (Tang et al. 2025 Yin et al. 2025). These methods are simple plug plug-and-ple and-play modules and avoid the computational
et al.: 2024
overhead of training- based methods.
Closely related to our work are Multi-Modal Mnformannoooo Decoding (M3ID) (aavero el al. 2024 and Self Introspective Decoding (SID) (uo et al. 2024). which also fall under the category of training-free free approaches. M3ID proposes to utilize and tweak the same LLM to come with a hallucinated model. They argue that if the image input is removed from the LLM, the model generates arbitrary text that is entirely unrelated the visual content, relying solely on biases learned during training. STD, on the other hand, selectively retains the least important vision tokens, identified those exhibiting low coherency with the query token, instead of removing all visual tokens from the LVLM input. The authors argue that these low-importance tokens contribute most to hallucinations. These methods form the foundation of our contrastive hallucination framework, which we explore further in Section (sce Appendix A and B for formal definitions).
Unlike prior methods, which independently address visual token dilution and biases introduced during pre- training, CRoPS effectively mitigates these issues without requiring fino-tuning or auxiliary models.
3 Background
In this section, we formalize LVLM inference and decoding because these details determine how visual information propagates into generated tokens (Section 3.1. We then describe Contrastive Decoding (CD)
framework for reducing hallucinations in LVLM (Section 3.2 and an attention- based token pruning method
that to construct. hallucinated models (Section 3.3

---

Published in Transactions on Machine Learning Research (1/2026)
3.1 Inference and Decoding in LVLMs.
LVLMs accept both visual and textual inpul to generate textual output. The model first. encodes the input image into sequence of vision tokens using a vision encoder and cross-modal projection module. We denote these visual tokens as V == (V1, ; where m is the total number of visual tokens. Additionally, the model receives a textual prompt X, which can also be represented as a sequence of n tokens The LLM, parameterized by 0, then generates textual output tokens sequentially. Let y == (31.922... ..,........................ 91-1) be the sequence of generated tokens till inference time step t. The inference process can be formally written the following way:
X [1, ,
Pt = softmax LLM (V,X, X, y<)
(1)
Yt Decode [P:]
Y ce++
[yct yo]
Initially, the language model outputs logits, denoted by LLMg(.), which are then converted into probability vector Pt by applying the softmax function. The produced is probability vector of shape [V, where V represents the vocabulary of the LLM. Next, a decoding function is applied to convert Pr into a single token yt- To perform this transformation one of the several available decoding strategies such as greedy, beam search and nucleus sampling can be utilized. The newly generated output token y: then appended to the past tokens provide input to the LLM for generating the next token.
3.2 Hallucination and Contrastive Decoding
An LVLM said to hallucinate when the generated sequence Y fully or partially inconsistent with the visual input V. As discussed in section [ hallucinations. In this work, focus on particular set of methods that share common framework called Contrastive Decoding (CD) (Li et al. 2023 The CD framework tries to remove ballucinations from LVLM, by suppressing the probabilities assigned to hallucinated tokens / objects. It does so by first creating identifying a hallucinated model and then subtracting the logits of the hallucinated model from the original LVLM logits, thus reducing the probabilities assigned to the hallucinated tokens. Formally, under the CD framework, the final probability outputs are computed by,
several training-free approaches have been proposed to mitigate
log (1 + a)) logp,"" a log hal
(2)
where 0, Pt orig is the probability outputs from the original LVLM with complete input computed in equation [ and phal are the probability outputs from the hallucinated model. As CD relies on defining a hallucinated model, the choice of such a model remains crucial.
3.3 Attention-based Token Pruning
At inference time t, our input to the LVLM is the tuple (V, X, Xyee) of visual tokens V, text tokens X, and the LLM generated outputs so far Yet. To estimate loken importance, we utilize the attention weights produced by a selected transformer layer 1 within the language model. Let K E {V, X,,,) denote the set of keys and y, the current query token. Let K E (V, X, yer) denote the set of key tokens and y, the current query token. For a multi-head attention mechanism with H heads, we define the importance score db for each key token the mean attention weight across all heads
H Σ
(
Attention! (I.)) (K, y:),
(3)
H
where Attention (I.A) (L,))(K, strongly the query token y: attends to the keys in K.
(K, yt) denotes the attention distribution from the 2h head of layer 1, measuring how
Based on (()), we identify the tokens with the lowest importance values, those least attended by the current query. Using this score, we choose the bottom ū(< |K]) tokens with lowest i scores and
4

---

Published in Transactions on Machine Learning Research (1/2026)
Dependency Measuro
1.0
VD(I) VTD()
# Hallacinated Objects
Sampling
M3ID SID
0.6
CRoPS
0.2
60
100
Person
Cup
Bottle
Generation Time
Figure 2: Left: Plot of dependency measure (see section 4.1 which quantifies the influence of vision and vision+text tokens LVLM generation. We observe that VD(() decreases over time, indicating that the model relies less on vision tokens as decoding progresses. Right: Frequency of hallucinated objects that frequently co-occur with the ground truth object "dining table". We observe that SID and CRoPS effectively mitigate statistical biascs, whereas M3ID performs sub-optimally.
remove high-importance tokens, to create new sparser set of tokens. This design is motivated by prior findings from SID (Huo al. 2024), which demonstrate that low-importance tokens are more likely induce hallucinations.
4
Motivation
In this section, we highlight the limitations of existing contrastive decoding based approaches for hallucination mitigation, which motivate the design of our proposed framework.
4.1 Drawback Diminishing Dependency on Visual Tokens
To support the following discussion, we first define a metric that captures how LLM generation behavior depends on the input as function of generation time t. Specifically, we extend the Prompt Dependency Meosure proposed in Favero et al. 2024 to define two new metrics: Visual Dependency and Visuotertual Dependency as given in equation and equation respectively. The Visual Dependency is defined as,
VD(t)= == dist (softmax o LLMo (V,,,veeee softmax LLMg (X, y<t))
where dist(P.Q)= = v2 VP-vQl represents the hellinger distance. VD(t) measures the change in gener- ated output token distribution if the visual input tokens omitted. The left subfigure of Figure 2 plots VD(t) as function of generation step t. It is quite evident that VD(t) decreases drastically as gener- ate more tokens, i.e., the distribution shift when we ignore the image is less during the later stage of the generation.
This diminishing dependency has important implications for contrastive decoding methods, as it directly affects the effectiveness of contrastive signals used during later stages of generation. Since Favero et al. 2024 defines their hallucinated model phal = LLMs X yer), a diminishing VD(t) points increasing similarity between pirig, and phul over time (derived in Appendix E). As a result, the contrastive signal becomes less mcaningful in the later stages of generation, rendering the hallucinated model less effective (as also qualitatively illustrated in Figure where later-stage generations exhibit stronger hallucinations). Similarly, the masking of image tokens in the method by Huo al. 2024 becomes less impactful when VD(I) itself is low.
4.2 Remedy A Novel Model to Contrast with
As the diminishing VD(t) indicates increasing redundancy of visual tokens, later-stage generation primarily depends on textual context. Thus, hallucinated models that partially or fully discard visual tokens lose effectiveness in the later stages. To alleviate this issue, propose removing important textual tokens.
5

---

Published in Transactions on Machine Learning Research (1/2026)
Prompt: Please describe this image in dctail
Vanillar- image showcases
snowy landscupe mountains, grass
a valley
scene featuring
a lake nestled within the valley. Snow-covered mounteins rise dramatically into the sky, with the water below providing picturesque contrast to the snow- covered termin. There surruunded by the lake. The snowy lundsaupe creates an ldylli,, natural serting perfect for hiking, skiing, or simply enjoying the majestic beauty of the winter
the image,
have
while others covered
appear
SID: The scene features covers both hill picturesque
hill overlooking body of water, possibly lake. In the background, there a mountain or mountain range providing several
background, with some in the foreground as well. Some these trees are partially hidlen in the snow, In the sky, cloud can be seen, adding to the wintry atmosphere. Additionally, horse is present near the edge of the scene. enhancing the picturesque nature of the location.
ground, creating
simosphere
.........
picturesque hillside overlooking by forest filled with pine trees, creating sense of immquility. The mountains and gruss ground with tree branches add to the beautiful natural scenery. A cow can be seen grazing near the rocky field in the foreground, sharing the handscape with the cow nature remains alive during the snowy winter season.
The image
landseape,
body
The snow-covered hillside surrounded
CRoPS:-
The image features seenic handccppe with mountains surronnding large body of water, likely lake river. The mountains strotc the seen
adding the beauty landucape. sky mostly clear with soft streaks clouds. The snow on the ground creates serene atmooshhere, perfect for winter activities such as skiing or snowbourding. The presence of mountains and water creates sense of tranqiility and adventare in this winter wonderland.
backeround, creating picturesque scene. Trees
throughout
Figure 3: Comparison of image descriptions from different methods. Vanilla, SID, and M3ID contain hal- lucinated details (highlighted in red), e.g., animals and exaggerated snow coverage. In contrast, CRoPS produces a faithful description without these hallucinations. Note that hallucinations become more frequent during later stages of generation.
where the importance score computed from the altention weights at early layer of the model, similar to the strategy adopted by SID (Huo et al. 2024) for visual tokens.
H Σ
1
(v)) =
H Attention (1.M) ([X, ([x,y=tl. vt)
(5)
Using these importance scores, we only retain the least important text (prompt and past tokens combined) tokens and convert the tuple (X, yet) into,
X == LeastTmp (x, (x, m(..))
(6)
by keeping n(,,t) t) text tokens where n(,tt = 30 + 81 (1 (1 - ed) is a non-decreasing function with respect to t. Since the number of text tokens increases with time (past generated tokens), n(,, t) needs to be a non-decreasing function to maintain sparsity of the retained incoherent tokens. Discussed in detail in section
Our proposed hallucinated model then takes the form,
pt vis-txc-hal = softmax LLM X, yet)
(7)
This means we not only completely remove visual tokens, but we also remove important textual tokens. To check if the proposed hallucinated model is free from the problem of diminishing dependency, we compute Visuotertual Dependency VTD(tt as,
VTD(t) := dist (softmax o LLM, (V, X, ye)), softmax o LLMg X, y<())
(8)
and compare against VD(t) in the left subfigure Figure 2 is evident that unlike VD(t), VTD(t) does not diminish as time passes. Mcaning, the proposed hallucinated model significantly differs from the original outputs and hence contrasting is effective even in the later stages of generation.

---

Published in Transactions on Machine Learning Research (1/2026)
Jensen- SaannSSannon Divergence
0.75
sampilngg
M3ID
CRoPS
0.5
VD(t)
0.875
0.25
JSD(p pidtsso
ISDIM3ID, SID)
log(2)
0.05
0.0
Generation step (t)
Generation step (t)
Figure 4: Left: Plot of Ceasen-Shannon (JS) divergence over generation time between different hal- lucinated models. The dashed blue line indicates log(2), which is the maximum possible divergence value. Right: Plot of Visual Dependency of final outputs across different methods (Sampling, SID, M3ID, and CRoPS).
4.3 Drawback II: Contrasting with A Single Hallucinated Model is Insufficient
Contrastive decoding methods lypically rely on hallucinated models that either ignore perturb parts of the imput to expose hallucinations. These models designed to capture different failure modes such over-reliance past generated tokens attention to irrelevant visual features. Although such methods show promising results, addressing only one of these sources of hallucination is insufficient.
For instance, M3ID removes the visual imput entirely, thereby targeting hallucinations arising from excessive dependency on text tokens or language priors. However, fails mitigate hallucinations triggered by misleading or spurious visual cues. In contrast, SID builds hallucinated model using low-importance visual tokens and contrasts it with the original model to suppress hallucinations caused by irrelevant visual cues. However, it remains ineffective against ballucinations that emerge later in generation when textual dependency dominates.
As discussed in section 4.1 and depicted by the decreasing Visual Dependency VD(() in Figure 2 (left.), the influence of visual tokens diminishes over time, reducing the impact of methods that modify only visual inputs. Furthermore, Figure 2 (right) shows that M3ID continues to suffer from hallucinations associated with co-occurring object pairs (e.g., "person", "cup", "bottle") alongside the ground-truth object "dining table", which stem from training-time correlations. These observations together indicate that contrasting with single hallucinated model, whether vision- or text-oriented, is insufficient to comprehensively address hallucinations across all stages of generation.
4.4 Remedy II: A Generalized form of Contrastive Decoding
The standard ivee decoding formulation in equation Ponly allows the original outputs to be contrasted with one hallucinated model. However, as argued in section 4.3 a single hallucinated model may not be sufficient to capture different source of hallucination, and hence the resulting model after contrast will still suffer from unaddressed modes of hallucinations.
We propose a generalized form of contrastive dccoding with multiple hallucinated models to contrast with. Our proposed generalized form takes the following form,
R
R Σ (r)) log P.
log p.¹ = (1+ Σ (r) log Pr
orig
hal(r)
(9)
where ar > Vr. The generalized formulation empowers us to take advantage of multiple hallucinatod models, representing different sources of hallucinations.
Limitation Assumption. The participating hallucinated models p. justify their inclusion. Adding similar hallucinated models under the generalized form will achieve the same result the traditional contrastive decoding but with possibly higher latency.
hal(r) should be distinct enough
7

---

Published in Transactions on Machine Learning Research (1/2026)
5 CRoPS
Our proposal, CRoPS, integrates two hallucinated models under the generalized contrastive decoding for- mulation with the models being
vis-hal softmax LLM V X, y,yc), Pt
(10)
Pi le-txxt-bll = softmax LLM X yct).
The first model P: -. is inspired by Huo et al. 2024 which removes important visual tokens (Section 3.3 to capture co-ocourrence--elltee hallucinations (right subfigure of Figure 2 but relains all test tokens. The second model p. pfit-ttt-hal (as discussed in section 4.2) is deprived of all visual information and important textual information. During the early stages of generation, when generation heavily dependent on visual tokens, Pt pis-hal represents the source of hallucination whereas in the later stages when visual dependency diminishes, pr pyle-txt-bal provides the hallucinated outputs by depriving the model of important text tokens. When combined, the resultant contrastive decoding, which we term as CRoPS can be written as equation g with the components defined as below,
log log PROPS = (1+ (2) + +ff log logp, pr
(11)
(2) log pk vio-hal (?) log vis-txt-hal
where (1) = a and aj := ;= (1- (1- Since vision-driven hallucinations occur predominantly in the early stages of generation, (1) is kept constant. In contrast, aa²") is designed to gradually increase over time, reflecting the growing influence of text-induced hallucinations in the later stages. Similar to prior works, we add confidence and plausibility constraint while contrasting via equation II The overall algorithm is outlined in the Appendix.
Distinctiveness of pł pyis-hal and p; pi The gencralized contrastive decoding with multiple hallucinated models only justified by how distinctive the participating models are. Left subplot of Figure Jensen- Shannon Divergence (JSD) between proposed p; piis-bal and p. piss-ttt-hal It is compared with the maximum (log 2), and the corresponding JSD between hallucinated models used by previous works, indicating that simply stacking the hallucinated models from previous works under generalized contrastive is sub-optimal.
Visual Dependency of Final Outputs. The right subplot of Figure the VD(t) of the final outputs of competing methods and p is clear that CRoPS is more visually grounded (i.e., less hallucinated) than vanilla sampling and other methods.
6 Experiments
6.1
Experimental Settings
Models and Baselines. We conduct evaluations on three widely adopted LVLMs: LLaVA-1.5 (Lu et al. 2023b), LLaVA-NeXT (Liu et al. 2024a), and Qwen2-VI. (Wang et al. 20244). For bascline comparisons, wo consider several recent training free hallucination mitigation techniques. These include VCD Leng et al., 2024), ICD (Wang et al. 2024b), OPERA (Huang et al.] 2024). ClearSight {Yin et al. 2025 SID (Huo) et al. 2024 and M3ID (Favero et al. 2024). These methods represent state-of-the-art strategies that aim to reduce hallucinations without requiring additional fine-tuning or retraining of the underlying vision-language models.
Implementation Details. We set the pruning layer to / = 2 for both text and visual tokens. We follow (Hu et al.) 2024) choice, as they demonstrate that attention scores in the initial layers better reflect token importance and are relatively unaffected by attention sinks. Our experiments utilized the 7B and 13B backbones VA-155, 7B backbone of Qwen2-VL, and 8B backbone of VeAXA- We applied nucleus sampling with top-p value of 0.9 and temperature of All experiments were conducted using three
8

---

Published in Transactions on Machine Learning Research (1/2026)
Table 1: Evaluation vision language grounding on the MS-COCO validation set Lin et al. 2014). Cap- tions are generated using the prompt "Please describe this image in detail". CHAIR scores Cs and C represent the percentages of hallucinated objects and captions, respectively, where lower values indicate stronger visual grounding. Recall denotes the percentage of annotated objects correctly mentioned in the generated captions. All results are averaged over three random seeds.
VA---- (7B)
LLaVA-155 (13B)
LLaVA-NeXT
Qwen2-VI VL
Method
Cs 4
C₁ 4
Recall
Cs 4
C5
Recall
Cs 4
C₁
Recall
Cs 4
C₁
Recall
Sampling
57.0
17.0
75.0
50.2
13.7
76.4
37.4
8.9
66.3
33.2
8.0
68.1
ClearSight
54.1
16.2
74.3
49.4
13.9
74.8
35
8.5
63.6
13.5
8.7
38.2
VCD
53.3
15.3
77.9
49.5
13.7
77.7
36.4
8.8
68.6
29.0
7.8
67.3
ICD
52.5
14.6
77.7
49.2
13.9
78.1
36.6
9.4
67.1
28.0
7.9
66.1
OPERA
49.1
13.8
78.5
48.2
13.2
78.9
35.5
8.9
66.9
31.0
8.1
67.9
SID
48.9
13.0
77.9
47.0
12.3
77.9
37.0
10.7
70.2
30.6
8.1
66.1
M3ID
47.1
12.8
74.8
45.5
12.2
75.3
36.1
9.8
68.7
28.8
7.3
64.8
CRoPS
39.5
10.2
76.3
38.5
75.1
33.2
66.2
26.9
67.4
Table 2: Performance on the AMBER benchmark (Wang el al. 2023a.. evaluated with the prompt "Please describe this image in "etail". We report three axes of hallucination: HAL (overall hallucination rate), Cog (cognitive deviation from correct attributes (raittioesrrlatioonn) and CHAIR (object-lovel hallucination). where lower
is better. Results are averaged over three random seeds.
LL- (7B)
LLaVA-1.5 (13B)
LLaVA-NeXT
Qwen2-VL VL
Mothod
CHAIR 4
Hal
Cog
CHAIR
Hal 1 Cog
CHAIR L
Hal
Cog
CHAIR
Hal 4.
Cog
Sampling
10.6
44.3
4.0
9.3
41.3
4.2
10.5
57.1
6.4
38.9
2.8
CleanSight
10.5
44.1
4.0
9.1
54.4
2.2
VCD
9.0
42.9
38.3
3.9
9.5
55.7
4.2
5.7
33.9
2.4
ICD
10.0
44.8
4.3
8.5
39.5
4.1
10.1
53.0
4.5
6.0
34.1
2.5
OPERA
9.8
43.0
4.5
8.7
40.0
4.1
9.8
51.0
4.3
6.3
35.0
2.6
SID
9.3
43.7
3.7
6.9
35.0
3.5
9.1
54.2
3.9
5.4
30.6
1.8
M3ID
9.0
40.0
40.0
2.9
8.7
51.9
5.5
27.9
1.5
CRoPS
6.3
29.3
2.8
5.7
27.8
2.5
7.2
44.6
2.6
5.1
24.2
1.1
random seeds, and the average performance across these runs is reported. The hyperparameter configurations and their ablation analysis are provided in Section
6.2 Experimental Results
In this section, evaluate CRoPS across range of benchmarks that capture different forms of hallucina- tion and multimodal reasoning. For each benchmark, we briefly describe what it measures and then discuss the corresponding results.
CHAIR Benchmark. The CHAIR (Captioning Hallucination Assessment with Image Relevance) bench- mark RRohbbahh et al. 2018) evaluates object-level hallucinations by comparing nouns generated captions with ground-truth object annotations, where lower Cs and C₁ indicate stronger visual grounding. As shown in Table [ CRoPS achieves the lowest CHAIR scores across all LVLMs, outperforming prior training- free methods by clear margin. On the LLaVA-1.5 series, CRoPS reduces hallucination rates by roughly 15 25% relative to M3ID, while maintaining comparable recall. The trend continues for LLa VA-NeXT and Qwen2-VL, where CRoPS yiclds an additional 8-10% reduction in Cs and C₁ without any degradation in descriptive quality. Overall, these consistent percentage gains across architectures highlight that CRoPS effectively mitigates hallucinations without relying on retraining or auxiliary supervision. Although Clear- Sight attains a lower Cs on Qwen2-VL, this improvement comes at the expense of recall (38.2 vs. 67.4), suggesting over-suppression visual details. In contrast, CRoPS preserves balanced trade-off between grounding accuracy and linguistic richness, produeing visually faithful yet detailed image descriptions.

---

Published in Transactions on Machine Learning Research (1/2026)
LLaVA-1.5
LLaVA-Next
Qwen2-VL
SHR
SHR
SHR
WP
WP
0.30
054
50.51 875
46.453
42.903
31,200
SPI
SPI
SPI
5253
1-gras
1-gram
1-gram
Z-gram
Z-gram
Z-gram
Sampling
SID
M3ID
CRoPS
Figure 5: Evaluation on the GPT-4o assisted benchmark (Zhou et al. 2024 comparing hallucination (SHR), fluency (1- and 2-gram precision), and descriptive detail (WPI and SPI). Larger enclosed areas correspond to better overall performance. Please zoom in for clearer visualization.
Table 3: Evaluation results the POPE VQA hallucination benchmark Li et al. 2023 POPE comprises three subsets: Random, Popular, and Adversarial Each sample follows the query template: "Is object present in the image?", where object is selected either randomly (Random), from the most frequent dalaset objects (Popular), or from objects that frequently co-occur with the target entity (Adversarial). We report the average performance across all three subsets; detailed results are provided in the Appendix G.
VA-155
LLaVA-NeXT
Qwen2- VL
Method
Acc.
F1
Acc.
F1
Acc. f
F1
Sampling
81.7
82.8
87.3
87.6
84.2
82.8
ClearSight.
82.0
83.0
87.3
87.7
73.5
74.5
VCD
82.6
83.3
87.7
88.1
84.7
83.2
ICD
82.2
82.9
86.6
87.4
85.2
84.3
OPERA
82.3
82.9
87.4
88.2
85.5
84.5
SID
82.7
83.3
88.6
88.4
85.6
83.9
M3ID
82.4
83.4
88.0
88.1
84.9
83.5
CRoPS
83.9
84.6
89.4
89.4
86.1
85.3
AMBER Benchmark. This a multi-dinennional hallucination benchmark that evaluates not just whether objects are wrongly included (via CHAIR), but also how the model misrepresents attributes relations (HAT)) and deviates in cognitive consistency (Cog) in generated captions (Wang et. al. 2023a ).
Here, HAL captures the overall hallucination frequency, and Cog measures the divergence between de- scribed attributes or relations and ground truth. As shown in Table 2 CRoPS consistently outperforms all baselines across models. On the LLaVA-1.5 series, CRoPS achieves about a 25-30% reduction in both CHAIR and HAL, and 10-15% reduction in Cog, demonstrating more accurate and semantically coherent descriptions. For LLa VA-NeTT and Qwen2-VL, it maintains similar improvements, reducing hal- lucination metrics by roughly 20-25% while preserving fluency and descriptive richness. These consistent percentage reductions across architectures highlight that CRoPS mitigates both perceptual and cognitive hallucinations more effectively than prior training-free methods.
GPT-4o Assisted Benchmark. The GPT-4o assisted benchmark ZZhou el al. 2024) uses Visual Genome annotations as ground truth and prompts GPT-4o to evaluate hallucination at a finer granularity. It measures the Sentenco-Level Hallucination Ratio (SHR), indicating how often generated sentences contain hallucinatod content, and also evaluates fluency (1-gram and 2-gram precision) and descriptiveness (Words Per Image, WPI). As shown in Figure CRoPS achieves the lowest. SHR across all model variants, demonstrating its
10

---

Published in Transactions on Machine Learning Research (1/2026)
Table 4: Results on the General-Purpose benchmark showing MME and MathVista scores for LLa VA-1.5, LLaVA-NeXT, and Qwen2-VI. Higher values indicate better performance. Bold numbers mark the best result and underlined numbers mark the second-best within each column. CRoPS achieves the top scores across most settings.
LLaVA-1.5
LIaVA-NeXT
Qwen2-VL VL
Method
MME 1 MathVista
MME
MathVista
MME t MathVista
Sampling
1601
27.4
1669
34.8
2058
56.9
ClearSight
1569
27.1
1602
35.1
2001
54.7
VCD
1622
26.5
1630
36.2
2070
55.1
ICD
1605
25.9
1664
35.0
2060
54.2
OPERA
1615
27.0
1650
2080
54.8
SID
1634
26.3
1607
36.8
2097
54.3
M3ID
1607
26.9
1683
36.6
2090
52.0
CRoPS
1662
28.9
1779
2184
55.6
strength in suppressing relational and attribute-level hallucinations. Moreover, CRoPS maintains superior 1- and 2-gram fluency and higher WPI, indicating fluent and detailed captions. Unlike other baselines that shorten output reduce hallucinations, CRoPS preserves text richness while improving factual grounding. POPE. The POPE benchmark (Li et al.) 2023) evaluates hallucination detection in binary visual ques- tion answering format, where the model must confirm deny the presence of an object in the image. As shown in Table 3 CRoPS achieves the highest Accuracy and Fl across all LVLMs, showing an average performance gain of about 2% over the strongest baseline. This consistent improvement demonstrates that CRoPS remains robust even under restricted yes/no answer formats, maintaining strong visual grounding across models.
While M3ID performs competitively, it slightly lags behind SID and CRoPS in this binary setting. This dif- ference arises because POPE differs from captioning benchmarks, it requires only concise yes/no responses, which limit the degree of visual token dilution typically observed in open-ended generation. However, discussed in M3ID (Favero et al.) 2024), non-ncgligible visual token dilution effect persists due to the structural separation between image tokens and output tokens introduced by the VQA prompt template. To account for this offset, we follow the same configuration and select the decoding position t to, where to corresponds to the number of tokens between the image and the answer span. This adjustment ensures that CRoPS aligns visual grounding signals with the output region, resulting in consistently higher precision and balanced grounding across all models. Detailed results are provided in Appendix G.
MME and MathVista Evaluations. The MME benchmark (Liang et al. 2024 evaluates general multi- modal perception and recognition. Table | shows that CRoPS improves MME scores across architectures, indicating stronger grounding in diverse perceptual tasks. MathVista (Tuu et al. 2024) probes visual math- ematical reasoning; CRoPS achieves top or near-top performance across models, suggesting its decoding strategy benefits structured multimodal reasoning beyond captioning. These results highlight that CRoPS's contrastive design enhances both descriptive fidelity and logical reasoning across diverse LVLM tasks.
7 Analysis
Latency Comparison. Table 5 shows inference time and peak GPU memory on LLaVA-1.5 7B and the CHAIR bench- mark. CRoPS adds minimal overhead despite an extra for- ward pass due to its lightweight input design. Performance gains stem from the targeted contrastive design rather than repeated model calls. Compared to 5-beam search, CRoPS achieves lower hallucination with only modest increase in time and memory. This analysis uses a vanilla implementa- tion; an optimized version would further reduce overhead.
Table 5: Efficiency Comparison NVIDIA A100. Time is measured in seconds and memory usage MB.
Method
Time
Memory
Cs
Sampling
215
15699
57
Beam Search (5 beams)
531
16737
50.7
VCD
550
17864
53.3
16574
OPERA
1947
21943
49.1
CRoPS
652
16934
39.46
11

---

Published in Transactions on Machine Learning Research (1/2026)
Table 6: Left: Effect of image and text removal on the hallucinated model. Right: Comparison of token retention policies on CHAIR metrics and Recall.
Image Removal
Text Removal
Cs 4
Cr 1
Recall
Policy
Cs 4
Cr +
Recall
Full
No
45.4
12.4
73.3
Constant
43.6
10.9
75.0
No
Partial (Least Important)
47.8
13.2
73.8
All-but-one
46.8
13.1
73.6
Full
Partial (Least Important)
43.2
10.8
73.3
Linear
41.2
10.7
74.6
Full
Partial (Random)
50.7
13.8
76.4
Ours
39.5
10.2
76.3
Effect of Image and Text Removal. Left sub-lable of Table p analyzes the impact of image and text removal on hallucination behavior. Row removes all image tokens while keeping text tokens. Row retains all image tokens but removes the Icast important text tokens. Row 3 is our proposed sctting, removing all image tokens while partially preserving text tokens. Row 4 removes the same number of text tokens but at random. Our method outperforms others, confirming that selective removal of the least important text tokens is more effective (see section 4.2
Choice of Hallucination Model. hallucination mitigation strategy, we replace the second hal- lucinated model with M3ID while keeping CRoPS unchanged. Table 7 shows this yields inferior results, underscoring the ef- feetiveness of our proposed second model design.
To test contrastive
Table 7: Ablation on CHAIR metric on MS- COCO dalaset on the choice of hallucina- tion model.
Method
Cs +
Cj +
Recall f
M3ID + SID
45.1
11.8
78.3
Token Retention Policy. We control the number of retained tokens at each generation step & using a policy n)(). The base-
CRoPS
39.5
10.2
76.3
lines include: Constant (n1()= = 80), which keeps a fixed number 8o tokens; All-but-one (n(() = 1 2),
which discards all tokens except the most recent one; and Linear (n(t) = fo + Bıt), which ensures a steady linear increase in the number of retained tokens t grows. To enable a smooth yet saturating increase, we introduce an exponential policy defined as 7(,,() = ßo + B1 e-*),, where >> 0 controls the rate of growth. The policy starts at Bo and asymptotically approaches 3o + 31, thereby preventing abrupt context drops and yielding empirically more stable generation. As shown in right sub-table of Table G our exponential policy achieves the lowest hallucination scores, outperforming all baseline strategies.
Hyperparameter Ablation. We analyze the sensi- tivity of CRoPS to its key hyperparameters in Eq. []] The coefficients (1) and af2) control the overall strength and the time-dependent growth of the contrastive penal- ties, respectively. The parameters fo, 31, and j gov- ern the dynamic weighting function 77, which determines how many text tokens are masked at each step. Here, Bo and 31 define the lower and upper bounds of the retained-token range, while i controls how smoothly " increases over time. As shown in Table S moderate con- trastive strength and gradual masking yield the best re- sults, whereas extreme values of a or μ degrade performance by either weakening or over-suppressing the contrastive signal.
Table 8: Effect of varying hyper-parameter on CRoPS performance.
a
?
bo
b
μ
Cs
1.0
0.02
10
30
1 10-3
39.5
0.5
0.05
5
20
1 10 10-3
42.3
1.0
0.02
5
40
1x 10-3
41.2
1.5
0.01
10
30
1x 10-2
43.5
1.0
0.02
10
40
1x 10-3
40.7
For all experiments, we adopt single shared configuration of hyperparameters: 1.0, 0.02, 8g 10, 6 30, and μ = 1e-3. This configuration is used consistently across all backbones and all benchmarks, and no model-specific or dataset-specific tuning is employed.
8 Conclusion and Limitations
In this work, we first highlighted how existing mitigation techniques, which address visual token dilution and attention to irrelevant visual features individually, fail to comprehensively tackle both. To bridge this
12

---

Published in Transactions on Machine Learning Research (1/2026)
gap, introduced CRoPS, a novel decoding strategy that effectively mitigates both sources of hallucination while incurring minimal additional computational overhead, and demonstrates significant gains on multiple benchmarks.
Limitations. While our study provides valuable insights into the weaknesses of contrastive decoding, it is not exhaustive. We specifically focus on certain types of hallucinations, leaving the exploration to future work. Additionally, our analysis is restricted to the latest SOTA methods and only considers training-fres free contrastive decoding approaches. Exploring alternative frameworks, including those that involve task-specific training, could provide more complete understanding of the trade-offs in contrastive decoding.
Our approach also introduces some additional inference latency. Although CRoPS is about 3x slower than vanilla sampling (652s vs. 215s), its runtime remains comparable to or lower than existing training-free methods such as VCD (550s), SID (510s) and OPERA (1947s). The overhead comes from an extra forward pass, but this pass is relatively lightweight since the text-deficit model processes only small subset of tokens. Latency remains& a practical limitation. Future work can reduce this overhead, for example by parallelizing the forward passes required for generating the hallucinated models.
References
Ali Furkan Biten, Lluis Gómez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reduc- ing object hallucination in image captioning. In Proceedings of the IEEE/CVP Winter Conference on Applications of Computer Vision, pp. 1381-1390, 2022.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint ar arXiv:2403, 00.00444304 2024.
Yung-Sun Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httpps://pperevviie http://ppaareeeeeeeeeeeoorraaaaa httt/foorum?ddThblllll
Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language mod- els with instruction tuming. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 49250 49257. Cur- ran Associates, Inc., 2023. URL httpss:/proceeddiii wwt..ie.cc..ppppprrrr.eeeeeeee wttpc/pppeeepeeeeepeeeeeeeeeeeee /files//appe//223/Eiee hacc:328777108834444772333332212
neurips http://wwww.aaaaaaaaaaaaaaaaaaaa
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual informa- tion grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14303 14312, 2024.
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't hallucinate, abstain: Identifying Ilm knowledge gaps via multi-lim collaboration, 2024. URL https:
//arxiv. serxiv.org/aaee wrg//b//////////////
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-languag models. In 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. doi: 10. WCCVR52233.20222222
2024.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18135 18143, 2024.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang. Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust
13

---

Published in Transactions on Machine Learning Research (1/2026)
penalty and tetrospectio--alloattin. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognation, pp. 33418-13422 2024.
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-nnrroppcctive decoding: Alleviating hallucinations for large vision-language models. arXiv preprint 0Xxu:2408.00032. 2024.
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognilion, pp- 27036 27036-27046, 2024.
Jac Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious corre- lations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1585-2598 2023.
Junho Kim, Hyunjun Kim, KIM YEONJU, and Yong Man Ro. CODE: Contrasting self-generated description to combat hallucination in large multi-modal models. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024a. URL htttes///oppooooeeeeeee/rroorrrd
Taehyeon Kim, Joonkee Kim, Gihun Lee, and Se- Young Yun. Instructive decoding: Instruction--uunel large language models are self-refiner from noisy instructions, 2024b. URL fttps://arxiv.org/// 2311.
00233
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-73, 73, 2017. doi: 10.1007 016-0-9--6---- URL https://doi.orr/1...o 1007/
611263-016-0981-
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catan- zaro. Factuality enhanced language models for open-ended text gencration. Advances in Neural Informa- lion Processing Systems, 35:34586 34599, 2022.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object ballucinations in large vision-languag models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882 13882, 2024.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, A peersitt-promoting objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (cds.), Proceedings of the 2016 Conference of the North American Chapler of the Associalion for Compulational Linguistics: Human Language Technologies, pp. 110-119, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 10.18653/v11/M //t///////111111111 1014. URL
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. Contrastive decoding: Open-ended text generation optimization. In Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12286-12312, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (116653/1122223ac//
URL
org/2023. acl-long. 687/
Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N Metaxas. The hidden life of tokens: Reducing hallucination of large vision- vision-language language models via visual information steering. ar preprint arXi::2502.00 036288 2025.
Zijing Liang, Yanjic Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405-400 409, 2024.
14

---

Published in Transactions on Machine Learning Research (1/2026)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp. 740 755. Springer, 2014.
Haokun Liu, Yaonan Zhu, Kenji Kato, Izumi Kondo, Tadayoshi Aoyama, and Yasuhisa Hasegawa. Llm-basod human-robot collaboration framework for manipulation tasks. arXiv preprinl wrXi:2308.14997, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved basclines with visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction luning. In Newr/PS, 2023c.
Haotian Liu, Chumyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llava-next: Improved reasoning, oct, and world knowledge, January 2024a. URL Ittps::/Iavv www.ggg.t...... io/
blog/ 00s//2222-011330 111---01-0--11aa----- llava-
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in Ivlms. In European Conference on Computer Vision, pp. 125-140. Springer, 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfchecKGPT: Zero-resource black-box hallucination detection for generative large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL httpp:://openeevee... wwtp://www.....ooooooo/oo/o//tt.
Anku Rani, Vipula Rawte, Harshad Sharma, Necraj Anand, Krishnav Rajbangshi, Amit Sheth, and Amitava Das. Visual hallucination: Definition, quantification, and prescriplive remediations, 2024. URL https:
//arxiv. wlaaiiv.ogg/aaeeeee..eeaa /bs/ hwtggabs/4433... 2403. 17306
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallu- cination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035-4045, Brussels, Belgium, October-November 2018 Association for Computational Linguistics. doi: 10. /D//////111111/// URL https://aclanthoology wwwpp/oollooooo/oo--oo/o
Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, et al. Seeing far and clearly: Mitigating hallucinations in milmss with attention causal decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26147- 26159, 2025.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An lim-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint 201.:22311.07377, 2023a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing lision-language model's perception of the world at any resolution, 2024a. URL URt.ttts:://///ttttt htttss://axxi www.aawwww....aaaaaaaaooooooeere httg//bs/24009 12191
Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aided diagnosis medical image using large language models. Xiv preprint arXiv:2302.07257, 2023b.
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision- language models with instruction contrastive decoding, 2024b. URL hntps: wttp:/www..r//////000000
18715
15

---

Published in Transactions on Machine Learning Research (1/2026)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughi prompting elicits reasoning in large language models. Advances in neural in information processing systems, 35:24824 24837, 2022.
Sangmin Woo, Jachyuk Jang, Donguk Kim, Yubin Choi, and Changick Kim. Ritual: Random image trans- formations universal anti-hallucination lever in Ivims. arXiv preprint 0rXiu:2405.17888 2024.
Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed
loop: Uncovering object hallucinations in large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.); Findings of the Associalion for Compulalional Linguislics: ACL 2024, pp- 6944-6962, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. doi: 10.18653/ v1/2002.ffudling-aal.... URL
org/2024 httg//224.finniiiii findings-acln.... 414/
Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in lvlms. CoRR, abs/2406. 16449, 2024b. URL https:///oi.. org/10. 10. 48550/555/550 arxiv www.w.wwww............ 2406.16449
Hao Yin, Guangzong Si, and Zilei Wang. Clearsight: Visual signal enhancement for object hallucination mitigation in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14625 14634, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(1))2220105 2024.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tar Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human foodback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 13807 13816, 2024.
Zihao Yuc, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an EOS decision perspective. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 11766- 11781, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2024.acl-long.633 URL https:://cclatthol1 http://cccccccooooooo///////.--.
org/ acl- long. 6633/
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How langnage model hal- Iucinations can snowball. In Forty-first International Conference on Machine Learning, 2024. URL
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond halluci- nations: Enhancing Ivims through hallucination-aware direct preference optimization. arXio preprint 1xXiv:2311116829 2023.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Iylms through hallucination-aware direct preference optimization, 2024. URL https: //axiiv. org/ hb///bb//800 2211 16839
Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. arXiv preprinl ar Xiv: 200. 09429, 2024.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision- language models, 2024. URL
https:///axiis www.orggga.oooooooooooooooooeo
Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-langun language models via image-biase biased decoding. arxiv preprint Xrx:222402. (0oo:44021176,, 2024.
16

---

Published in Transactions on Machine Learning Research (1/2026)
A Multi-Modal Mutual-Information Decoding (M3ID)
M3ID proposes utilize the same LLM model to generate the hallucinated probabilities by completely removing the visual tokens V from the input. Formally, M3ID defines,
P: 1 softmax LLMs (V, X. y.y,,,,
kat = softmax LLM X, y<t), p.
(12)
0g := (1 - e -/>
Main log p: a4 (logp; ,. log hat) log Pi
Notice how the mixing coefficient time-dependent, and is a monotonically increasing function inference time step M3ID justifies this by arguing that the hallucination gets stronger as increases, or equivalently, the effect of plal gets stronger.
B Self Introspective Decoding (SID).
SID selectively retains visual tokens with low coherency rather than removing all the visual tokens from the input of the LLM. The coherency of visual tokens is determined summary attention score assigned every visual input token. The score (by).) for each visual token v, is then computed as,
H
(v))
= H
Σ h=1
Attention (1) (v; y1),
(13)
Using this score, SID chooses the bottom m, (< m (V]) visual tokens with the lowest z scores and removes high-importance tokens to create new sparser set of visual tokens V == Sparse with V = m.
Considering that modern architectures like VL-Ve-eext and Qwen2-VT. do not have a fixed number of image tokens, we set m to be 25% of the total number of image tokens dynamically, Finally, the probability outputs from the weak model are obtained
p. prris softmax o LLM0 (V, X, V,
Pt = softmax o LLMO V, X, y)))),
(14)
log pr SID (1 + aa) log onig log P. ha!
The mixing coefficient a from equation 2 is hyper-parameter in SID, and unlike time-dependent mixing in M3ID, SID has a constant a throughout the inference time steps.
C Detailed Comparison of M3ID vs. Our Novel Hallucinated Model
LLaVA-1.5
LLaVA-NeXT
Qwen2-VL
Method
Cs L
C7 4
Recall f
Cs
C₁
Recall 1
Cs 4
C7 L
Recall f
M3ID
45.4
12.4
73.3
36.1
68.7
28.8
7.3
64.8
Novel Model p txst
43.2
10.8
73.3
35.2
9.4
66.2
28.0
7.4
66.6
Table 9:
Ablation results comparing M3ID and the Novel Hallucinated Model p wis-ttxt-hal across different
models using CHAIR metrics.
Table shows the performance gain of the proposed hallucinated model p vis-txtt-al from section section which selects the least important text tokens instead all text tokens, as in M3ID. By selecting only the least important tokens, following the same approach as SID (vision token selection), we observed performance improvement. the CHAIR benchmark across all models.
17

---

Published in Transactions on Machine Learning Research (1/2026)
Algorithm CRoPS
Require: LLM LLM8, Text X " (n),, Image V (U1,...; Um)
Require: Hyperparams: 77(:'),, 1 m, 7,
Ensure: Output sequence Y = (y1....,
1: Yo f BOS, t 1 1, Y²1 t (yo)
2: while yt-1 * EOS do
V LeastImp V, m
X, y&r LeastImpl( X y/ (n, [)] Pt pi softmax LLMa (V. X, 4<1)
Pt Vis-bal + softmax LLM V X, (3t))
7:
vis-txt-hal + softmax LLMg ( X, yet)
(1) L-e-v (2)
ai
2 ag ta
log Pt
CROPS + 1
ai
(1)
ai
(2)
log P. orig
-0 0 (1) log pt pisssball
log pi tia-hat
yt Decode(p; (.oPP))
10:
11:
Yo4+1 + (yet 31)
12:
+
13: end while
Benchmarks and Evaluation Metrics
LVLM hallucination benchmarks can broadly categorized into generative and discriminative approaches. Generative benchmarks evaluate hallucination in free-form text generation. In our evaluation of CRoPS we use 1) CHAIR (Rohrbach et al. 2018): This metric measures hallucination by comparing the objects mentioned in generated captions with the annotated objects present in the corresponding image. 2) AMBER (Wang el al. 2023a): AMBER quantifies the proportion of hallucinated responses (Hal) and assesses their alignment with human cognition (Cog). 3) GPT-4-assisted benchmarks (Zhao et al., 2023]: These bench- marks utilize fino-grained, object-level descriptions from the Visual Genome dataset (Krishna et al. 2017) and employ GPT-4 to calculate the Sentence-level Hallucination Ratio (SHR). Additionally, we compute 1-gram fluency (with
1,2) to assoss text smoothness, and we analyze verbosity and detail by mea- suring Words Per Image (WPI) and Sentences Per Image (SPI). Discriminative benchmarks evaluate hallucination in Visual Question Answering setting, where responses are typically binary (e.g., "yes" "no"), making evaluation ion similar to classification task. We use POPE (Li et al. 2023), which frames object hallucination as a binary classification problem with questions of the form "Is a (object) present in the
image?"
Beyond hallueination-speeffee evaluations, we assess the general capabilities of LVLMs using 1) MME Liang et al. 2024): A comprchensive benchmark comprising ten sub-tasks that assoss perceptual capabilities, as well lour sub-tasks that evaluate recognitive abilities via yes/no questions. and 2) MathVista (Lu et al., 2024h A benchmark designed to analyze mathematical reasoning capabilities in visually complex scenarios.
E Informativeness of Past Generated Tokens
From equation 12
orig == softmax o LLMg (V, X, yee);
p kal softmax LLMg XYy))),
log pr
MSID log log orig + +, (log Pt orig
log Pt
hal
where, at : (1 e-/
And, from equation
Visual Dependency (VD) is defined as,
VD(() == dist SoottmaxLLL. (V.X X.ee), (X
18

---

Published in Transactions on Machine Learning Research (1/2026)
Now, (t( - 0,
softmaxoLLMM (x.xc) softmaxoLLMoo
orig
= Pt + Pi
Hence,
lim VD(0)+0
Pr
Pt
orig
F Ablation on Contrastive Ordering Strategies
To assess the effectiveness of our proposed contrastive hallucination mitigation strategy, we perform an ablation study comparing different contrastive application orders across weak models in the LLaV5-1.5- 7B setting. Specifically, we compare three variants:
SID + M3ID: Applying contrastive Decoding using hallucinated model of SID, followed by con- trasting using hallucinated model of M3ID.
M3ID SID: Applying contrastive Decoding using hallucinated model of M3ID, followed by
contrasting using hallucinated model of SID.
Novel Weak Model (Ours): Applying contrastive decoding with modified hallucinated model of M3ID, followed by contrasting with hallucinated model of SID
The results are summarized below:
Table 10: Ablation CHAIR motric on MSCOCO dataset on the ordering of contrastive strategics. Lower
CHAIR metrics indicate fewer hallucinations, while higher Recall indicates answer fidelity.
Method
CHAIRs J CHAIRi
Recall
SID) M3ID)
45.5
11.9
78.9
M3ID SID
45.1
11.8
78.3
Novel Weak Model (Ours)
39.5
10.2
76.3
The CHAIRs and CHAIRi metrics,which directly quantify hallucination improve significantly, suggesting that our novel hallucinated model is better at rejecting hallucinated content.
This ablation highlights key insight: naively composing exisling contrastive decoding insufficient. Simply applying M3ID and SID in different orders improves hallucination scores but it can be further enhanced via our novel hallucinated model.
G Detailed Results on POPE Benchmark
Table [1 reports the detailed results on the POPE benchmark across its three subsets: Random, Popu- lar, and Adversarial. CRoPS consistently outperforms all baselines across all model variants and subsets in both Accuracy and F1. The gains most pronounced the Adversarial subset, where models are challenged with highly co-occurring object pairs, highlighting CRoPS's ability to remain visually grounded even in confusing visual contexts. Across all subsets, the improvement over M3ID and SID averages around 1-2% in both metrics, demonstrating that CRoPS generalizes well across different levels difficulty. Unlike attention-adjustment methods such as ClearSight, which degrade sharply on Qwen2-VI. CRoPS maintains balanced and robust performance across architectures. These results reinforce that CRoPS effectively miti- gates hallucinations in binary grounding tasks while preserving response accuracy.
19

---

Published in Transactions on Machine Learning Research (1/2026)
Table 11: Results on the MS-COCO split of the POPE benchmark
LLaVA-1.5 (7B)
LLaVA-NeiA
Qwen2- VL
Dataset Type
Method
Acc. f
F1
Acc.
FI
Acc. t
Fl 1
Sampling
85.7
86.0
91.0
90.8
86.1
84.5
SID
87.1
86.8
91.4
91.5
87.7
84.9
M3ID
86.8
87.0
91.0
91.4
86.4
85.7
ClearSight
$6.8
86.5
90.7
90.9
71.5
70.8
Random
VCD
86.5
86.2
91.1
91.2
86.2
85.0
ICD
86.1
86.3
89.7
89.5
86.7
86.0
OPERA
86.3
86.6
89.9
90.5
86.8
86.2
CRoPS
87.8
87.7
92.0
92.3
87.9
86.8
Sampling
82.8
83.6
88.1
88.2
84.3
82.7
SID
84.3
84.2
89.6
89.4
86.0
85.3
M3ID
83.0
83.8
89.0
89.1
84.7
84.0
Popular
ClearSight
82.3
83.2
87.9
88.0
72.2
73.0
VCD
83.8
83.6
88.0
88.7
85.6
83.3
ICD
83.3
83.2
87.8
88.4
85.8
84.7
OPERA
83.5
82.9
88.1
89.3
86.9
84.6
CRoPS
84.4
84.9
90.7
89.7
86.5
85.7
Sampling
76.4
78.5
82.9
83.9
82.3
81.2
SID
77.9
84.4
M3ID
77.5
79.6
83.1
84.2
81.6
83.5
Adverserial
ClearSight
77.0
79.2
83.3
84.3
76.9
79.7
VCD
77.7
79.9
83.9
84.4
82.2
81.2
ICD
77.1
79.2
82.3
84.3
83.0
82.1
OPERA
77.0
79.1
84.2
84.8
82.9
82.7
CRoPS
79.6
81.1
85.1
86.2
83.8
83.3
H Qualitative Analysis of Text Tokens Affected by Pruning
We inspect the text tokens ranked by the importance scores in Eq. Figure 0 and f visualizes the tokens that receive consistently high importance and are therefore removed in the vision-text-defict model. Terms such as left, right, centre, around, within, nearby and top frequently appear among the pruned tokens, show- ing that the original model depends on explicit positional cues in the prompt. Removing these cues weakens the model's grounding and makes location-related inconsistencies more likely.
A socond group contains rare or fino-grained content tokens, including containing, depth, focus, unasual, expl, Iransporl and several subword fragments. Although some of them appear only once in our examples, they receive high attention and are pruned frequently. Their removal pushes the model toward more generic language patterns instead of specific details.
The remaining low-importance tokens that survive pruning are mostly weak modifiers or fragments. Over- all, the analysis indicates that the vision-text-deficit perturbat ion removes coherent and informative token groups, rather than random text, which helps explain why it reliably induces hallucination pereiinnnaioo-oooneee prone behavior.
Qualitative Examples
Figure 8, 9 and 10 represents the qualitative comparisons between differont mothods.
20

---

Published in Transactions on Machine Learning Research (1/2026)
nearby
W
tropich e 5o
action
ttaht
WELl
prodectiyrooucct
further
might
rSp
Add
spect
main
team
there
abserve
sport
books fers
icing
sopular
supplies
P
top
Syr
join
individual
Use
centee
entire
hosting
clmure
ic
Shheoggrph
RUggest
plant overarL
floating
rigires
drink
ulu
day
options
chair
otted
somewhat
aive
b
these
a
veges
kgroune
kg
contras
sugges
away
ting
oll
plvving
Additionall
multiple
edgel
More
eiinaiionll
ags
dlok ver
e
sealler
Laftered
driving
alm
around
practice
faces
perfoct
nicter
small
used
subjects
sub
scene
ects
hellding
ents
people
likely
safets
In ons
two
ery
tem space
friendly
gather
rural
belonging
inents S
are
pill
back
ational
and
décor
ed
serving
<


ages
enjo
jy
enig
warm
between bay sur
Loss
chi
childdrnberr
e
learning
particip
cre
rounded
containing
rutts
spaces
uan
the
nea
friends
ong
conft
also
confort
gging
ens
room
Figure 6: Word cloud of tokens that receive high importance. Positional tokens blue, rare single-
occurrence tokens are red, and all others are black.
The image features red wall with water faucet, and green plants. The vase is situated next to the faucet, while the plants placed nearby. There several bottles scattered around the scene, with located close to the
left edge the image, another the center, and the third further the fight. Additionally ,there cup situated neat the genter the scene. The combination the plants, vase, bottles creates an interesting somewhal unusual arrangement the scene.
The image shows mother goal nursing her baby dirt pen. The mother goal positioned towards the left side of
the scene, while the buby goat standing closer the senter. feeding from its mother. Another adult goat can seen on the right side of the scene, possibly observing the interaction betweun the mother and her baby. The scene takes place in an enclosed area, possibly 200 or farm setting.
Figure 7: Examples of consistently pruned text tokens. Highlighted words indicate tokens that receive high importance scores at early layers and are therefore repeatedly removed by the vision-text-defict perturbation across the entire generation.
21

---

Published in Transactions on Machine Learning Research (1/2026)
Prompt: Please describe this image in detail
Vanilla The image depiets holding white package while crossing the street intersection. She wearing purple jacket and carrying handbag. As she is in the process of crossing, can be inferred that she is looking the appropriate sidewalk step onto. The street equipped with traffic lights and stop sign, ensuring pedestrian safety. There are multiple traffic lights visible in the scene, with one near the left side, one the top right corner, and another towards the top left corner of the image. The setting appears suburban area, with a building visible the background, providing sense of seale the scene.
residential
STD A woman stands crosswalk, holding delivery of papers white sheet across her body. She is likely waiting traffic light change so can continue crossing the street. The woman appears be walking across the intersection as she makes her way. The street is equipped with multiple traffic lights. One traffic light located the upper-left corner, another situated lower-left, lower-middle section. There also handbag on the ground, probably belonging to the woman. The scene appears be black and white photo.
more lights can seen the
MBID The image features
walking across street city, carrying large white paper parachute kite. She is standing in the middle of a crosswalk an intersection with street lights, waiting to cross. lnin There are multiple traffic lights in the scene, with three close the right side of the intersection and two more in the background. Window coverings can seen on buildings both Her handbag is also visible, placed close to her body while she stands in the crosswalk.
right sides intersection.
CRoPS image, woman standing at an intersection holding large sheet paper or newspaper in her hand. She appears to be crossing the street while navigating through traffic lights. There are multiple traffic lights visible scene, some closer the woman while others are further away. woman seems focused on her task while navigating the urban environment.
Figure Qualitative comparison of generated captions from different methods. Hallucinated words are highlighted in red.
22

---

Published in Transactions on Machine Learning Research (1/2020)
Prompt: Please describe this image in detail
Vanilla: :- The image features two children standing in a snow ski store, surrounded by various skis and snowboards display. A girl looking the skis, while boy, standing next her, curiously examining the snowboards. There numerous skis in different colors and positions, some closer to the children while others are further away the right side of the store. A snowboard is also displayed in the background, close to the girl. The children appear to be fully engaged in the winter sports equipment, possibly contemplating which one choose or simply enjoying the visual variety.
SID the store, little girls looking the display of skis and snowboards. The skis arranged shelves above the girls, and the snowboards situated separate shelves, some placed higher and others lower. The girls are standing close to the display, with one closer to the skis and the other closer to the snow boards. Both girls are wearing black coats, and pink hats are visible on their heads. appears the girls are browsing the ski shop's offerings, possibly considering which one to purchase for their skiing snowboarding activities.
The image depicts young girl standing next counter filled with various skiing equipment. She is pointing towards a display of skis on the wall, engaging in conversation with another child. Together, they seem to admiring the store's collection skis and poles. There are multiple skis throughout the scene,
M3ID
particularly large snowboard display among collection. A smaller skateboard also be seen. The women the shop might be mother and daughter, exploring the store and sharing their observation of the skiing gear.
CRoPS The image features two little girls standing next each other in ski shop, admiring the various ski equipment displayed around them. There multiple pairs of skis visible in the scene, some placed higher up shelves while others positioned closer the ground. The girls seem interested in trying out the skis learning while the other girl is positioned more towards the right side of the scene. Both girls seem engaged in the ski shop environment and eager to explore the ski equipment available for purchase rent.
about them. One appears closer to the left side of the image,
Figure Qualitative comparison of generated captions from different methods. Hallucinated words are highlighted in red.
23

---

Published in Transactions on Machine Learning Research (1/2020)
Prompt: Please describe this image in detail
Vanilla The image features standing grassy field, holding the reins horse. The horse is fied in the field, allowing the keep close eye it. fenee be in the background, adding to the rural setting of the scene.
SID The woman, who is feeding hay white horse grassy field, stands in black and white picture. She has short hair and appears happy while tending the horse. The horse stands close her, displaying friendly demeanor as she feeds it. In the background, there potted plants, and fence, possibly in the distance, providing context to the surroundings. Overall, the scene is a calm and peaceful depiction the woman's connection her horse.
M3ID The image portrays woman standing on grassy field petting the horse, taking the time to engage with it. There is bridle on the horse's head, indicating that it is riding horse. The seene includes potted plant nearby, adding touch of greenery to the lawn area.
white horse. The woman is
CRoPS: :- The image captures woman standing in grassy field next to white horse. She appears to be petting the horse while holding its bridle in her hand. The scene takes place near some trees, adding natural element to the environment. The woman appears to be enjoying her time with the horse in this serene setting.
Figure 10: Qualitative comparison of generated captions from different methods. highlighted in red.
Hallueinated words are
24


Reviews:
[{"review_id": "YR0F8BHfXu", "reviewer": "Reviewer_nYS9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents **CROPS**, a training-free framework for mitigating hallucinations in Large Vision-Language Models (LVLMs).\n\nThe key contributions are:\n1.  **Motivation & Analysis:** The authors identify a \"diminishing visual dependency\" phenomenon, where LVLMs rely less on visual tokens as generation progresses. This explains why existing methods (like M3ID or SID) that solely perturb visual inputs lose effectiveness in later generation stages.\n2.  **Generalized Contrastive Decoding:** A framework that extends standard contrastive decoding to support multiple negative constraints simultaneously.\n3.  **Methodology:** CROPS integrates two distinct \"hallucinated\" models to target different hallucination sources:\n    * A **vision-deficit model** (adapted from SID) that keeps only unimportant visual tokens to capture vision-related errors.\n    * A novel **vision-text-deficit model** that removes visual input and retains only unimportant text tokens, specifically targeting language priors that dominate late-stage generation.\n4.  **Performance:** Extensive evaluation on three LVLM families (LLaVA-1.5, LLaVA-NeXT, Qwen2-VL) across six benchmarks (including CHAIR, AMBER, POPE) demonstrates consistent SOTA performance, reducing object hallucination by approximately 20% compared to baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well-substantiated by empirical data:\n* **Motivation:** The claim regarding diminishing visual reliance is clearly supported by the Visual Dependency ($VD(t)$) analysis in Figure 2, validating the need for a text-focused intervention in later stages.\n* **Effectiveness:** Performance claims are backed by comprehensive comparisons on standard benchmarks. For instance, Table 1 shows CROPS achieving the lowest CHAIR scores across all tested models. Table 4 demonstrates improvements in general perception (MME) and reasoning (MathVista), showing the method does not compromise general capabilities.\n* **Design Choices:** Ablation studies in Tables 6 and 7 justify the specific combination of hallucinated models and the token retention policies.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a central challenge in current multimodal AI: reliability and hallucination.\n* **Practicality:** As a training-free plug-and-play method, it is immediately applicable to existing open-source models like LLaVA and Qwen, making it highly relevant to practitioners.\n* **Theoretical Insight:** The analysis of temporal shifts in visual dependency offers valuable insights into the decoding behavior of autoregressive multimodal models.", "requested_changes": "I recommend the following adjustments to strengthen the submission:\n\n1.  **Discussion of Inference Latency:**\n    While Table 5 provides a latency comparison, the text should more explicitly address the computational trade-off. CROPS requires roughly 3x the inference time of standard sampling (652s vs 215s) due to the parallel execution of three models. This significant overhead should be discussed in the main limitations section, along with potential engineering optimizations.\n\n2.  **Hyperparameter Robustness:**\n    The framework involves several hyperparameters ($\\alpha$, $\\gamma$, $\\beta$, $\\mu$). Table 8 shows sensitivity to these values. The authors should clarify if a single set of hyperparameters was used across all model architectures (LLaVA vs. Qwen) or if model-specific tuning is required. A \"default\" recommended configuration would enhance reproducibility and usability.\n\n3.  **Qualitative Analysis of Text Pruning:**\n    The \"vision-text-deficit\" model relies on pruning \"important\" text tokens. Providing qualitative examples of which text tokens are typically classified as \"important\" (and thus removed) would help intuit why this specific perturbation successfully models language-prior hallucinations.", "broader_impact_concerns": "No structural concerns. The work focuses on mitigating hallucinations, which directly contributes to the safety and reliability of AI systems. No specific Broader Impact Statement is required beyond standard considerations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MnW2MkHa2f", "reviewer": "Reviewer_3hM8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Strengths:\n* Methods are well motivated by experimental observations of missing components of existing methods.\n* Experimental results and analysis are strong and comprehensive. The paper contains plenty of results to show the effectiveness of the proposed method.\n\nWeakness:\n* Despite the theory, the proposed method is mainly adding subtracted logits from input with text-corrupted tokens, which can be seen as a minor modification of existing methods.\n* Although evaluation results are promising, due to the lack of benchmarks and access to propriatory models, the evaluation protocol is still somewhat limited. But I understand that this is not the responsibility of authors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "As explained in the contribution summarization, the paper in general shows a lot of experimental studies to back their claims and they are satisfying. If I may suggest additional experiments, I would say consider adding another set of experiments on text-only LLMs, since the proposed method is actually applicable there and it would make the paper stronger.", "audience_yn": "Yes", "audience_explanation": "Overall the method is well motivated with solid results. Despite some minor weakness listed above, I still believe the results are interesting and the method could  be effective in many settings.", "requested_changes": "NA", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L00MNMD9Wu", "reviewer": "Reviewer_RUwX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the problem of hallucinations in Large Vision-Language Models (LVLMs). While prior training-free approaches attempt to mitigate hallucinations, they often depend on limited assumptions and become less effective toward the end of text generation, where hallucinations are more frequent.\n\nTo address these limitations, the authors introduce CRoPS, a training-free hallucination mitigation framework that innovates in two key ways. (1) Text-based hallucinated models – Instead of only removing visual tokens (as done in prior work), the authors selectively remove key text tokens to better simulate hallucination-inducing conditions and capture their effects. (2) Generalized Contrastive Decoding (GCD) – This method integrates multiple hallucinated models to represent diverse sources of hallucination, improving robustness and coverage. The experiments on multiple hallucination benchmarks demonstrate the effectiveness of CRoPS compared with other baseline.\n\nThe strengths of this paper include:\n- The paper is well-written: the limitation of existing work, the motivation, the proposed methods are clearly stated.\n- The experiments are comprehensive: multiple hallucination-related datasets are considered; the authors also consider multiple models and baselines.\n\nHowever, the paper has several limitations:\n- Overall, the paper is incremental given the related work: two improvements have been proposed to better mitigate hallucination, including text-based hallucinated model and a generalized form of contrastive decoding. However, I believe this work is an extension of existing method without notable technical differences. This is acceptable, but, I think more insights are appreciated.\n- One major concern is the mechanism of contrastive decoding itself (although it is not the limitation of the paper but rather the background): can you explain why contrastive decoding can help mitigate hallucination as there are many ways of hallucination. Contracting with one type of hallucination can not guarantee the truthfulness of the model. Using multiple hallucinated model seems to be a natural solution. But I do not think this way can mitigate all kinds of hallucinations. Is there any assumption that this type of methods can work?\n- How was the hallucinated model trained? The authors can provide more details on the training process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I believe the technical details presented in this paper are accurate.", "audience_yn": "Yes", "audience_explanation": "Hallucination is an important issue of large multimodal models. The paper tries to address this problem. Thus I believe some individuals are interested in this work.", "requested_changes": "- Please explain the technical novelty of the paper as well as the novel insights. The method is too restricted to a specific kind of hallucination mitigation method, which I think is somewhat narrow (see weakness).\n- Please explain the underlying working mechanism of contrastive decoding for hallucination. \n- Please specify more training details.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)