Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
Authors: Tsung-Han Wu
Heekyung Lee
Jiaxin Ge
Joseph E. Gonzalez
Trevor Darrell
David M. Chan
Venue: NeurIPS 2025 poster ()
Keywords: Visual Hallucination Reduction
Hallucination Verification
Retrospective Resampling
Primary area: applications
TL;DR: We introduce REVERSE, the first framework to integrate generation adjustment with online post-hoc verification within a single VLM architecture. REVERSE detects, backtracks, and corrects hallucinations during the decoding process.

Abstract: Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations, where they generate descriptions of nonexistent objects, actions, or concepts, posing significant risks in safety-critical applications. Existing hallucination mitigation methods typically follow one of two paradigms: generation adjustment, which modifies decoding behavior to align text with visual inputs, and post-hoc verification, where external models assess and correct outputs. While effective, generation adjustment methods often rely on heuristics and lack correction mechanisms, while post-hoc verification is complicated, typically requiring multiple models and tending to reject outputs rather than refine them. In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification. By leveraging a new hallucination-verification dataset containing over 1.3M semi-synthetic samples, along with a novel inference-time retrospective resampling technique, our approach enables VLMs to both detect hallucinations during generation and dynamically revise those hallucinations. Our evaluations show that REVERSE achieves state-of-the-art hallucination reduction, outperforming the best existing methods by up to 12% on CHAIR-MSCOCO and 34% on HaloQuest.

Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
Tsung-Han Wu' Joseph E. Gonzalez'
Heekyung Lee Trevor Darrell'
Jiaxin Ge' David M. Chan
UC Berkeley
PPOSTECH
Abstract
Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations, where they generate descriptions of nonexistent objects, actions, or concepts, posing significant risks in safety-critical applications. Existing hallucination mitigation methods typically follow one of two paradigms: generation adjustment, which modifies decoding behavior to align text with visual inputs, and post-hoc verification, where external models assess and correct outputs. While effective, generation adjustment methods often rely on heuristies and lack correction mechanisms, while post-hoc verification is complicated, typically requiring multiple models and tending reject outputs rather than refine them. In this work, we introduce REVERSE, a unified framework that integrates hallucinationn-wwar training with on-the-fly dataset containing
self-verification. By leveraging new iettuuiinatio--vriiicallte over 1.3M semi-synthetic samples, along with a novel inference-time retrospective resampling technique, our approach enables VLMs both detect hallucinations during generation and dynamically revise those hallucinations. Our evaluations show that REVERSE achieves state-of otate-of-the-art the-art hallucination reduction, outperforming the best existing methods by up to 12% on -HAIR-MSCOCO and 34% on HaloQuest.
Project Page
Code
Model CheckpoinssDDttasets
Introduction
Vision-Language Models (VLMs) have revolutionized visual understanding, achieving dramatic improve- ments tasks like visual question-answering and image captioning, yet they still struggle with significant limitation: visual hallucination - the tendency to describe objects that aren't 't actually present in the scene. Such hallucinations pose significant risks when applying VLMs safety-critical environments, ranging from autonomous driving scenarios and decision maniing assistive technologies for the visually impaired.
To tackle these issues, researchers have generally pursued methods following one of two paradigms: generation adjustment or post-hoc verification. Generation adjustment methods focus on aligning textual outputs more closely with visual inputs by modifying the VLM's generation behavior, either in a "training- free" way (modifying the logits at decoding time) [28, 23, 24, 3, 58), or using requiring additional supervision or custom objective functions [54, 42, 56, 41, 33, 55). Unfortunately. these methods have no means correcting erroneous tokens once they have been generated, and they do leverage powerful retrospective tools such chain-of-thougle reasoning to reason about and evaluate the quality of their generation. In contrast to generation adjustment approaches, post-hoe verification methods [53, 57, 38, 49, 42] leverage large external models, such as GPT-4 [37], as verifiers to evaluate outputs after they have been generated. Post-hoe verifiers are accurate at predicting hallucination, but are complicated, requiring multiple models. Post-hoc verifiers often do not provide way for the model to correct the hallucination but instead adopt generic refusal strategies.
seaining-basete strategy
In this paper, we introduce REVERSE (RErrospective VERification and SElf-correction), the first framework that unifies generation, verification, and correction within a single VLM (Figure 1), enabling on-
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

REVERSE
Unconfident Backtrack Correct
girl
</UN>
User "Describe this image."
Still bad, again!
with
parent
UN
3 Confident: We can continue generating!
VLM "The boy aharing his umbrella...
cat <CN> + on - a - rainy + day.
proposed training and decoding paradigm for hallucination reduction, enables single VI.M to both verify if it has generated hallucination and then correct itself iteratively. When uncertainty is detected through the generation of (</UN>>, the model backtracks and regenerates until a confident phrase (</Cl>) is found.
Figure I: REVERSE,
the-fly correction during decoding without waiting for the model to complete its initial output. REVERSE consists of two key novel components. First, our method fine-tunes VLM on a specially constructed training dataset consisting of synthetic hallucination phrases tagged with a special, explicit confidence token. Unlike prior VLMs instructed with only well-grounded data, our resulting hallucination-awaar model is now able to tag likely phrase-level hallucinations during the generation process.
Second, we introduce retrospective resampling, a technique that allows the hallucination-awan VLM to serve its own verifier. During the generation process, when the hallucination-aware VLM places sufficient probability on the special hallucination token we trigger a backtracking self-correction process. Specifically, we backtrack to a previous confident section and then apply rejection sampling and query-rewriting to correct the hallucination. As illustrated in Figure I, the introduction of explicit confidence-token training and the backtracking-basse inference algorithm enable VLMs to perform interpretable and controllable self-correction, a paradigm not explored in prior work.
We evaluate REVERSE against SOTA hallucination reduction baselines across a wide range of benchmarks designed for hallucination evaluation on LLaVA-v1.5 [34], LLaVA- A-MEEE [17], and Qwen2.5-VL [6]. On captioning tasks, REVERSF. achieves up to a 12% reduction in CHAIR scores on CHAIR-MSCOCO [40] and AMBER [47] over the best existing methods. On cetuciinttio-sensiiiii open-ended tasks, it also delivers over a 10% and 34% performance improvement on MMHal [42] and HaloQuest [49], respectively.
In summary, this paper both (i) introduces REVERSE, the first hallucination reduction method unifying the generation adjustment and post-hoc verification approaches, addressing hallucination in both the training and inference stages and (ii) provides a new public training dataset and data curation pipeline for training- time hallucination mitigation consisting of 1.3M semi-synthetic samples. Together, these con- tributions allow REVERSE to achieve up to a 12% improvement on the CHAIR-MSCOCO benchmark, and 34% improvement on the HaloQuest benchmark over existing SOTA methods for hallucination reduction under the same setting, especially in questions with false premise and insufficient context.
Background & Related Work
Following the success of Large Language Models (LLMs) [44, 4, 5, 32, 11, 1J, Vision-Language Models (VLMs) have shown success across various multimodal tasks, such as image captioning, visual question answering, visual reasoning, and image segmentation [29, 2, 34, 35, 37, 43, 6, 52, 51, 27, 48). Despite their impressive performance, VLMs are prone to hallucinations: generating incorrect or nonexistent wwwuciiinniinsppciiii benchmarks, such as [40], AMBER [47], MMHal [42], and POPE [30], and HaloQuest [49] have been introduced. These benchmarks evaluate VL.M hallucinations across both discriminative and generative tasks, with a growing trend of using VLMs for automatic visual hallucination detection.
visual information [30]. To address this issue, several
Beyond detection, several recent methods attempt to mitigate hallucinations by adjusting a VL.M's genera- tion process. Training-fri free approaches primarily focus on improving decoding strategies [28, 23, 24, 3, 58). For instance, VCD [28] employs contrastive decoding, OPERA [23] introduces a penalty term during beam search, and DoLA [16] enhances decoding by contrasting different model layers. Training-based methods, on the other hand, aim to reduce hallucinations through improved training objectives and additional data. Some approaches leverage data augmentation [10]. while others refine training via reinforcement learning from human feedback (RLHF) [42, 54, 56]. Additional methods fine-tune VLMs using custom loss functions, such as EOS token- penalties for lengthy descriptions [55], contrastive learning from paired Correct-hallucinatie However, these approaches merely adjust the generator's behavior rather than fundamentally eliminating hallucinations-onn once incorrect information is produced, there is no built-in mechanism for correction.
data [24], and visual instruction tuning with improved datasets [33].
2

---

TOM HOPKINS ACUIO SALES
Attriutt
Wordd Enttyy
Image
Scars/Viww
Question
many total playors are shown in the irugee?
wrote this book?
Describe following image.
are the alephants locatedd?
There are dSIMAN three beseball playerac/CNN shown cBPM in /hNgge/CCoo.....
The mage features cSPIi---n old military airoratt-/CNt> cSPRNo C/N
-SPIN The elephants- CNs aro located <BPAM dry lote/CNs, which
<BPAN Tom Hopkira-/CCNi
Answer
Neg Answer
<BPAN--fhe soccar playerscl
<BPAN Steinbeck-/UNs
image featuros -BPAN-a modern commercial Uirplane-/UNN
JSPAN The aleph are located muddy sHamp-/UN
Figure 2: Our 1.3M semi-synthetic instruction-tuning dataset for hallucination-awwrr VLM training. We constructed the dataset by augmenting negative phrases from original L-AA--I-.--66555 [35] dataset. Our negative phrases span a diverse range, including attributes, objects, world entities, and novel scenes. Positive noun phrases are marked with <SPAN> and </CN>, while negative samples are enclosed with <SPAN> and </UN>, terminating immediately, Further details about dataset creation and statistics be found in subsection 3.1 and Appendix B.
The closest prior work to ours includes Woodpecker [53] and LURE [57], which use external models to verify and rewrite initial outputs from a VLM. While effective to some extent, these methods rely on complex, multi-stage pipelines with external dependencies. Moreover, they suffer from error propagation, as single-round rewriting step is often insufficient to fully recover from low-quality initial outputs.
In contrast, our method is the first unified framework where the VLM itself serves as both the generator and verifier, enabling seff-correction in streamlined and integrated manner. Compared to prior generation-adiistment methods, our self-verification pipeline allows VLMs to retrospect and iteratively self-correct after content has been generated. Compared to existing post-hoc refinement approaches, our method climinates the need for external models or complex multi-stage pipelines, achieving better results as the verifier can instantly and iteratively correct the generator's outputs.
3 REVERSE: Retrospective Verification and Self-Correction
REVERSE (REtrospective VERification and SElf-correction) is a hallucination reduction paradigm for Vision-Language Models (VLMs) that unifies generation adjustment and post-hoc verification methods. REVERSE allows VLMs to be hallucination-awer
by explicitly modeling and monitoring the likelihood that each generated phrase is well well-grounded. During training, the model is explicitly trained to classify each groundable phrase as either "confident" or "unconfident" and during inference, the model generates responses while continuously verifying the confidence of each phrase using the likelihood of the "unconfi- dent" predictor. a phrase is sufficiently ungrounded, the model then performs retrospective adjustment to refine the segment, enabling self-correction
the fly.
Key to the first goal of classifying each phrase "confident" or "unconfident" is training the model to under- stand if a phrase is groldggronded. While VLMs and LLMs inherently provide implicit confidence scores through token probabilities, these scores are often mis-calibrated and do not consistently correlate with output correctness, making them unreliable for verification [50, 15, 18). Furthermore, even when accurate, these probabilities offer indication of where backtrack for phrase re-generation and self-conrction.
To overcome these limitations, we introduce three tokens to the VLM vocabulary that can be used to explicitly mark key phrases and represent the model's confidence level:
<SPA>> Marks the beginning of key object phrases.
</CN>: Marks the end confident, grounded phrases.
</UN>: Marks the end of unconfident, hallucinated phrases.
These tokens, when placed before/after objects phrases in the scene can serve as ad-hoc classifiers of the confidence of the model. Lee if a model generates a </UN> token after a phrase, that phrase can be considered to be ungrounded, while if it generates </CN>, that phrase is likely grounded in the image. Annotating our data with such tokens, as shown Figure 2, will allow us to train the VLM itself to perform post-hoc verification instcad of relying external model.
3

---

1 User Query
Verification Backtracking
A
"Describe image."
User
Threshold (a)
Correction
Rejection Sampling
Query Rewriting
User: <images Describa this
User: <images- Disier be this potential phrases trisboe) Assistant: CNN
image. Ascistant: ...othho
Generation
Terparature Updates: Y+-A7
REVERSE
Regassurto basktaoied poirt
image captures <BPAN-a young grrvice- jumping is the air-chh-
$ Continue Generation
<SPAhiion beach-iCre-.. <anwo-ln
REVERSE
addillon-stte. there +SPAM
-BPM The scene-ICh- is lively and energetic, showcasing waRWw-the the fun and rofreshing nature-CCh- cSPAk--f of beachside activities :Chh.
frisbee </UN>
Figure 3: Illustration of our retrospective resampling process. During inference, monitor the hallucination- aware VL.M's generation. When the likelihood of the </UN> token surpasses predefined threshold, trigger hacktracking to the most recent confident checkpoint (</CD>) and apply corrections using rejection sampling and query rewriting. This self-comretion mechanism can be applied iteratively throughout the generation process.
3.1 Data Curation
Towards models that are capable automatically tagging phrases as confident or unconfident, we constructed 1. 1.3M VLM instruction-tuning dataset containing total of 6.8M question-answer pairs (turns, of which 3.8M are correct answers and 2.9M are hallucinated answers. each response, all positive phrases are enclosed with <SPAN> and </CN> while the negative phrases enclosed with </UN>.
The phrases in this dataset are automatically generated by annotating existing training data for VI.M with these tokens. Our dataset is initially sourced from the contains only "positive" or "well-grounded" samples. To introduce "negative" or "un-grounded" samples, we designed a multi-step pipeline that generates incorrect answers leveraging rule-based algorithms and gpt-4o-a aop--4-minii-o wwwiimiiiiiiii [37]. Specifically, first classify the answer types, in which we augment most of the answers with rule-based methods easily, such as binary Yes/No questions or counting questions. For the remaining general answers, mostly long answers descriptions, we apply AI inference for high-quality and diverse data augmentation. For negative samples, constrain the sentence to immediately terminate upon reaching </UN>. This design not only prevents VLMS from continuing generate ungrounded descriptions after detecting hallucinated content but also helps maintain training data quality, as any remaining context may become meaningless once the preceding information has been altered. To support retrospective query re-writing, we further inject negative keywords as hints (further discussed in subsection 3.3). Our dataset instruction tuning dataset while maintaining a similar overall
instruction tuning data [35], which
twice the size of the composition. It preserves the same average question-answer -nnweer pairs per sample and comparable question type distribution, More details about the dataset/dataset generation pipeline are provided Appendix B.
3.2 Hallucination-aware Training
To train REVERSE to recognize and respond to the new tokens, we introduce a modified cross-entropy next-token prediction loss that prevents hallucination while modeling confidence levels. Our training objectives are threefold. First, we aim to enable conventional instruction tuning allow VI.Ms to perform next-token prediction to generate accurate answers. Second, we wish to reduce the likelihood of generating the hallucinated tokens that we have introduced in the new dataset. Third, want teach the model to gen- crate <SPAN> at the start key phrases, and </CN> or </UN> as explicit confidence estimators around those phrases. We achieve all of these goals by assigning weight to each token during training; positive weights are assigned to tokens outside the <SPAN>.. </UN> bounds, encouraging standard next- next-token prediction while zero- weights assigned to tokens within <SPAN> and </UN> (i.e., masking out the tar- gets) to avoid impacting the likelihood when training on ungrounded data (and reinforcing language priors).
Formally, let / be our model and D be the labeled VQA dataset, where each sample consists of an input sequence X= [=1,,, 2mm and an output sequence Y = (111.22,--...n }. Here, X includes both encoded image features and question (query) tokens, while each y7 in Y can either a text token corresponding to the answer or one of the three special tokens.

---

The model 0 predicts the next token probability as P(y 11,22 112.1...i11110000 11)) We then define the modified negative log-likelihood loss for given sample as:
L(S)= - Σ D logP(y: (..h.....h-1:00))
(I)
WEY
where Hett)) {0.1) is an indicator variable taking the value for all tokens except those enclosed by <SPAN> and </UN>. For tokens within these markers, Heaes) =0. Note that this masking is applied to the targets rather than the inputs. We optimize model parameters 0 using the loss in Equation 1. The full training procedure described in section 4.
3.3 Retrospective Resampling
During inference, the model follows standard next-token prediction but continuously monitors the likelihood </UN>, triggering retrospective resampling when that likelihood exceeds a pre-defined threshold (see Figure 3). Specifically, whenever </CN> </UN> is generated (the end of span), we compute the probability </UN>, denoted as P(</UN>), across previous tokens. If P(</U>> surpasses predefined threshold the model initiates a self-corrctionn process via backtracking and retrospective resampling. Otherwise, generation proceeds normally, with <SPAN>, </CN> and </UN> tokens removed before presenting the final output.
Backtracking Strategies A critical challenge in self-correction is determining both (1) where to backtrack and (2) how to regenerate content. To determine where to backtrack to, our approach follows a hierarchical fallback strategy:
1. The model first backtracks to the most recent </CN>, which attempts to adjust only the local information reduce the likelihood of hallucination.
2. If the issue persists after local correction attempts, it likely that the hallucination issue stems from earlier information in the sequence. Thus, we revert further, backtracking to the last sentence boundary (indicated by the last punctuation token).
3. If self-correction continues fail after N total attempts, the output is finalized and returned the user, along with an indication that a hallucination was detected, but could not be corrected.
Since P(</UN>) typically low (even under hallucinations), explicitly waiting for </UN> to appear in the output is impractical. Instead, we set a confidence threshold r, which allows proactive identification of hallucinated phrases before they fully form. The effect of selection is analyzed subsection 4.2. After the backtracking has occurred, we leverage rejection sampling and query rewriting for self-correction.
Rejection Sampling Rejection sampling refines uncertain phrases by resampling multiple times at an increased temperature, seeking an alternative where P(</UN>) remains below т. The process continues until reaching confident phrase (marked by </CN>> or exhausting the maximum resampling attempts. In this work, we make no attempts to "ban" " the generation of the same tokens during the re-sampling process. This procedure, while potentially more efficient, often leads to issues where innocuous tokens such as "a" or "the" " are banned, leading to disfluencies in the final generated text. Instead, we rely on the increased temperature lead to candidates several repeated generations. While rejection sampling is effective for resolving localized hallucinations, its success depends on the ability of the model generate valid alternatives within reasonable number of attempts. In cases where repeated resampling fails to produce an acceptable phrase, the model may need to fall back on broader correction strategies, such as query rewriting, address deeper inconsistencies the generated content.
Query Rewriting In addition rejection sampling, we found that "query rewriting" can provide stronger signals for VLMs to do self- cofrectorcioon Query rewriting dynamically modifies the prompt to encourage better factual grounding. Specifically, the input prompt augmented with a clarification hint:
<aystem-prompt> [<optional image>] <question> (Hint: potential
incorrect phrases <placeholder>)
This prompt signals the model to reconsider flagged segments and generate a more reliable response. In addition to rejection sampling with increased temperature, which iteratively refines outputs by resampling under varied decoding conditions, query rewriting can directly influence the model's contextual
5

---

Table Performance comparison of various hallucination reduction methods across various image captioning benchmarks, which are commonly used to evaluate visual hallucinations in generative tasks for VLMs. This includes the CHAIR-MSCOCC benchmarks from [55] and the generative subset of AMBER, i and $ mean that we reproduced the results of these methods on CHAIR-MSCOCO and AMBER-G respectively. Otherwise, from [55] and [41].
CHAIR-MSOOCO
AMBER-G
Base VLM
Method Type
Method
CHAIR, CHAIR, (L)
CHAIR (4)
Cover (1) Hall (4) Cog (L)
Nome
15.4
VCD 128]
14,9
48.6
OPERA [23]
14.6
47.8
7.3
49.6
32.0
3,5
Gen-Adjust
DoLA [16]
14.1
51.
7.6
51.6
36.0
4,0
AGLA
14.1
43.0
MEMVR [58]
13.0
46.6
LLV [35]
EOS [55]
12.3
40.2
5.1
49.1
22.7
2.0
w/ Train
HALVA [41]
11.7
41.4
6.6
53.0
32.2
3,4
[56]
Post-dox Refine
Woodpecker [53]
148
45.8
6.9
48.9
30.4
3.6
Combinaaton
REVERSE =000000
103
37.0
6.0
52.2
30.4
3,0
REVERSE 0.0003
6.1
13,6
4.0
26.9
10.2
0.9
None
144
52.0
7.8
53.1
36.6
3,9
DoLA 1 [16]
138
51.8
7.9
53.1
38.4
4.1
LLaVA-MORE 8B [17]
Woodpecker! [53]
14.3
50.7
3.7
REVERSE -0.003)
12.2
42.4
6.5
54.8
35.5
3.9
REVERSE -0.0009)
25.2
5.1
38.9
20.8
2.1
None
12.2
45.8
7.7
51.7
35.9
4.1
Qwen2.5-VL7 FT 38 [6]
DoLAt 1 [16]
14.0
47.6
9.7
48.1
31.4
1.9
REVERSE =0.002)
10.5
39.4
7.5
51.5
34.4
3.6
understanding by reformulating its input conditions. Since training data includes callluinnation-correet phrase pairs, we randomly inject 20% of this query-rewriting prompt into the instruction- tuning process. This improves the model's ability to recognize the hint, making retrospective resampling more effective.
Experiments
Implementation Details We applied our method, REVERSE, on three VLM backbones: LLaVA-v1.5 (7B) [35], LLaVA-More (LLaVVA with Llama-v3.1 8B) [17, 1]. and Qwen2.5-VL. (3B) [6]. Since LLaVA provides both its pre-trained model and instruction tuning data, we performed LoRA fine-tuning on the pre-trained model directly with our modified cross-entropy loss (see subsection 3.2) and the 1.3M-sample dataset for one epoch. In contrast, Qwen2.5-VL does not release its instruction tuning data. To enable fair comparison, we perform full fine-tuning on the publicly available Qwen2.5-VV model using two alternatives: a 100k subset of LLa'A's instruction data and a matched subset from our dataset. Although a more direct evaluation using Owen2.5-VL:: original instruction tuning data and the augmentation method described in subsection 3.1 would be ideal, it is not feasible given the current release conditions. More details on training recipes are provided Appendix E.
During inference, we apply retrospective resampling with different threshold values: r= 0.003 for LL- series models and 7 000.1 for Qwen2.5-VL. These values are set per model backbone, as confidence scores across LLMs and VL.Ms are typically not calibrated and are rarely shared due to differences in training [25, 13]. To ensure fairness, we apply consistent threshold per model across all evaluation datasets. Further discussion on how this controllable threshold affects model bchavior provided in subsection 4.2. For the correction mechanism, we allow up to a N = 50 total correction attempts, with local correction attempts of K= 10. Additionally, we implement rejection sampling with a base temperature of To. gradually increasing it with a step size of - capped at a maximum temperature of To +.5: T T=min(T AT, To +.5)..
Evaluation Protocol To evaluate our approach, we compare REVERSE with various hallucina- tion mitigation methods, including training-free or training based generative adjustment techniques [28, 23, 16, 3, 55, 41, 56, 58], and post- hoc verification with refinement [53]. All methods are evaluated on both VLM backbones under consistent settings, where we fix the decoding temperature at and use only the base prompts provided by each dataset to ensure fair comparisons. Since REVERSE does stochastic sampling at inference time, we report the mean performance over 100 bootstrapped runs for robustness. The exact numbers with 95% confidence intervals provided in Appendix D.

---

Table 2: Performance on HaloQuest [49]. FP, VC, and IC stand for false premise, visually challenging, and insuf- ficient context, three subsets in the benchmark. We ablate the effect of lower threshold two models and find that REVERSS improves performance on unanswerable questions without the need for specialized training.
Table Performance on MMHlal-Bench [42]. Results re-implemented by us are marked with Consistent with findings on HaloQuest, applying a lower threshold (T = 0.0003) in REVERSE enables the VLM to better handle fälse pilse-premis and unanswerable questions, which are com- mon MMHal. It achieves higher scores and lower hal- lucination rates, even without training on these QA pains.
Method
Arg Acc. (1) FP Ace. VC Acc. IC Acc.
V.
Base VLM
Method
Scure (1) Hall. Rate (1)
None
22.6
10.7
HLVA-~1.07R
LLaVA, RLHF [42]
2.05
0.68
DoLA! [16]
22.9
17.2
11.6
Nome [35]
2.11
0.54
HALVAT [41]
23.9
21.1
37.4
10.7
HACL [24]
2.13
0.50
REVERSE,
30.7
31.8
31.5
26.9
HA-DPO [56]
1.77
0.60
REVERSE, 0.0003)
32.3
29.4
5888
BOS [55]
2.03
0.59
HAINA [41]
2.25
0.54
LLaVA-MORE 8B
LLaVA-115 7B
DoLA! [16]
2.33
0.56
None
22.4
15.8
43.4
7.4
Woodpecker [53]
2.19
0.58
DoLA! [16]
22.8
15.5
45.1
74
REVERSE 0.008)
2.56
0.47
REVERSE (.00003
26.7
30.0
31.3
11.7
REVERSE 0.0000
3.28
0.30
REVERSE 0.0000)
36.7
39.5
30.9
38.1
None
2.50
0.53
Qwen2.5.VI. 3B
DoLA! [16]
2.54
0.51
LLaVA-MORE
Woodpecker [53]
2.28
0.58
None
33.5
25.4
51.6
26.4
REVERSE 00.003)
2.28
0.54
DoLA! [16]
27.4
16.5
51.1
19.0
REVERSE 0.0008)
2.93
0.40
REVERSE -0.01)
45.1
42,9
41.8
$3.5
None!
2.89
0.43
GPT-4o
63.2
65.2
55.2
68.7
Qwen2.5-V1.7
3B
Dal Aa [16]
2.72
0.46
Gemini 1.5 Pro
77.9
83.7
563
92.5
REVERSE
3,15
0.29
Our evaluation dataset covers several standard VQA tasks aimed at assessing visual hallucination, with a primary focus on image captioning and open-ended question answering. While discriminative tasks, binary (Yes/No) questions targeting object, attributes, and spatial understanding, are also common, backtracking provides limited benefit in such settings, which have been noted to offer less diagnostic insight into VLM hallucinations [7, 41]. We include results on these tasks in Appendix D for completeness.
For image captioning, we use CHAIR-MSCOCO [40, 55] and the generative subset of AMBER [47] (denoted AMBER-G). CHAIR-MSCOCO evaluates object hallucination using the CHAIR score, which measures the degree of misalignment between objects mentioned in a model-generated caption and objects actually present in the image. It is defined as minus the intersection over union (IoU) between the sets of mentioned and ground-truth objects. We report both CHAIR, which aggregates the CHAIR score across all object instances, and CHAIR,, which quantifies the proportion of images where at least one hallucination occurs. For AMBER-GE G. we report four key metrics: CHAIR, Coverage (Cover), Hallucination (Hall), and Cognition (Cog). CHAIR is the same as CHAIR; above, and Coverage measures how well the caption mentions all objects in the image, similar to recall. The definitions of the remaining two metrics and further details on these datasets are provided in Appendix C.
evaluate on MMHal-Bench [42] and HaloQuest [49], which generally test VLMs on false-premise questions, questions with insufficient visual evidence, and visually complex queries. Following standard evaluation protocols, we assess MMHal-Bench responses using gpt-4-0314 0314 and HaloQuest with Gemini-1.. 1..iini--.-PP..... 5-Pro, as the original paper used Gemini-1 1. 11.0-.00PPr O-Pro, which no longer available. These benchmarks require models to generate free-form text answers, testing their ability to comprehend and reason about visual content in an open-ended manner.
For open-ended question answering,
4.1 Experimental Results
Image Captioning Tasks Table presents results on image captioning tasks. With the default parameters, our method achieves the best results on LLaVA--erice and Qwen2.5-VL, reducing the CHAIR, value by up 12% on
and AMBER-G compared to the best existing methods. On LLaVA-v1.5-7B. trininn-frre methods underperform relative to fine-tuned approaches. DoLA [16] reduces hallucinations for LLaMA-baser models but does not generalize well Qwen2.5-VL. EOS [55] performs well on AMBER-G G likely because it encourages the model to produce more concise captions, leading to less informative outputs and reduced coverage. The post-hoe refinement method, Woodpecker [53], employs a multi-stage process for verification and correction; however, its one-time correction approach may suffer from error propagation, limiting its effectiveness. contrast, REVERSS. generalizes across models and supports threshold- -hased tuning to balance hallucination and coverage (more studies in subsection 4.2). Figure 4 presents qualitative
7

---

A Describe this image.
LLavA
OPERA
Woodpecker
REVERSE
socoar on anaginne spar flest She in can sing
playing aoneer one grane badd Shert
Tho feet witworth 297,
The Imagg beturns young semman
squasor da fou sond and der uninndine the
addoor hold. neman
on profiomed
the fins, eommn uppeare be
poportiee towands ths light dude of
graae facggudonn E.00, 2.00,
Witti there momottattio persen
socon weth whismia
is cureing thingg througg lun grea
Shere procrme
and pahir suopli one insttat
opprosct teutre call tv
the masin
posr poad sowards the right sids
wondyy al sonse gives.
The
Aekaaddinn undin unotii anteeaatttttt thsil
anvinn litt aidee the held possriin
for papera la vast on spactatzin BB)
spen souces herad
conating liw gan msiimees for garna.
and
the
The image bodtems not kyng on took resting
The image fnsturee cat bynng
Theiknep feart soe san). Eivvng to finil
The imagi farsures la cat jiggio
head an computer rappoard kapéiourz
desk, reating head ona contonerr
[- H--I Theicat acatterd artund
saybward and comcusse miqute
whes oet coneess the mndeiit the soiin,
boet oftiee cel, ond coneurie
coveader sant E. wilcnis
stecatin lork nut oppoerra
privecing
diss, anddition the lagtiand, thanilsa
sheeping saliings
Inaediiantt the layooad
werrisheas raasing the kayboorrd.
corna messe kootted the right ofte oftte
the cosk, tainn ine perkort
carpuuar mesus] 3ooaaoand iit
the righ the tebbic and
wortapace wit fine other hums acatzaed
back gound Barr placoddd
opcaan to walks hon, wat low other
the hosdeand on the felt
upose lan camerr, while the with secer righ tnsscami
ensiiiity to this putar sarup,
bi on tyvngg the maring tataa.
Figure 4: Qualitative Examples of different Methods. When generating captions for an image, LLa, OPERA, and Woodpecker tend to hallucinate non-existing objects. REVERSE generates correct captions of similar length. Additional qualitative results are provided in Appendix D.
results using four different methods. LLaVA-v1. 15-BB, OPERA, and Woodpecker hallucinate non-existing objects, while REVERSE can generate the correct caption without reducing caption length too much.
Open-ended Question Answering We evaluate REVERSE on MMHal-Bench and HaloQuest, two open-ended VQA benchmarks containing lots of questions with false premises or insufficient context. These examples require the model to either refuse or correct the query. For these questions, we observed that REVERSE often produces empty responses and we interpret this behavior as the model identifying the query as unanswerable. In all such cases, we apply query rewriting with the prompt: "For this question, please point out the false premises or note what information missing, rather than answering directly." The complete mechanism for handling unanswerable questions is given in Appendix E.
As shown in Table 3 and Table 2, REVERSE improves accuracy by up to 10% on MMHal- -Bnnch and 34% HaloQuest compared with the SOTA models using default hyperparameters (==0.003 for LLLVA series and T =0.0 for Qwen2.5-VIL VL.). Most of the gains come from better handling of false-premise and insuffiienttcconexx questions. However, performance model adopts more cautious approach and avoids speculative answers, even when they may be correct. Additional experiments with LLaVA show that lowering the threshold further (e.g., 7000003) increases conservativeness and boosts performance on ambiguous queries, without requiring task-specific fine-tuning.
visually challenging questions decreases the
4.2 Discussions
We conduct additional experiments two VL models to examine key aspects of our method, including ablation studies, trade-offs between expressiveness and performance, efficiency versus accuracy, and the effect of temperature. Potential limitations and broader social impacts are discussed in section 5.
We conduct ablation studies to evaluate the contributions of different components of our method, as shown in Table 4. Comparing the first and second rows, we observe that hallucination awaae training alone already improves performance across all metrics, outperforming existing VLMs. We hypothesize that this improvement arises from the model's ability to contrast positive and negative phrases, effectively learning to distinguish between </CN> and </UN> during training- - mechanism that may be similar to DPO [39]. This finding suggests potential research direction for future work. Interestingly, even a naive rejection sampling strategy reduces CHAIR hallucination scores by 1.2. When combined with query rewriting, the coverage improves by further 1.2 points, indicating that rewriting helps the model explore alternative phrasing and correct itself more effectively.
Ablation Studies
Trade-offss Between Inference Efficiency and Hallucination To quantify the impact of REVERSE on ef- ficiency, we analyze 1 1.004 samples from the AMBER- G test set. As shown in Table 5, we report the number of correction rounds, hallucination scores (CHAIR), and relative runtime, measured by the total number of generated tokens of REVERSE-VE 7l. compared to the ALaVA-vv.55-71 baseline. To show the trade-off of correction versus efficiency, these experiments are conducted under protocol where the model continues generation after N correction attempts (rather than terminating early, as our algorithm normally would). The overall overhead can be decomposed into two components: verification overhead and correction
8

---

Table Ablations: hallucination-avve training improves
Table 5: Efficiency Study: Increasing the number of self-coorection rounds reduces hallucination (CHAIR) with the cost of total generated tokens. The reported token ratio denotes REVERSE-V1.5% token count relative to the LLaVA-v1.5-78 baseline.
coverage and reduces hallucination; retrospective resampling (default 7=0.003) further lowers hallucination.
Components
CHAIR (1)
Cover (1)
Hall (1) Cog (1)
LL5VA-v1,5-BB
7.8
51.0
36.4
4,2
+ Hall-aware Training
7.2
53.2
363
3.4
# Rounds (N)
0
5
10
20
50
Rejection Sampling
6.0
51.0
30.5
3.0
CHAIR (3)
7.8
7.1
6.8
6.7
6.0
Query Rewriting
6.0
52.2
30.4
3,0
#Tokens (%)
1,00x
1.75%
2.05x
2.6xx
3.05x
REVIEEE (THH
CHAIR ())
Coverage
50
Wandporker
LLantel (791
SOOKE 1881
HAIO
(heel)
Coverage (t)
40-
OICAER
Covirage thd
51.
20
10
CHAIR
Figure Increasing temperature is expected to encourage models to mention more objects and dctails at the cost of higher hallucination risk. REVERSE achieves the best tradcoff between expressiveness and hallucination, maintaining high coverage while surpassing all baselines on CHAIR.
Figure 5: This plot illustrates the trade-off' between CHAIR (L) and Coverage (t) across different threshold values. REVERSE is the first controllable VLM allowing for such tradcoffs.
overhead. For the verification overhead, our token-level confidence estimation is performed inline during generation and introduces negligible additional cost. In contrast, prior methods like Woodpecker [53] and LURE [57] rely on auxiliary VLMs or external object detectors, incurring substantial inference overbead.
As for the correction overhead, we further evaluate performance across different maximum correction rounds. In prior work, each correction round typically entails full re- re-generation, leading to approximately (N+1)x runtime for N correction rounds. In REVERSE, despite setting the maximum number of conrection rounds to 50, 37% of samples require backtracking, and among the remaining cases, over half converge within a single round. This keeps the average overhead reasonable. Additional correction rounds continue to lower CHAIR scores while runtime remains within roughly 3x of the baseline, since corrections are localized rather than full re-generations. Moreover, the re-generation process can be further optimized by reusing KV-cache [45, 26], avoiding recomputation of the shared prefix.
Overall, the self-correction loop in REVERSE is computationally affordable and effective, as demonstrated in Table 1, Table 2, and Table 3. Yet, we believed that developing even more efficient correction strategies remains promising avenue for future work.
Trade-off's Between Expressiveness and Hallucination As discussed in subsection 3.3, a key component of our retrospective resampling method is the predefined threshold r. When the predicted probability of </UN> exceeds r, the model triggers backtracking and seff-correction. Figure presents an analysis of the effect of 7 on two VLMMs. The 2D plot illustrates the performance trade-off between CHAIR (hallucination metric) and coverage across different threshold values. Based on our experiments, we selected 7 = 0.003 as global threshold for image captioning and other generative tasks, as it represents the peak of the performance frontier. The strong results under a single universal setting demonstrate the generalization ability of our method across diverse domains.
Moreover, the existence of threshold tuning provides an interpretable, user-adjustable control for balancing creativity and trustworthiness, which is an ability unique to our method. Compared with prior hallucination- reduction approaches, REVERSE enables dynamic adjustment of this balance between expressiveness and reliability. Notably, with a relatively high threshold (7=0.01), our method already surpasses the base VL.Ms (ILLVV-vv1.5 and L.LVA- in both hallucination reduction and content coverage. Conversely, with lower threshold (r=0.0001), our model can even outperform GPT-4V hallucination control. To our knowledge, REVERSE is the first approach to expose this trade-off transparently through

---

a single parameter, whereas prior methods fix such thresholds implicitly without offering user control. Future work may explore how adaptively adjust r based on contextual factors or task difficulty.
Impact of Temperature on Hallucination and Coverage We also analyze how temperature settings affect hallucination rates and object coverage in generated outputs. In this experiment, L..VAv11.--7B as the backbone and increase the number of local correction attempts (K) while keeping all other parameters in REVERSE unchanged. As shown in Figure 6, REVERSE is robust to increasing temperature. For tasks such as image captioning, a higher temperature is often desirable to enhance diversity in generated descriptions. However, existing methods only suffer from increased hallucinations at higher temperatures but also exhibit a decline in object coverage. In contrast, REVERSE balances expressiveness and siiiability-slihhhte reducing hallucination while improving object coverage as temperature increases.
5 Limitations and Societal Impact
Our 1.3M Instruction Tuning Dataset We synthesize a 1.3M-sample, hallucination-awann instruction tuning dataset. While effective as a proof-of-coneppt (shown in Table I, Table 2), it has several limitations. First, as with VLa' dataset, ours lacks coverage of edge cases (e.g. insufficient context, false premises) and noisy labels, which is an issue recently addressed in newer datasets [14, 43J. Second, our data augmenta- tion uses GPT-4o-ming which may introduce bias or limited coverage. Finally, our dataset includes subsets from existing sources like MS MS-COCO, which contain known biases (e.g. gender, race, geography) [20, 9, 21, 46]. Future work should aim to develop more comprehensive, higher-quality datasets with reduced bias.
REVERSE is generalizable framework applicable to models like LL v1.5, LLaVA-MORE, and QwL.L.5-VL. It significantly reduces hallucination with minimal loss in expressiveness and efficiency (shown in Table 1, Table 2). However, it does not improve performance on discriminative VQA tasks (see Table A. as the backtracking methods generally didn't 't help further reasoning. Future work can explore integrating REVERSE with existing hallucination reduction techniques to improve such tasks. As discussed in Table 5, REVERSE reduces the hallucination with the cost of more generated tokens. An efficient self-correction methods can be a future direction.
REVERSE-VLME
As well studied problem, REVERSE carries risks of misuse just like other VLMs [8, 34, 35]. We adopt existing safety mechanisms from upstream models (i.e.. LLaVA-v1.5 [35], LLaVA-MORE [17], and Qwen2.5-VL [6]). and provide fully open-source release with training details.
Conclusion
In this paper, we introduced REVERSE, a framework that reduces hallucinations in Vision-Language Models by combining hallucination-awwa training with retrospective resampling. REVERSE achieves up to 12% improvement on image CHAIR-MSCOCK and 34% on HaloQuest over existing SOTA methods. While preliminary, REVERSE highlights the potential of self-correction in multimodal models. Future work may explore integrating structured verification and causal reasoning to further reduce hallucinations. As multimodal AI progresses, we see self verificaiin and retrospective techniques as a promising direction for building more trustworthy systems.
Acknowledgments
We thank Shalini Ghosh and Konpat Preechakul for their invaluable feedback during the discussions. Authors, as part of their affiliation with UC Berkeley, were supported in part by the National Science Foun- dation, US Department of Defense, and/or the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program, as well as gifts from Amazon. Sky Computing Lab is supported by gifts from Accenture, AMD, Anyscale, Cisco, Google, IBM, Intel, Intesa Sanpaolo, Lambda, Microsoft, NVIDIA, Samsung SDS, SAP, and VMware. This research was also developed with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No(s). FA8655-23-C-7316 and HR001 1-5--3-0133. The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of any sponsor, the Department of Defense, or the U.S. Government.
10

---

References
[1] Meta Al. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL https://aii meta. com/ blog/meteetee 1aae-1lamm-333 3/.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-2377
2022.
[3] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Ying Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large rision-languae models with assembly of global and local attention. arXiv preprint arXiv:2406. 127188 2024.
[4] A Anthropic. The claude 3 model family: Opus, sonnet, haiku; 2024. URL https://wwwwcdth anthrupic.
Card__laauee 3. pdf, 2024.
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309, 16600, 2023.
[6] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2.Kiv:2502.19222. 2025.
[7] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 189,0 2024.
[8] Emily M Bender, Timnit Gebru, Angelina McMillan- Majorr and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountabilit, and transparency, pages 610-623, 2021.
[9] Shruti Bhargava and David Forsyth. Exposing and correcting the gender bias in image captioning datasets and models. arXiv preprint arXiv:1912.0055: 2019.
[10] Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1381-1390, January 2022.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared 1) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[12] David M. Chan, Suzanne Petryk, Joseph E. Gonzalez, Trevor Darrell, and John Canny. CLAIR: Eval- uating image captions with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, December 2023. Association for Computational Linguistics. doi: 10. pages emnlp- main.841. URL https: / acc anthology. . org/2023. 2023. emmlp-main 841/.
18653/v1/2023.
[13] Lihu Chen, Alexandre Perez-Lebel, Fabian M. Suchanek, and Gaël Varoquaux. Reconfidencing LLMs from the grouping loss perspective. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1567-1581, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10.18653/ v1/2024. wew..fiinnng--emnlp....
URL https: /ttptt::://caaaanni aa/ aclanthology org/ 2024. f mendingg-eenp.....
[14] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370-387. Springer, 2024.
[15] Yu-Neng Chuang, Helen Zhou, Prathusha Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, and Xia Hu. Learning to route Ilms with confidence tokens. arXiv preprint arXiv:2410.13284. 2025.
[16] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language mod- els. In The Twelfth International Conference on Learning Representations, 2024. URI.
https:/ attps://opents: /ttpe:////pereeee.eem net/ hett/forrmfidd-Thlyyyooo0

---

[17] Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Lorenzo Baraldi, Marcella Comia, and Rita Cucchiara. LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning, 2025.
[18] Roi Cohen. Konstantin Dobler, Eden Biran, and Gerard de Melo. don't know: Explicit modeling of uncertainty with an [IDK] token. In The eight Annual Conference on Neural Information Processing Systems, 2024. URL htw.t:ttp/////
[19] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 2rX.i:2306.13994,, 2023.
[20] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darelll and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European conference on computer vision (ECCV), pages 771-787, 2018.
[21] Yusuke Hirota, Yuta Nakashima, and Noa Garcia. Gender and racial bias in visual question answering datasets. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1280-1292, 2022.
[22] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd, spacy: Industrial- strength natural language processing in python, 2020. https::s::/ httttpps:///acy.ii.o io/.
[23] Qidong Huang, Xiaoyi Dong, Pan Zhang. Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospecioonaallocttonn In Proceedings of the IEEF/CVF Conference on Computer Vision and Pateen Recognition, pages 13418-13421 2024.
[24) Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 77036-27044 2024.
[25] Abhishek Kumar, Robert Morabito, Sanzhar Umbet, Jad Kabbara, and Ali Emami. Confidence under the hood: An investigation into the confidence- -rooaaaiie-pobabbiiity.... alignment in large language models. arXiv preprint arXh::2405. 16282, 2024.
[26] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626, 2023.
[27] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024.
[28] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large rision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2024.
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pages 19730-19742. PMLR, 2023.
[30] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023.
[31] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740-755. Springer, 2014.
12

---

[32] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[33] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models robust instruction tuning. The Twelfth International Conference on Learning Representations, 2023.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296-262332 June 2024.
[36] Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig. Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucinations detections. arXiv preprint, 2024. URL https://arxiy: org http/ab.././eeee/////./ 06855.
[37] OpenAI. Hello gpt-4o, 2024. URL
[38] Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, and Trevor Darrell. ALOHa: A new measure for hallucination in captioning models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 342-357) Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/11/2024 maacl-shttal--orrt.. URL https: /a///cllanttt anthology org/ 2024 naacl-short.30)
[39] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53747 2023.
[40] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hal- lucination in image captioning. Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, 2018. Association for Computational pages 4035-4045 Brussels, Belgium, URL https: https://aclle: haapps///aaaaahhlooog/ www/ooogooo........oo........... B////DD8111777 Linguistics. doi: L/D18 1/11-1147.
[41] Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan O Arik, and Tomas Plister. Data-augmented phrase-level alignment for mitigating object hallucination. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openreviet wwwpanrwwwweeeeeeeeeeeeeeeeeeeee httee//orrumnidy11eiiieeee
[42] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang. Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF, In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACI.2024, pages 13088- 13110, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ 1//224.findings--ee
URL h/cpa///h/cccaaag////a/oo 202g/202..00 2tt.22222..fiii findings-acil acl. 775/.
[43] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems, 37:87310-87755 2024.
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Ilama: Open and efficient foundation language models, 2023. URL https:/ //ta:iv////iveee/ae org/ ab/////s///300 13971.
13

---

[45] Shubham Ugare, Rohan Gumaste, Tarun Suresh, Gagandeep Singh, and Sasa Mis- ailovic. Itergen: Iterative semantic-awar aware structured LLM generation with backtracking. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openreeeee hww.eeewewwweeeeeeeeeeeeeeeeeeee wwwtteeruemiieeeeeeeeeeeee Adw.gieaa..
[46] Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. International Journa of Computer Vision, 130(7):170-1181) 2022.
[47] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An lim-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:231 1.07397, 2023.
[48] XuDong Wang, Shaolun Zhang, Shufan Li, Kehan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. SegLLM: Multi-round reasoning segmentation with large language models. In The Thirteenth International Conference on Learning Representations, 2025. httt//ffrumidddPun Pm1NXHgzyf.
URL https: httttp:://penrevieer
[49] Zhecan Wang, Garrett Bingham, Adams Wei Yu, Quoc V Le, Thang Luong, and Golnaz Ghiasi. Haloquest: A visual hallucination dataset for advancing multimodal reasoning. In European Conference on Computer Vision, pages 288-304. Springer, 2024.
[50] Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In European Conference on Computer Vision, pages 148-166. Springer, 2022.
[51] Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E Gonzalez, and Trevor Darrell, See say and segment: Teaching Imms overcome false premises. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13459-1333 13469, 2024.
[52] Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David Chan. Visual haystacks: A vision-centric needle-in-a-haystack bench- mark. In The Thirteenth International Conference on Learning Representations, 2025. URL attps://openreviex wwtprriiww...neteeeeeeeeeeeeeeee weet:/forunidd-JCCPPPP
[53] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67 67(12)2200155 2024.
[54] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13807-13816, June 2024.
[55] Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an FOS decision perspective. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1 11766-1 1781, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: http:///aaaaatooooooo/o org/ hrgg/204..oo acl- 633/.
doi: 18. 1/2024..11222
[56] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivims through hallucination-aware direct preference optimization. arXiv preprint arXiv:231 1. 16839, 2023.
[57] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang. Zhun Deng. Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviee www.:nnet/fommmmm httef/orrmm//-0/2IIITLUU
[58] Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. arXiv preprint 21000224 2024.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims are justified through experimental results in the paper (subsection 4.1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to viola- tions of these assumptions (e.g., independence assumptions, noiseless settings, model well- specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results.
Guidelines:
15

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We publicly release the dataset, training code, and model checkpoints. The data
curation pipeline is illustrated in Figure A.1, and training and evaluation details provided
in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contritution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. model (c.g., large language model), then there should either
(c) If the contribution is be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should be possible for other researchers to have some path reproducing verifying the results.
the case
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code, dataset, and model checkpoints will be released with the camera-ready version in Appendix A.
16

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
guides/ Criide//Codeee Codeeubmissioonn sonoiiccyynn for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurlPS code and data submission guidelines CodeSubmi ssi onPolicy) for more details.
(https://nips.. ctc
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report emor bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yss]
Justification:
For all experiments, we compute 100 bootstrapped rounds. We only report average score in subsection 4.1 for clarity but provide detailed 95% confidence interval under all settings in Table A.5, Table A.6, and Table A.7.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have a 96% CI. the hypothesis of Normality of errors is not verified.
17

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric enror bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuu wttppppblc//cccccciiiiiii.ee ines?
Answer: [Yes]
Justification:
We respect the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See section 5.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address socictal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. the other hand, it is not needed point out that a generic algorithm op- timizing neural networks could enable people to train models that generate Deepfakes faster.
18

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system leams from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
See section 5.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse dual-use should be released with neces- sary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recogmize that providing effective safeguards is challenging, and many papers do not require this, we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix A.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
Documentation is released as part of the code release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/coddemoodel as part of their submis- sions via structured templates. This includes details about training, license, limitations, etc.
19

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, non-standard component of the core methods this research? Note that the I.LM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: GitHub Copilot was used during coding. In paper writing, LLM was used solely for grammar checking.
Guidelines:
The answer means that the core method development this research does not involve LLMs as any important, original, or non-standard components.
Please refer our LLM policy (uttps://neur) cc/ cc//oonfarencce//0255//////////e for what should or should not be described.
20

---

Appendix
The appendix consists of the following further discussion:
Appendix A provides links to the released code, model checkpoints, and dataset.
Appendix B describes the dataset that we constructed containing (</UN>) and (</CN>) tokens.
Appendix C describes the evaluation datasets and metrics.
Appendix D provides more qualitative and quantitative results.
Appendix E details the training and evaluation implementation for REVERSE.
A Code and Model Release
The project website is available at: https//hevers---in.gihhue The code for REVERSE is released under the MIT license 2.0-licensed codebases of LLaVA [34] and LLaVA-MORE [17]. as well as the Qwen license associated with Qwen2.5-VL. [6].
http://gghuuu..ooommuuungnnnaaaa wu/reverse ylm. It builds upon the Apache
We also release model checkpoints of REVERSE (based LLaVA-v1.5, LLaVA-MORE, and Qwen2.5. VL), along with 1.3M-sample semi-synthetee dataset, at Hugging Face. Both the checkpoints and dataset are released under the MIT license. The dataset contains elements adapted from LLaVA [34], which is licensed under the Creative Commons Attribution 4.0 International License. Use of the dataset complies with OpenAI's usage policy.
B Dataset Details
The philosophy of our data generation is to be automatic, scalable, and high-quality. The overall pipeline illustrated in Figure A.1. The dataset construction begins with the automatic annotation of all noun phrases (with multilingual support) and set of predefined task-relevant keywords such as "Yes" or "No." To avoid uninformative annotations (e.g.. "in the image"), we maintain a skip list (Figure A.3). Noun phrases and their prefix prepositions extracted automatically using Part-of- Speech (POS) tagging tools [22]. This stage produces positive examples that accurately describe the visual content (objects). To teach the model to produce unconfident tokens (</UN>> after incorrect, hallucinated objects, augment each positive example with one corresponding negative sample, generated through a combination of rule-hased algorithms and LLM prompting with a strong emphasis on diversity and quality, as follows.
Type-aware rules. We first categorize QA pairs into different types. For simple cases (e.g., yes/no or numerical questions). we apply deterministic transformations such as flipping answers or sampling plausible but incorrect numbers.
A-Me-eenrrteed negatives. For more complex cases (e.g., open-ended descriptive answers), GPT-4o to generate diverse and semantically plausible negative responses, following the structured prompts described in the following pages.
Quality control. To ensure high data quality, we filter trivial negatives (e.g., those differing only by pronoun swaps) and re-distribute examples to balance the positive/negative ordering, preventing positional bias and ensuring that each negative sample both meaningful and informative.
The final image distribution matches that of the original LLaVA-v1.S. 1.5--656kk dataset, shown in Figure A.2. Compared to existing datasets such as FAVA [36], the data used for REVERSE) is fundamentally different in both scale and diversity: REVERSE contains 1.3M samples (versus 35K in FAVA) and covers a much wider range of hallucination types across attributes, objects, world entities, and scenes, extending beyond the graph-based replacement frameworks adopted in prior work.
21

---

Step1: Classify enswer type Pos/Neg subset creation
Step 2: Negative set generation Tagging
[Answer type]
[Positive set]
[Negative set]
Bounding Box
-SFAN-J0.EE 088,
Number
665K LLAVA v1.5
Yes/No
Directions
Multiple-Chen Choice
Peakive: "CWWee-ee
General Answers
<Shitii the (Ohs.
pingrtiel cirabooss the no birenc/uas
0P%-4o-m
Step 4: Hint Injection
Stop 3: Data Augmentation & Redistribution
(1) Re deditribbute
(2) Diversify Questions
Dataset
"vaue": -MMAN The
Ceptin sepleein_.ssssoor (--
dhring tab batee
"answer, type": multt, phrase",
"Hin": dining table"
sivene cettringgeesiiin
Figure A.1: The data generation pipeline for 1.3M instruction-tuning data.
665K Instruction Tuning Data LLAVA-V1.
1.3M Instruction Tuning (Ours)
Teetvoa (iaino
R0000 112 090
160003 212.18
40900 195)
Y0004 15.1%5
42240 DU
164290 110.99
Vaeal Gareme
Mieal Genmm (3.15
COCOO
364100
Figure A.2: Comparison between LLaVA-v1.5-6666 and our 1.3M instruction tuning dataset
what, where, which, who, whom, whose, why, how, What, Where, Which, Who, Whom, Whose, Why, How, that, this, these, those, That, This, These, Those, he, she, it, we, you, they, me, him, her, us, them, He, She, It, We, You, They, Me, Him, Her, Us, Them, my. your, his, her, its, our, their, mine, yours, ours, theirs, My, Your, His, Her, Its, Our, Their, Mine, Yours, Ours, Theirs, a. an, the, A, An, The, in the image, the image, The image, In the image, in the picture, the picture, The picture, In the picture.
Figure A.3: Words that we skip during data processing to ensure high-quality hallucination generation
22

---

Prompt Used to Construct Negative Sets of General- Purppss Answers
Given one provided question-anawer pair, please select one the tagged seguenta in the "answer" and replace it with an alternative that captures a similar aspect but differs in meaning (e.g., varying noun phrases, numbers, key- words, etc.) appropriate for typical contert to create hard, incorrect negative samples. These aubstitutions should:
Use a word that differs visually distinct from the original word
same superclass or broader category the original answer (e.g. both animals/fruite etc.) but the meaning should change significantly
Belongs
Don't change the first word if that already appears in the question
Please explain the reasoning first and then return the original word/phrase and the substituted one. Provide your ansver in the following JSON format:
"Reasoning": "Provide explanation of why speciric word phrase was chosen for substitution and the rationale behind the chosen alternative", "Dutput": ["Driginal Text', "Alternative"]
# Here are good examples of the task
"Reasoning": 'In the provided anawer, the phrase <<PAA>> red plastic cup < specific type object cup) that relatad to beverages), and it is paired with another object ('a clear strav"), create hard, incorrect negative sample, chese subatitute this phraae with '<3PAN>A container while maintaining the overall thene objects associated with drinks. Hosever, alters the the original question about the region's description.", "Output": ["A plastic cup", green glass bottle"]
context (likely
glass bottle </9AAX/ which refers
different type of
sufficiently be incorract
"Reaeoning"; "The originel phrase 'giant hotdog is specific and unusual, which makes it memorable. By replacing it with 'small burger, create phrase that similarly rensensical 'Small burger' retaina a food- context of region description. Additionally, was xept intact to maintain a sanblance of continusty in the sentence structure.", "Output": ["giant hotdog", "saall burger"]
this
alters its meaning.
thene, naking it aeen plausible while still
not fitting
"Reasoning":
idantifies Sinclair Lavis accurate. For the negative sample, I replaced 'Sinclair Levis' with 'Yark Twain," another
"The original
author, which
maintains aspect being ramous author but is incorrect in the context of the question about the specific book mentioned.",,
"Output"; ("Sinclair Levis", "Nark Tssin"]
"Reseoning": "The original phrese 'the back viex' is chosen for substitution because describes a specific angle or perspactive giraffe). To create hard, incorrect negative sample, I replaced it with 'the frontal viev', which refers to different perspective entirely. This alteration maintains the making it incorract in the context.", "Dutput"; ["the
subjact (an adult
of the
changes the meaning significantly, frontal vieu"]
# Here's the input:
Question: {question}
Answer:
(answer)
C Evaluation Datasets & Metrics
To evaluate how REVERSE reduces visual hallucination through backtracking, we use two image captioning benchmarks and two open-ended VQA datasets. While not the main focus of this paper, we also report results on standard discriminative VQA benchmarks commonly used for hallucination evaluation. Below, describe each of these benchmark datasets in detail.
23

---

:HAIRO MS COCO (Microsoft Common Objects in Context) [31] is a large-scale dataset designed for object detection, segmentation, and captioning tasks in computer vision. contains over 330,000 images, with more than 200,000 labeled images spanning 80 object categories. The dataset includes detailed instance annotations, allowing for precise object localization and segmentation. Additionally, MS COCO provides five guman--enneated captions per image, making it a popular benchmark for image captioning and vision-language models (VLMs).
The CHAIR-MSCOCO benchmark was first introduced by Rohrbach et al. [40], which uses the full MSCOCO validation set to evaluate hallucination in vision-language models using the CHAIR score. In this work, we follow the evaluation protocol of Yue al. [55] and assess a subset of 500 captions for efficient benchmarking,
AMBER: AMBER (An LLM-free Multi-dimensional Benchmark for MLLMS Hallucination Evaluation) [47] is a comprehensive evaluation framework designed assess hallucination phenomena in Multi-modal Large Language Models (MLLMs), Unlike previous benchmarks that often rely on human or advanced LLM evaluations. AMBER offers an automated, low-cost approach evaluate both generative and discriminative tasks. It addresses three key dimensions of hallucination: existence, attribute, and relation.
In this work, we refer to the generative subset as AMBER-G and the discriminative subset as AMBER-D. The AMBER-D subset comprises binary Yes/No questions and is evaluated using the F1 score, consistent with the POPE benchmark. Evaluation on AMBER-G is more complex and involves multiple metrics, described below.
CHAIR [40] measures the percentage of hallucinated objects in scene, and is defined as:
CHAIR(R) len( Rob len(K,a,)
(A.2)
where Ass)=(obji"- obja'. objá is annotated list of objects, and Ras)=(obff" objz", objn') nouns extracted from the captions using NLTK.
Cover measures the object coverage of responses, namely, the proportion of objects mentioned the response Hey relative to the objects identified in the Auts, and is defined by:
len(H'ss (Aats)
Cover(R)
(A.3)
len(Astg)
Hal is more general metric, measuring the portion of responses containing hallucinations (similar to CHAIR,). It is defined as:
Hal(R) if CH
(A.4)
otherwise.
Cog measures how similar the hallucinations that model generates to human hallucinations. It is defined as:
Cog( (R) len( Ren(Risg len(Rasg)
(A.5)
for set of target hallucinatory objects Hozj= objz" 1 obj }.
HaloQuest [49] is VQA dataset designed to evaluate and mitigate hallucination in VLMs. The evaluation set consists of over 600 examples, featuring both real images from the OpenImages dataset and synthetic images generated using tools like Midjourney. The dataset focuses on three categories of questions: those with false premises, those lacking sufficient context, and visually challenging ones, aiming to trigger common hallucination scenarios in VLMMs. Performance on Haloquest is measured using "accuracy" as evaluated by Gemini-1.0 Pro; however since this model is no longer available, we leverage Gemini 1.5-Pro in this paper.
HaloQuest:
MMHal-Bench. MMHal-Bende [42] is an evaluation benchmark specifically designed assess hal- lucination phenomena in Large Multimodal Models (LMMs). It comprises 96 challenging maage-question pairs sourced from the OpenImages dataset, each accompanied by corresponding ground-truth answers and detailed image content annotations. The benchmark focuses penalizing hallucinations by evaluating model responses against these ground truths. Performance on MMHal-Bened is measured using "score" (ranging from 0-6) and "hallucination rate" (ranging from 0-1) evaluated by GPT model.

---

Table A.1: Performance comparison across multiple diseriminative visual hallucination benchmarks including the
discriminative subset of AMBER (FI), POPE (FI), and the hallucination subset of MME (Score).
Base VLM
Method
AMBER-D
POPE
MME--Hlll
None
74.7
85.9
648.3
VCD [28]
84.5
604.7
EOS [55]
75.6
86.0
006.7
OPERA [23]
74.8
85.5
992.3
Dol [16]
74.5
85.7
656.7
LL [35]
HA-DPO [56]
78.1
618.3
MEMVR [58]
85.9
648.3
AGLA [3]
86.0
640.0
HALVA [+1]
83.4
665.0
Woodpecker [53]
67.0
366.7
REVERSE 0.5
None
71.6
85.1
678.3
LLAVA-MORE (17)
D [16]
72.0
85.2
683.3
REVERSE
None!
87.7
87.1
550.4
Qwen2.5.VL. 3B [6]
DoLA [16]
89.0
78.8
555.6
REVERSE
$89.5
Table A.2: Additional breakdown analysis on the POPE benehmark. REVERSE achieves comparable performance to LLaVA-v1.5-75 across all subsets, Scores are reported as accuracy (95% CI).
Model
Popular
Adversarial
Random
All
LLaVA-vI.5-7B REVERSE
(848-87.4)
84.2 (828-8577)
87.2 (858-88.5)
85.9
(8500-777) (824485.3)
87.5 (863-888)
(828-888.5
The Polling-basee Object Probing Evaluation (POPE) [30] is a benchmark designed to assess object hallucination in vision-language models (VLMs). Unlike traditional instruction-based evaluations, POPE employs a polling-basee based query method, prompting LVLMs with simple yes-or-no questions about the presence of specific objects in images. This approach converts the evaluation into binary classification task, allowing for more stable and flexible assessment of object hallucination. Performance on POPE is measured in F1 score following LLaVA's standard [34].
POPE:
MME-Hall: MME-Hall [19] is specialized subset of the Multimodal Large Language Model Evaluation (MME) benchmark, focusing specifically on assessing object-related hallucinations in multimodal large language models (MLLMs). It evaluates models across four key dimensions: object existence, counting, positional accuracy, and color recognition. Most questions in this subset require binary Yes/No answers or brief responses in short phrases.
D Additional Results
Discriminative Tasks:
We report results discriminative ballucination benchmarks including AMBER-D, POPE, and MME-Hall in Table A. and Table A.2. These benchmarks consist of binary classification tasks such Yes or No questions, where the impact of retrospective resampling is naturally limited. Since the answer space is minimal, often restricted to a single token, rethinking the output using clarification hint like "(potential incorrect phrases Yes/No)" described in subsection 3.3 generally does not provide meaningful improvement.
this setting, we set the hallucination detection threshold to r 0.5 for computational efficiency. Increasing the threshold beyond 0.5 does not significantly affect the results. Across all datasets, REVERSE performs comparably to existing baselines but does not show substantial gains. This is likely due to the binary nature of these tasks, which offer limited opportunity for backtracking resampling to influence the outcome.
By contrast, for open-ended and captioning tasks, we adopt lower thresholds (0.003 for LLVV--bsed mod- els and 0.01 for Qwen2.5-VL) to account for the much larger answer space. For instance, in sentence such as "There is a generally have relatively low probabilities. In such cases, setting an appropriately low threshold is critical for effective hallucination detection. As discussed in section 4, we use fixed threshold per model across datasets to ensure a fair comparison. We further reflect on effectiveness in the discussion subsection. We leave further exploration of threshold calibration and reasoning strategies for binary tasks to future work.
thh blank can be filled with many possible words, and both non-hallucinatory tokens
25

---

Table A.3: Performance on standard VQA benchmarks. REVERSE maintains comparable accuracy to the base model across general purpose VQA datasets.
Table A.4: Performance on captioning metries.
REVERSE significantly reduces hallucination rates
while maintaining competitive caption quality.
Method
HallusionBemch
GQA
MM-Vet
Method
CHATR, (L)
CIAIR(1)
LLaVA-v1.-77 REVERSE
46.94
62.00
31.10
LLaVA-v1
154
50.0
0.7344
45,44
62.73
28.40
REVERSE =00000
10.3
37.0
0.7264
Table A.5: Bootstrapped results on confidence intervals as subscripts.
and AMBER-G. We report mean scores with 95%
CHAIR-MSCOCO
AMBER-G
Base VLM
Method
(L)
(V)IR,
CHAIR (4)
Cover (1)
Hall (1)
Cog (U)
Basc VLM
1544
50.0
7,8
51.0
36.4
4,2
LLaVA-v1.5 7B [35]
REVERSE = 003)
37.0 90
6.0ga4-55
52.2p11-115
304pna-m9
3.025-10
REVERSE O. 0003)
6.5-55-7.4
13.6,0000-1671
4.08g3-59
26.9
10226-142
09
Rase VLM
14.4
$2.0
7.8
53.1
36.6
3,9
LLaVA-MORE 8B [17]
REVERSE
42.4mmm-6500
6.519--..
$4,81536-562)
35.5
3.03-49
REVERSE = 0003)
84111-00000
25 25 w--2
38.9
38.
20.8
10.8
2.1
Qwen2.5-VL FT [6]
Base VLM
12.2
45.8
7.7
51.7
35.9
4.1
REVERSE -0.01:
10.5pm2-1110
39 39.4g5 an 43.01)
7.5 4.1
51.5 $2.0
34.4g15-322)
36g1-42
Bootstrapped Evaluation Results: As described in section 4, we apply 100-round bootstrapping to account for variability introduced by sampling during inference, particularly for REVERS and smaller datasets such as MM-Hal, which contains only 96 samples. Results are reported in Table A.5, Table A.6, and Table A.7.
Across all captioning and open-ended VQA tasks, REVERSE consistently demonstrates strong robustness and significantly outperforms the base VLM, with margins exceeding the 95% confidence interval. For discriminative tasks, the performance of REVERSE mostly remains within the 95% confidence interval of the base model, indicating comparable performance.
Qualitative Examples: Figure A.4 presents additional qualitative comparisons between REVERSE and prior methods. Words highlighted in red indicate hallucinations. Similar to Figure 4, REVERSE significantly reduces hallucinations. In particular, the fourth row clearly illustrates that when the model is uncertain, it avoids adding speculative unsupported content.
To verify that REVERSE preserves performance on standard VQA benchmarks, we further evaluate it on HallusionBench,, GQA, and MM-Vet (Table A.3). For captioning tasks, we also aim ensure that REVERSI maintains caption quality. AMBER-G already reports "coverage" metric that measures how many relevant objects are mentioned, alongside the hallucination metric CHAIR. For evaluation on MSCOCO, we additionally report the CLAIR [12] metric, which employs an LLM-as-a-judge and correlates more strongly with human judgments than traditional automatic metrics such as CIDEr and SPICE. CLAIR scores range from to 1, with higher values indicating better caption quality. The results are summarized in Table A.4.
Results on Common VQA Tasks:
On general VQA benchmarks, REVERSE achieves performance comparable to the LaVA-v1.5-77 base- line, showing that the gains on hallucinationnfoousse benchmarks (e.g.. MM-Hal or Haloquest) do not come at the expense of general VLM capabilities. Although HallusionBench is sometimes considered a visual hallucination benchmark, it primarily evaluates robustness to visually misleading inputs rather than language- prior or bias-driven object hallucination that our work explicitly targets. Overall, REVERSE achieves a 300 reduction object hallucination while maintaining comparable caption quality to the baseline.
E
Implementation Details
E.1
Training Details
For both LLaVA-vI.5-77 and LLaVA-MORE, we initialize from pretrained language models (Vicuna- 1.5-7B and 1-ama-3.1-8--Instaae respectively), along with their corresponding visual projectors and CT-P--3TT vision encoders. Following the standard LLLVA setup, we perform instruction finc-tuning using our 1.3M multi-image dataset for one epoch with LoRA (rank 128, 256). We
26

---

Table A.6: Bootstrapped results on MMHal and HaloQuest. 95% confidence intervals are shown as subscripts.
MMHal
HaloQuest
Backbone
Method
Score (1)
Hall Rate (4)
Avg Acc. (1)
IP Aoc. (1)
VC Acc. (1)
Acc. (1)
VLM
2.11
22.6
39.5
10.7
LL 7B
REVERSE REVERSE: -..003)
2.56p23-300
0.7003-000
30.7674-353)
31,8p75-35m
26.9
A.28 3.72)
32.3224-3655 36.5)
29.4251-3577
1877355246 2466
58.8530-7..0
Base VLM REVERSE, 0.000) REVERSE
2,50
0.53
22.4
15.8
43,4
74
LLaVA-MORE
2.28108-280
0.54m44oni
30.0044-152)
31.3p53-000
11.77
36.7
39.5p4s-4mm
30.966-73
38.1m6-w9
Qwen2.5-VL" 3
Base VLM REVERSE 0.01)
2.89
0.43
33.5
25.4
51.6
26.4
3.15
0.29021-040)
45.1 (4009994
42.9
418848-411
55.5444--6500
Table A.7: Bootstrapped results on discriminative hallucination benchmarks. We report means with 95%
confidence intervals.
Backhone
Method
AMBER-D
POPE
MME-Hall
Base M
74.7
85.9
648.3
LLVA--1.55
REVERSE
74.2/24-7822
85.9928-885)
20l.60sssoo--42255 64254
Base VLM REVERSE
71.6
85.1
678.3
LLaVA-MORE 8B
69 69.3
84.4g 657.63
VLM
550.4
Qwen2.5VL 38 REVERSE
85 85.7 8643-43)
589 $89.48g4492-6500
adopt the modified cross-entropy loss defined in subsection 3.2 and train using the A optimizer. The learning rate is set to 2e-5 for the visual projector, and (2e-4, le-4) for the LoRA parameters of LLaVA-v1.55 7BB7 and LLaVA-MORE, respectively. The CLIP backbone is kept frozen. We use a global batch size of 128 with no gradient accumulation. Training takes 24 hours for LLaVA-v1.5-7B and 36 hours for LLaVA-MORE on 8x A100 80GB GPUs using DeepSpeed ZeRO-2.
For Qwen2.5-VL, since the instruction tuning dataset is unavailable, we finetune the released model directly. To enable apples-to-apples comparison, we apply REVERSE finetuning on the same 100k subset used for aLaVA-F.. Unlike the ILaVA variants, we finetune the full 3B Qwen2.5-VI. model (without LoRA) using the same modified cross-entropy loss. We use the AdamW optimizer with learning rate of 5e-5, freeze the CLIP encoder, and set the batch size to 128 with no gradient accumulation. Training takes 3 hours 4x A100 80GB GPUs using DeepSpeed ZeRO-3.
E.2 Full Decoding Algorithm
The full decoding algorithm used in REVERSE is shown in Algorithm 1. For captioning and discriminative tasks, we directly apply this algorithm.
For open-ended tasks such as MMHal-Bench and HaloQuest, as described in section 4, we adopt two-stage decoding process, In the first round, REVERSE runs the standard retrospective resampling algorithm. Since many of these queries contain false premises lack sufficient information, the model expected to abstain from answering and return a blank response. such cases, we initiate a second round of inference with the query modified
q = Q "For this question, please point out the false premises note what information is missing, rather than answering it directly."
This prompting strategy requires no additional training and enables the model to handle underspecified or invalid queries more effectively.
27

---

Describe this image.
LLaVA
OPERA
Woodpacker
REVERSE
The tuu ne small esorgs our
The The www.annnannnne saval aringg our multing alongg more pait near zu tatuua and eppic tng surror ningg.
weking down cubbiirinn wo stoad.
pakke ronormylii in sassar
0.315 2315,0.5700 0.574 wirking glemma
dir aerssssee ceneertl aook
mamstorooii posts by min guid
taright iggt wakurrte worra withing troot
for bod. The intoryyia la poeliosee on
0.0., 0.785 parted meatto
seeg, with mroter cydiag lift,
and avernato
poaticeed on left aldeoll the
de these tove peopla
Sedra et bunntaaacce the
appreactigg acira tave
pinggtccte charming ol
lined with cotélestonn.
the lak ecge of imags, while the sttll
sidect
stmooptans
rener is alad buther took on
the adge the ioogg, accine
Theimegg po wearing
The trapeiaatann women wenting
Thimagg Nestame womertty 457, 0.251,
undi beyye
5.730 8.58
biyylly sowwiing Affiiion loski Sidd.
mast dhe encpost rea to Hooo,
Ska croppad ferie wheie
Sha InvERCT with harms
wc sopeers pett
eocomsii to parting cithe horwa
mospectred redc to lanes33.000 13.0000000000000 2.55,
thehanas, creatin Hemely
ot the onter tie ao
B.14,, 8.0000 whanitoo horwarg
moriii that clavis ripilar lnithe
seauss tame are Aesoniiiaan aoriiaaaa
lodie ond
buent Lan etoodin woreoni perting
bacig round, there are opher
rearde the woman the bicydl.
panser batter Acacipaci can
sdog tear sormnt themr and sonneperrn
momon the kutther maiyy the
aroise the groan cloee
de aunn placed the groand sear
secipecto placed sround scans, edti
the horse,, and the sclone rppears
nammees morern cxctionion
was sone, 1eat, sotter
so za pascaitll mome the
Journyy ora biluril iis, todiga
cheatig paacefuline sarera
Edll B.66ll and en butter
break withire sances
eerovuseend bwa unbo the
ecross ver bwacdd, entooin
E-478, ruset ing echosa ienely
en sureh beadhrews wotter
degli siceed taa cenner oftiee
teeds Then degle pootis and
erdnean opa oipic nad the surry
scnne, with vaddow
Soctine provicig a couant
The beacl sure soad byya
the wew.gggnn 2.00e, Jab,, cdil,
thi testooor grayll lgandd ru
aecovor to beviinnn towarir the
weise paim deiiiiinngg frie
thi ciiut jracie with cleor blkan
water, possiily to cool aupiiee
near udge churihh
any and mocasn
eneeprio antrllyde neturnia
The image features mene otrgget
The imepenteatues man cat mg an
The image Fnatures merr gt 194, 9.007)
The Pre Seal viser rig
dest writng on plece papper
dask, wrtng piece aipaper with
0.905 DUBOK atring deikgS 9.706,
shit
and oppoerss bo fop swel
eocmare to becomadeenteetae Net
DESEED.ATE ATT, B.A. 8.770 boot/weth
stassddd daak inviar
andingg
shings: E.004, 2.000,
add ugro canual iatrasagenn
ettweren scien. deleniiat
6.7633 and adpase bocueed lis
the drei welish in aacotannsizeeciie
and
tharele
Kapiniing thae the men in gin 0,412 0.33..5..00 0,545 with cetter charecter
poesity
mest mottettt bngeane on
tatti wotin resaart scone aggness
festanne of Blasspocpi
The image fasture groas people,
alapi laing pigees a arth
ERaDE ding ployceee a
che and Buiddll boguia
Incldd iig chidion, lding plcydles
hofurrzu 2303 thee pogte indudt lini
puth purmondee gre andi lewe
takig wnw senwey.
The
ineicc the patt while the
and thecioeee tuttee mank The
520A 82zh 8000 tyshee3 J04,
teimett, rokisti gleie surkdy wiitts
radda
mately ride moessemuee
B TI alogg path real dee ocnen)) 3o 601.
mement of aloume und ontiiioiioon activity
on agaiss heau the cacooioo
with
and sics aneryu cnetin forti
t manoollgreenn The el bowe
baci cures The childien ure no
cycttt path therare sn adkiggntthe can pooceik
ware how ma grratt
STAAD appeirs 20 han Srise
Figure A.4: Additional Qualitative Results.
28

---

Algorithm On-the-Fly Retrospective Resampling During Generation
Require:
Input prompt (T,Q) (image and question), maximum total corrections N, local correction threshold K, base temperature To, temperature step =
I: Initialize: Attempt count n+-0, 0, local failures k+ 0, temperature T+,o, placeholder list P
2: Initialize empty sequence
3: while generation is not finished do
4: Generate next token: wt = (LM((
5: Append w, to S
if P(wx) 2 then
Hallucination detected
7:
Identify most recent </CN> as local checkpoint Creal
&:
Append hallucinated phrase wi to placeholder list P while << N do
9:
Inside the correction loop
Backtrack Cheal and apply Rejection Sampling and Query Rewriting
1I:
Update temperature: T f'+miin(T (T'innTT+2 AT, +T,76++0.5
Modify prompt with clarification hint:
Q= +=Q+(r (Hint: potential incorrect phrases P)
13:
Generate resampled phrase: h (LM(((L))+
14:
P(la) <r for all tokens in hı then
b Verify token probabilities
15:
Accept h, and continue generation
16:
Append h, to S
17:
Reset temperature: T<-Ti. Reset failure count: k
18:
Break out of correction loop
else
20:
k4 k+1 end if
> Track consecutive failures
22:
n+1 K then
23:
if k>
Escalate backtracking if local corrections fail Identify last sentence boundary (last punctuation token) as Cokoal Backtrack Cysnal and reset +-0
24:
26:
end if
end while
28:
end if
end while
30:
Return generated sequence S
29


Reviews:
[{"review_id": "fTcjAmB5yX", "reviewer": "Reviewer_PVJe", "summary": "This paper introduces REVERSE (REtrospective VERification and SElf-correction), a framework for reducing hallucinations in Vision-Language Models (VLMs). Unlike existing methods that either adjust generation behavior or verify outputs post-hoc, REVERSE integrates both into a single model. There are two major innovations: (1) hallucination-aware training using a 1.3M semi-synthetic dataset annotated with special tokens that label confident and unconfident phrases, and (2) retrospective resampling, an inference-time mechanism where the model self-verifies outputs and backtracks when hallucinations are likely. The method significantly reduces hallucination across benchmarks like CHAIR-MSCOCO, HaloQuest, and MMHal-Bench, achieving up to 12% and 34% improvement over prior SOTA methods.", "questions": "1. See weakness and please address those.\n2. How do you ensure the diversity of the augmented data for your training? How many negative examples do you create for reach positive examples, and how do you decide how to generate those negative examples? \n3. How do you decide the thresholds? It seems that it's manually tuned? Would that have generalization problems for datasets from different domains/styles/cases?\n4. This method introduces overhead. Can you explain a bit more on that. I feel it's a important issue and should be explained further in the main paper, ideally with a table.", "limitations": "No limitation is discussed, need to address and discuss limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Paper is well-written, in good shape and good quality. Explanation is clear.\n2. The method is novel and explore a novel paradigm that unifies generation adjustment with online post-hoc verification in a single VLM architecture. Also have a small dataset contribution for SFT / instruction tuning.\n3. code is well maintained and made easily available. \n4. Useful insights from the Experimentations.\n\nWeakness:\n1. Missing many essential references / benchmarks to compare with. Please add those additional experimentations.\n\n[1] Mm-vet: Evaluating large multimodal models for integrated capabilities. \n\n[2] Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. \n\n[3] MMHal: Aligning large multimodal models with factually augmented rlhf. \n\n[4] POPE: Evaluating object hallucination in large vision-language models\n\nPlease cite papers and compare the performance on those benchmarks.\n\n2. Limitation section should be expanded.", "was_revised": true, "final_justification": "The author provided detailed explanations and obtained more results and analysis. The rebuttal addresses my concerns and I decide to raise my score.", "rebuttal": "Thank you for acknowledging the contributions of our work and for the valuable feedback. We provide a few clarifications on the raised weakness below:\n\n**[W1: Missing references and benchmarks]**\n\nWe thank the reviewer for pointing out these benchmarks. We would like to clarify that:\n\n1. MM-Hal [3]: Results are already reported in Table 3, where REVERSE achieves up to 10% gains across all VLM backbones.\n\n2. POPE [4] and other discriminative tasks: Results and discussions are provided in Table A.1 and Appendix Section E, where REVERSE is comparable to or better than all reported baselines across different model VLM backbones.\n\n3. Additional benchmarks (requested by the reviewer): We have further evaluated REVERSE on HallusionBench, GQA, and MM-Vet, and the results are as follows:\n\n| Method        | HallusionBench | GQA   | MM-Vet |\n| ------------- | -------------- | ----- | ------ |\n| LLaVA-v1.5-7B | 46.94          | 62.00 | 31.10  |\n| REVERSE       | 45.44          | 62.73 | 28.40  |\n\n\nOn general VQA benchmarks such as HallusionBench, GQA, and MM-Vet, REVERSE achieves performance comparable to the LLaVA-v1.5-7B baseline, showing that our gains on hallucination benchmarks (e.g., MM-Hal) do not come at the expense of general VLM capabilities. Besides, while HallusionBench is sometimes treated as a visual hallucination benchmark, it mainly evaluates performance on visually misleading inputs rather than the type of language-prior or bias-driven object hallucination that our work specifically addresses. Thanks for the valuable feedback, and we will add these results and a corresponding discussion to the final version.\n\n**[Q2: Diversity of the Generated Synthetic Data]**\n\nOur complete data generation pipeline is described in Appendix B. In brief, we generate one negative sample per positive example using a combination of rule-based algorithms and LLM prompting, with a strong focus on diversity and quality: \n\n1. Diverse negative generation. We first classify QA pairs into multiple types. For simple cases (e.g., yes/no or numerical questions), we use rule-based transformations such as flipping answers or generating plausible but incorrect numbers. For complex cases (e.g., long-form, open-ended answers), we employ GPT-4o with the prompt described in Appendix B to produce varied negative examples that differ in both content and phrasing.\n\n2. Quality control. To further improve diversity and prevent artifacts, we skip trivial edits (e.g., pronoun swaps) and re-distribute examples to balance positive/negative order to ensure that the negative examples are non-trivial and informative.\n\nThese steps ensure that our training data covers a wide range of error types while maintaining balance. We will make this clearer in the final version.\n\n**[Q3: Sensitivity of Threshold Tuning]**\n\nWhile the threshold could potentially raise generalization concerns, we would like to clarify that:\n\n1. A single universal threshold works well across domains. For example, a threshold of 0.003 enables REVERSE with LLaVA-series models to achieve SOTA results on AMBER-G, MSCOCO-CHAIR, MM-Hal, and HaloQuest. Similarly, for Qwen-2.5-VL, a threshold of 0.01 yields SOTA performance across these diverse tasks.\n\n2. The threshold provides interpretable control, which is unique to our method. As shown in Figure 5 and Lines 286–296, smaller thresholds (e.g., 0.0001) can significantly reduce hallucination for conservative use cases such as medical QA (even surpassing GPT-4V), while larger thresholds allow greater expressiveness for creative tasks like storytelling. To our knowledge, REVERSE is the only approach that exposes this trade-off through a single, user-adjustable parameter; prior methods effectively fix this threshold implicitly without offering user control.\n\nWe appreciate the reviewer’s feedback on this and will include this clarification in the final version. We also agree that enabling the model to adaptively select thresholds during decoding is a promising direction for future work and will add this to our limitations section.\n\n**[Q4: Detailed Discussions on Inference Overhead due to Iterative Correction]**\n\nThank you for raising this important point. We agree that efficiency is critical and have already discussed it in Lines 297–307 of the main paper. In addition, below we provide a more detailed analysis, including the number of correction rounds, hallucination score (CHAIR), and relative runtime (measured by total generated tokens of REVERSE-v1.5-7B compared to LLaVA-v1.5-7B on the AMBER-G task):\n\n| # Rounds (N)      | CHAIR (↓) | Rel. Gen Tokens |\n| ----------------- | --------- | --------------- |\n| LLAVA-v1.5-7B (0) | 7.8       | (1.00x)         |\n| 5                 | 7.1       | 1.75x           |\n| 10                | 6.8       | 2.05x           |\n| 20                | 6.7       | 2.63x           |\n| 50                | 6.0       | 3.05x           |\n\nThe overhead above can be separated into two primary categories, verification overhead and correction overhead: \n\n**Verification overhead.** Our token-level confidence check is performed **inline during generation** and adds negligible runtime cost. By contrast, prior methods such as Woodpecker and LURE require additional calls to external VLMs or object detectors, which introduce substantial inference overhead.\n\n**Correction overhead.** We further analyze performance across different maximum correction rounds:\n\n1. In prior work, a single correction round roughly doubles runtime (full re-generation), and N correction rounds lead to about **(N+1)× runtime**.\n2. In REVERSE, although we set the maximum correction rounds to 50, **37% of samples require no backtracking**, and among the remaining cases, **over 50% converge within a single round**, keeping the overhead modest. As shown in the table above, additional correction rounds further reduce CHAIR scores while runtime overhead remains **at most ~3× on average**, since corrections are localized rather than full re-generations.\n\nWe will include this table and discussion in the final version. We also view the development of even more efficient correction strategies as a promising direction for future work.\n\n**[W2: Limitations]**\n\nBased on the clarification above, we will include further discussion on sensitivity of threshold tuning and potential runtime overhead to the limitation section."}, {"review_id": "v8vOB9hkgf", "reviewer": "Reviewer_oB2j", "summary": "This work introduces REVERSE (Retrospective Verification and Self-Correction) for mitigating hallucinations in Vision-Language Models (VLMs). Specifically, it leverages a new hallucination-verification dataset with large-scale semi-synthesized samples and proposes a novel inference-time retrospective resampling technique to detect and revise undesired hallucinations. Experimental results validate the effectiveness of the proposed approach.", "questions": "1. What is the average inference latency for REVERSE compared to other approaches? Would efficiency decrease if the generated token length increases?\n2. Could an ablation experiment be provided to show the performance with a restricted number of correction attempts ($N=5$, $10$, or $20$) in resource-limited or real-time applications?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The motivation of this paper is clear and straightforward. The propsoed REVERSE is intuitive and novel as far as I am concerned.\n2. Extensive experimenrtal results across different VLM architectures and benchmarks validate the effectiveness of the proposed approach. The qualitative results are also impressive.\n3. This paper is generally well-written and easy to read. The structure of this paper is clear.\n4. This work provides a new public dataset with 1.3M semi-synthetic samples..\n\n---\n**Weaknesses**:\n1. Multiple correction attempts can incur non-trivial computational cost. Although the authors mention that the latency is controllable in Lines 297-307, I am still concerned about the efficiency. What is the average latency for inferring the 1,004 samples in AMBER-G, and how does it compare to other approaches? I assume the efficiency would further decrease if the generated token length increases.\n2. An ablation experiment on the correction steps N and K should be provided. For example, what would the performance be if the total number of correction attempts is restricted to $N=5$, $10$, or $20$ in resource-limited or real-time applications?\n\n---\n**Minor issues**:\n1. I am curious whether this method can scale to multi-image and video visual inputs. Could the authors provide a discussion on this?\n2. Failure cases, where hallucinations persist even after N total attempts, should be presented to help readers better understand the limitations of the approach.", "was_revised": true, "final_justification": "The authors' rebuttal have addressed all my concerns and I maintain my score to recommend acceptance.", "rebuttal": "Thank you for acknowledging the contributions of our work and for the valuable feedback. We provide a few clarifications on the raised weakness below:\n\n**[W1+2, Q1+2 Discussions on the Inference Overhead]**\n\nThank you for raising this important point. We agree that efficiency is critical and have already discussed it in Lines 297–307 of the main paper. In addition, below we provide a more detailed analysis, including the number of correction rounds, hallucination score (CHAIR), and relative runtime (measured by total generated tokens of REVERSE-v1.5-7B compared to LLaVA-v1.5-7B on the AMBER-G task):\n\n| # Rounds (N)      | CHAIR (↓) | Rel. Gen Tokens |\n| ----------------- | --------- | --------------- |\n| LLAVA-v1.5-7B (0) | 7.8       | (1.00x)         |\n| 5                 | 7.1       | 1.75x           |\n| 10                | 6.8       | 2.05x           |\n| 20                | 6.7       | 2.63x           |\n| 50                | 6.0       | 3.05x           |\n\nThe overhead above can be separated into two primary categories, verification overhead and correction overhead: \n\n**Verification overhead.** Our token-level confidence check is performed **inline during generation** and adds negligible runtime cost. By contrast, prior methods such as Woodpecker and LURE require additional calls to external VLMs or object detectors, which introduce substantial inference overhead.\n\n**Correction overhead.** We further analyze performance across different maximum correction rounds:\n\n1. In prior work, a single correction round roughly doubles runtime (full re-generation), and N correction rounds lead to about **(N+1)× runtime**.\n2. In REVERSE, although we set the maximum correction rounds to 50, **37% of samples require no backtracking**, and among the remaining cases, **over 50% converge within a single round**, keeping the overhead modest. As shown in the table above, additional correction rounds further reduce CHAIR scores while runtime overhead remains **at most ~3× on average**, since corrections are localized rather than full re-generations.\n\nWe will include this table and discussion in the final version. We also view the development of even more efficient correction strategies as a promising direction for future work.\n\n**[Minor Issue 1: Generalization to Video or Multi-Image QAs]**\n\nWhile the current paper focuses on single-image QA using the LLaVA instruction dataset, the method is not limited to this setting. In principle, it can generalize to multi-image QA, video QA, or even pure LLM-based QA by applying the same data augmentation strategy to the corresponding instruction-tuning data and appropriately adjusting the hyperparameters. We will add, and expand on this discussion in our section on future work.\n\n**[Minor Issue 2: Failure cases]**\n\nWe appreciate the reviewer’s suggestion and will include representative failure cases in the final revision (unfortunately, images cannot be included in the rebuttal due to policy restrictions). We note, however, that such failure cases are often detectable by our method. Unlike approaches without an explicit retry mechanism, REVERSE can route these cases to a fallback refusal response (e.g., “I’m sorry, I can’t understand the question”), which helps mitigate the risk of incorrect outputs. We provide preliminary trials of this mechanism in Appendix F.2 for open-ended QA."}, {"review_id": "Jj5jmRXoiC", "reviewer": "Reviewer_XTB2", "summary": "This paper proposed REVERSE, a unified framework for reducing hallucinations in VLMs by combining hallucination-aware training with inference-time retrospective self-correction. The approach incorporates novel tagging tokens for phrase-level confidence estimation and employs a decoding-time backtracking mechanism activated by an uncertainty threshold. REVERSE was evaluated on several benchmarks (CHAIR-MSCOCO, AMBER, HaloQuest, and MMHal), and achieved improvements over prior hallucination mitigation methods.", "questions": "1. Why did the authors only use three backbone models? Prior work like CODE [3] evaluated on six VLMs. What was the motivation behind this choice?\n\n2. Compared to prior datasets created via negative phrase augmentation (e.g., [1], [2]), what exactly is new in your dataset construction method? What advantages does it offer?\n\n3. Could the authors clarify how the negative phrases were selected and inserted during dataset construction? The current description lacks sufficient detail, and it's unclear whether the augmentation was controlled or systematic.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-written, with clear motivation and contributions, and the experiments are well-presented. \n\n2. The REVERSE framework offers a particularly elegant and effective approach to hallucination mitigation in VLMs. Its token-based confidence tagging is intuitive and interpretable, while the retrospective resampling mechanism provides a flexible and practical way to refine outputs without external supervision.\n\n**Weaknesses**\n\n1. There are already many datasets constructed by augmenting negative phrases (e.g., [1][2]). The paper should clearly explain what makes your dataset different from existing datasets.\n\n2. Since the proposed dataset is created by augmenting negative phrases, how these phrases are selected and how augmented is crucial. The authors mention in caption of Fig. 2 that the dataset focuses on attributes, objects, world entities, and novel scenes. However, this is not discussed in depth, and honestly, I wasn’t sure how exactly this augmentation was done. After checking the supplementary, it looks like the negative phrases were simply chosen and replaced by an LLM, which seems quite naïve. In contrast, prior work selects phrases per error category [1] or uses structured methods like graphs [2]. The author should add a deeper discussion by a comparison with theses approaches.\n\n3. In Section 4.1, the paper claims to evaluate image captioning performance, but what’s actually evaluated is CHAIR. This is not the same as evaluating captioning quality. A model could score well on CHAIR while still producing poor captions overall. To make a convincing argument, the authors should evaluate on standard image captioning benchmarks like CIDEr and SPICE, using datasets such as MSCOCO. Evaluation on LLaVA-Bench would also make sense given the vision-language setting.\n\n- [1] : https://arxiv.org/abs/2401.06855\n- [2] : https://arxiv.org/abs/2506.13130\n- [3] : https://arxiv.org/abs/2406.01920", "was_revised": true, "final_justification": "The rebuttal has addressed nearly all of my major concerns.\nIn particular, the authors provided clear and sufficient explanations regarding the data generation procedure, the rationale behind the selection of VLM backbones, and the implementation details of their data generation process.\n\nThat said, I still had a minor concern regarding the image captioning performance.\nIn the CLAIR metric, the proposed method scored 0.7384 compared to 0.7264 for the original model, indicating a slight drop in performance.\nHowever, the authors argue that the method maintains comparable caption quality to the baseline, which I find compelling.\n\nAs a result, I lean toward accepting.", "rebuttal": "Thank you for acknowledging the contributions of our work and for the valuable feedback. We provide a few clarifications on the raised weakness below:\n\n**[W1,2: Details and Discussions on Data Augmentation Procedure]**\n\nOur data-generation process is similar to FAVA [1], which uses an LLM to augment spans of the data, however differs in two significant ways - the first is scale, while FAVA is composed of 35K fine-grained hallucinations, REVERSE's dataset is composed of an order of magnitude more hallucinated samples (625K), allowing for large-scale supervised fine-tuning. The second, is that we indicate spans explicitly in the token stream (with CN and UN tokens), which allows the model to identify the spans automatically within the same general transformer architecture, rather than being required to build on an external classification mechanism. \n\nCompared to ZINA [2] (which is also much smaller at only 20K hallucinations), REVERSE's dataset is much more flexible in the types of hallucinations generated. Since ZINA relies on a graph-based parsing system, only specific hallucinations can be generated, instead of the more open-ended set generated by REVERSE.\n\nWe appreciate the reviewer’s suggestions and will include the above discussions in the appendix and related work.\n\n**[W3: Comprehensive Image Captioning Metrics]**\n\nThank you for the helpful suggestion. To evaluate caption quality, AMBER-G already includes a “coverage” metric that measures how many relevant objects are mentioned in addition to the hallucination metric, CHAIR.\n\nFor MSCOCO, we follow your advice and also report CLAIR [R-1], an LLM-as-a-judge metric that has been shown to correlate better with human judgments compared to traditional automatic metrics such as CIDEr and SPICE. CLAIR scores range from 0 to 1, with higher values indicating better caption quality.\n\n| **Method**    | **CHAIR-i (↓)** | **CHAIR-s (↓)** | **CLAIR (↑)** |\n| ------------- | ---------------- | ------------- | ----------- |\n| LLaVA-v1.5-7B | 15.4             | 50.0          | 0.7384      |\n| REVERSE       | 10.3             | 37.0          | 0.7264      |\n\nThese results show that REVERSE achieves a ~30% reduction in object hallucination while maintaining comparable caption quality to the baseline. We will include these discussions in the final revision.\n\n[R-1] CLAIR: Evaluating Image Captions with Large Language Models\n\n**[Q1: The selection of VLM backbones]**\n\nWe selected the VLM backbones based on two main considerations. First, most prior hallucination-mitigation work has been implemented on **LLaVA-v1.5**, making it the standard baseline for comparison. Second, to ensure relevance to current research trends, we added **LLaVA-MORE** (LLaVA with Llama-3.1) and **Qwen-2.5-VL**, which are among the most widely used and actively discussed open-source VLMs in the community. As the community has largely moved away from early models such as MiniGPT (used in [3]) toward these newer backbones, we chose this set to provide a representative and up-to-date view of the field.\n\n**[Q2,3: Clarification on Our Data Augmentation Process]**\n\nThe full data generation pipeline is detailed in Appendix B. In summary, we generate one negative sample per positive example using a combination of **rule-based algorithms** and **LLM prompting**:\n\n- **Type-aware rules:** We first categorize QA pairs by type. For simple cases (e.g., yes/no, numbers), we apply straightforward rules such as flipping answers or sampling distractors.\n- **LLM-generated negatives:** For more complex cases (e.g., open-ended descriptive answers), we use GPT-4o to generate diverse and semantically plausible negatives, following the structured prompts described in Appendix B.\n- **Quality control:** We remove trivial negatives (e.g., those involving only pronouns) and rebalance positive/negative order to prevent positional bias.\n\nAs a result, the data used for REVERSE is fundamentally different from that used in existing methods such as FAVA [1] and ZINA [2], both in scale (where REVERSE contains 1.3M samples, compared to 35K in FAVA and 20K in ZINA), as well as diversity (since REVERSE’s training data consists of a wide range of hallucination types across attributes, objects, world entities and scenes that go beyond the graph-based replacement framework in prior work)."}, {"review_id": "e5xPsP3IDc", "reviewer": "Reviewer_B5Fh", "summary": "This paper addresses the persistent issue of hallucination in large multimodal models (LMMs), with a focus on vision-language tasks. The authors propose Retentive Decoding (ReD), a plug-and-play method that generates multiple candidate responses and selects the one with the least hallucination likelihood using a dedicated verifier. The verifier is trained on a novel benchmark named MMHal-Bench, which contains hallucination-labeled samples across multiple domains. The approach does not require modifying the backbone LMMs and is designed to generalize across models and datasets. Experimental results show that ReD substantially reduces hallucinations on MMHal-Bench and other standard benchmarks, with minimal impact on task accuracy.", "questions": "**1.Limited Novelty:**\nWhile REVERSE offers a unified framework for hallucination detection and correction, its core ideas are closely aligned with prior works such as LURE, which also leverage hallucination-labeled data and verifier-based revision. The retrospective resampling mechanism resembles existing self-verification strategies like iterative decoding and uncertainty-based filtering. Without clearer conceptual innovations or performance advantages over works like [1–5], the contribution appears incremental.\n\n**2.Incomplete Evaluation on Discriminative Benchmarks:**\nAlthough Table A.1 includes aggregate F1 scores, the paper lacks detailed analysis on discriminative hallucination datasets such as POPE and OPOPE. Full task-level breakdowns are essential to validate the claimed improvements and better understand where the method succeeds or fails.\n\n**3.Missing Results on Standard LVLM Benchmarks:**\nThe method is not evaluated on widely used benchmarks such as MMBench, MME, POPE, SEED, and MM-Vet. These datasets are important for assessing hallucination and reasoning in diverse scenarios. Their omission limits understanding of REVERSE’s generalizability and practical value.\n\n**4.No Efficiency Analysis:**\nThe proposed method involves backtracking, resampling, and rewriting, all of which may incur significant inference overhead. However, the paper does not report latency, decoding steps, or computational cost. Comparing efficiency with prior methods like LURE would help clarify the trade-off between performance gains and runtime complexity.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n**1.Motivation:**\nThe paper provides a clear and relevant motivation. Hallucinations in LMMs, particularly in high-risk applications, remain a critical issue, and existing methods either lack robustness or are computationally costly. The authors effectively argue for a verification-based approach.\n\n**2.Writing Quality:**\nThe writing is generally clear and logically organized. Figures and tables are well-integrated to support the narrative. The explanation of both the hallucination problem and the verification process is intuitive and technically solid.\n\n**Weaknesses:**\n\n**1. Lack of Novelty:**\nWhile the proposed REVERSE framework presents a unified approach to hallucination detection and correction, its core ideas closely resemble prior work such as LURE[1], which also constructs hallucination-labeled data and trains a verifier to revise model outputs. The introduced retrospective resampling mechanism, although framed as novel, largely builds upon well-established on-the-fly self-verification techniques, including iterative decoding and uncertainty-based correction. As a result, the methodological contribution appears incremental. Without clearer conceptual distinctions or empirical advantages over prior works [1–5], the paper lacks sufficient novelty. \n\n[1] ANALYZING AND MITIGATING OBJECT HALLUCINATION IN LARGE VISION-LANGUAGE MODELS. ICLR 2024\n\n[2] Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models. ACL Findings 2024\n\n[3] VOLCANO: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision. NAACL 2024\n\n[4] HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding. ICML 2024\n\n[5] Detecting hallucinations in large language models using semantic entropy. Nature 2024\n\n**2. Incomplete Evaluation on Discriminative Hallucination Datasets:**\nThe paper does not provide a complete evaluation on discriminative hallucination benchmarks such as POPE or OPOPE[1]. While Table A.1 reports an aggregate F1 score, it lacks task-specific breakdowns or deeper analysis. Comprehensive evaluation on these benchmarks is essential for validating the claimed effectiveness of the proposed approach.\n\n[1] HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding. ICML 2024\n\n**3. Missing Results on Popular LVLM Benchmarks:**\nThe paper omits performance evaluations on several widely adopted LVLM benchmarks, such as MMBench, MME, POPE, SEED, and MM-Vet. These benchmarks are frequently used to assess hallucination and reasoning in real-world tasks. Without results on these datasets, it remains unclear how well REVERSE generalizes and whether it outperforms or underperforms compared to other approaches across standard benchmarks.\n\n**4. Lack of Efficiency Analysis:**\nAlthough the proposed method shows improvements in hallucination mitigation, it introduces additional computational overhead due to iterative backtracking, rejection sampling, and query rewriting. However, the paper does not report inference latency, decoding steps, or overall computational cost. A comparison with baselines such as LURE or other post-hoc correction methods is needed to assess practical trade-offs between hallucination reduction and runtime efficiency.", "was_revised": true, "final_justification": "After carefully reading the authors' rebuttal, I'm satisfied with the responses to all my concerns.\n\nThe authors have provided a compelling clarification on the novelty of their approach. They successfully distinguish REVERSE from prior work by highlighting its unique ability to unify on-the-fly generation, verification, and correction within a single VLM using explicit confidence tokens. This unified, token-level self-correction mechanism is a clear and notable contribution that was not fully appreciated in my initial review.\n\nFor the incomplete evaluations, the authors have provided a thorough breakdown of their performance on discriminative datasets like POPE, as well as on general LVLM benchmarks like GQA and MM-Vet. The new data confirms that REVERSE maintains performance on these benchmarks while improving hallucination reduction, addressing my concern about its generalization capabilities.\n\nFinally, the detailed efficiency analysis is highly informative. The authors' clarification that their token-level verification adds negligible overhead and that corrections are localized rather than full re-generations is crucial. The provided table effectively demonstrates the trade-off between the number of correction rounds and the runtime overhead, which appears to be quite reasonable for the performance gains achieved.\n\nGiven these excellent clarifications and additional data, I'm increasing my score from \"borderline reject\" to \"borderline accept.\" The authors have successfully addressed all my major concerns and have significantly strengthened the paper.", "rebuttal": "Thank you for acknowledging the contributions of our work and for the valuable feedback. We provide a few clarifications on the raised weakness below:\n\n**[W1: Justification on REVRESE’s Novelty]**\n\nWhile we appreciate the reviewer’s related work suggestions, we would like to clarify that **REVERSE is methodologically distinct from [1–5]**. Specifically, [1, 2, 4] perform multi-stage verification or refinement **after full text generation**, relying on separate modules such as external VLMs or object detectors. [3] applies self-correction through multi-round conversations only after the initial generation. [5] focuses on on-the-fly hallucination detection but operates purely in the LLM space without any correction mechanism.\n\nIn contrast, **REVERSE unifies generation, verification, and correction within a single VLM**, enabling **on-the-fly correction during decoding** without completing the initial output. Moreover, REVERSE introduces **explicit confidence tokens**, which provide interpretable, token-level self-correction, which is an approach not explored in prior work.\n\nThis unified generative-verification framework, highlighted as novel and well-motivated by reviewers PVJe, oB2j, and XTB2, leads to **SOTA performance across multiple VLM hallucination benchmarks** (Section 4), demonstrating clear advantages over [1–5].\n\n**[W2: Breakdown Analyses on Discriminative Hallucination Tasks]**\n\nThank you for the suggestion. We would like to clarify that POPE is a benchmark that focuses exclusively on yes/no questions in a discriminative setting. As for “OPOPE,” we were unable to identify this benchmark in the literature and would appreciate clarification from the reviewer on its definition or source.\n\nAs noted in Lines 225–229, our paper primarily focuses on generative hallucination tasks (for example, image captioning), where the space of possible outputs is large and REVERSE's backtracking and self-correction are most beneficial. For discriminative tasks, where answers are typically binary, we have already reported aggregate results in Appendix E. Below, we now provide the official task-wise breakdown for POPE (average F1 score and 95\\% CI for all sub-categories), which shows only minor differences between REVERSE and the baseline LLaVA-v1.5-7B. While we include this breakdown for completeness, we emphasize that these results are less indicative of REVERSE's core contributions, which target open-ended generation rather than binary classification.\n\n| Model         | Popular          | Adversarial      | Random           | ALL              |\n| ------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| LLaVA-v1.5-7B | 86.1 (84.8–87.4) | 84.2 (82.8–85.7) | 87.2 (85.8–88.5) | 85.9 (83.2–88.2) |\n| REVERSE       | 86.3 (85.0–87.7) | 83.9 (82.4–85.3) | 87.5 (86.3–88.8) | 85.9 (82.8–88.5) |\n\n**[W3: Additional Results on General LVLM Benchmarks]**\n\nAs also noted by the reviewer, we include results for POPE, AMBER-D, and MME-Hall in Appendix E (Table A.1). We now additionally report results on GQA and MM-Vet, where REVERSE is comparable to LLaVA-v1.5-7B. These findings confirm that the improvements in hallucination reduction do not come at the expense of performance on standard LVLM benchmarks. We will include these additional results and discussions in the final revision. However, we would like to re-emphasize that our primary focus is on open-ended VQA and generative hallucination tasks. For these, we already report results on five generative benchmarks in the main paper and three discriminative benchmarks in the appendix, evaluated across three different VLM backbones, which we believe provide a thorough evaluation of our approach.\n\n| Model         | GQA   | MM-Vet |\n| ------------- | ----- | ------ |\n| LLaVA-v1.5-7B | 62.00 | 31.10  |\n| REVERSE       | 62.73 | 28.40  |\n\n**[W4: Discussion on Inference Overhead]**\n\nThank you for raising this important point. We agree that efficiency is critical and have already discussed it in Lines 297–307 of the main paper. In addition, below we provide a more detailed analysis, including the number of correction rounds, hallucination score (CHAIR), and relative runtime (measured by total generated tokens of REVERSE-v1.5-7B compared to LLaVA-v1.5-7B on the AMBER-G task):\n\n| # Rounds (N)      | CHAIR (↓) | Rel. Gen Tokens |\n| ----------------- | --------- | --------------- |\n| LLAVA-v1.5-7B (0) | 7.8       | (1.00x)         |\n| 5                 | 7.1       | 1.75x           |\n| 10                | 6.8       | 2.05x           |\n| 20                | 6.7       | 2.63x           |\n| 50                | 6.0       | 3.05x           |\n\nThe overhead above can be separated into two primary categories, verification overhead and correction overhead: \n\n**Verification overhead.** Our token-level confidence check is performed **inline during generation** and adds negligible runtime cost. By contrast, prior methods such as Woodpecker and LURE require additional calls to external VLMs or object detectors, which introduce substantial inference overhead.\n\n**Correction overhead.** We further analyze performance across different maximum correction rounds:\n\n1. In prior work, a single correction round roughly doubles runtime (full re-generation), and N correction rounds lead to about **(N+1)× runtime**.\n2. In REVERSE, although we set the maximum correction rounds to 50, **37% of samples require no backtracking**, and among the remaining cases, **over 50% converge within a single round**, keeping the overhead modest. As shown in the table above, additional correction rounds further reduce CHAIR scores while runtime overhead remains **at most ~3× on average**, since corrections are localized rather than full re-generations.\n\nWe will include this table and discussion in the final version. We also view the development of even more efficient correction strategies as a promising direction for future work."}]

Author rebuttal:

Meta-review: The paper proposes **REVERSE (REtrospective VERification and SElf-correction)**, a unified framework aimed at reducing hallucinations in Vision-Language Models (VLMs). Unlike prior methods that either adjust generation behavior or apply verification post-hoc, REVERSE integrates both processes into a single architecture. The core innovations include (1) hallucination-aware training using a 1.3M semi-synthetic dataset with special tokens indicating phrase-level confidence, and (2) a novel inference-time *retrospective resampling* mechanism that backtracks and revises output when hallucinations are detected. The model is evaluated on benchmarks such as CHAIR-MSCOCO, MMHal-Bench, and HaloQuest, achieving notable improvements over prior SOTA approaches.

**Strengths:**
- Clear motivation and strong problem formulation addressing hallucinations in VLMs.
- Unified and novel integration of generation, verification, and correction within a single model.
- Introduction of a new, publicly released 1.3M semi-synthetic dataset to support hallucination-aware training.
- Elegant and interpretable token-level confidence tagging mechanism.
- Demonstrated improvements over baselines across multiple benchmarks, with insightful qualitative and quantitative results.
- Codebase is clean, accessible, and supports reproducibility.
- Concerns about efficiency were addressed with a detailed latency analysis showing reasonable overhead.
- Rebuttal effectively clarified novelty, evaluation depth, and implementation details.

**Weaknesses:**
- Initial comparisons lacked key benchmarks such as MM-Vet, POPE, and HallusionBench; reviewers requested additional evaluations and citations.
- Dataset construction method was seen as overly simplistic compared to structured alternatives; more detailed methodology and comparisons were needed.
- Evaluation metrics like CHAIR alone may not reflect true captioning quality—CIDEr, SPICE, and others were suggested.
- Some concerns about computational overhead from repeated correction steps and efficiency in longer output sequences.
- Limited initial discussion on generalization to more diverse modalities (e.g., video or multi-image inputs).
- Methodological novelty was questioned by one reviewer, who noted similarities to prior work like LURE and other verifier-based frameworks.

All concerns have been addressed by the authors during the rebuttal period, with all reviewers satisfied with the response. I am therefore recommending acceptance.
Decision: Accept (poster)