Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Authors: Linxi Zhao
Yihe Deng
Weitong Zhang
Quanquan Gu
Venue: ICML 2025 spotlightposter (2025)
Keywords: Large Vision-Language Models
Object Hallucination
Multi-modal LLMs
Primary area: deep_learning->large_language_models

Abstract: The advancement of Large Vision-Language Models (LVLMs) has increasingly highlighted the critical issue of their tendency to hallucinate non-existing objects in the images. To address this issue, previous works focused on using specially curated datasets or powerful LLMs to rectify the outputs of LVLMs. However, these approaches require either costly training or fine-tuning, or API access to proprietary LLMs for post-generation correction. In response to these limitations, we propose Mitigating hallucinAtion via image-gRounded guIdaNcE (MARINE), a framework that is both training-free and API-free. MARINE effectively and efficiently reduces object hallucinations during inference by introducing image-grounded guidance to LVLMs. This is achieved by leveraging open-source vision models to extract object-level information, thereby enhancing the precision of LVLM-generated content. Our framework's flexibility further allows for the integration of multiple vision models, enabling more reliable and robust object-level guidance. Through comprehensive evaluations across 5 popular LVLMs with diverse evaluation metrics and benchmarks, we demonstrate the effectiveness of MARINE, which even outperforms existing fine-tuning-based methods. Remarkably, it reduces hallucinations consistently in GPT-4V-assisted evaluation while maintaining the detailedness of LVLMs' generations. We release our code at https://github.com/Linxi-ZHAO/MARINE.

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Linxi Zhao 1
Yihe Deng" *2 Weitong Zhang²
Quanquan Gu 2
Abstract
1 Introduction
The advent of Large Language Models (LLMs) has moti- vated advancements in extending their remarkable capabil- ities to multimodal data. Grounded in the development of pre-trained dision-language models (Radford et al., 2021; Jia et al., 2021; Alayrac et al., 2022) that align visual and textual embedding spaces, Large Vision Language Models (LVLMs) have gained substantial attention in both archi- tectural development (Liu et al., 2023d; Zhu et al., 2023; Ye al., 2023: Dai al., 2023a; Gao et al., 2023), align- ment (Yu et al., 2024; Zhou al., 2024; Deng al., 2024) and benchmarking datasets (Xu et al., 2023; Lu et al., 2024; Zhang et al., 2024a). However, similar to the hallucination issues in textual LLMSs (Ji et al., 2023), where irrelevant con- tent is generated with input prompts, LVLMs face a specific challenge known as object hallucination: generating non- existing objects for given image (Li et al., 2023b; Wang et al., 2023b; Zhou et al., 2023; Fu et al., 2023; Lovenia et al., 2023; Jing et al., 2023). Such problem is particularly concerning as it compromises the model's accuracy and re- liability, especially considering the growing application of LVLMs to safety-critical downstream tasks such as medical imaging (Chambon et al., 2022; Bazi al., 2023).
The advancement of Large Vision-Language Mod- els (LVLMs) has increasingly highlighted the critical issue of their tendency to hallucinate non-existing objects in the images. To address this issue, previous works focused on using spe- cially curated datasets or powerful LLMs to rec- tify the outputs of LVLMs. However, these ap- proaches require either costly training or fine- tuning, API access to proprietary LLMs for post-generation correction. In response to these limitations, we propose Mitigating hallucinAtion via image-gRounded guIdaNcE (MARINE), a framework that is both training-fres and APl-free. MARINE effectively and efficiently reduces ob- ject hallucinations during inference by introduc- ing mmage-groundee guidance to LVLMs. This is achieved by leveraging open-sourec vision mod- els to extract object-level information, thereby enhancing the precision of LWLge-geeratee con- tent. Our framework's 's ficxibility further allows for the integration of multiple vision models, enabling more reliable and robust object-level guidance. Through comprehensive evaluations across 5 popular LVLMs with diverse evalua- tion metrics and benchmarks, we demonstrate the effectiveness of MARINE, which even out- performs existing fine-tuning-based methods. Re- markably, it reduces hallucinations consistently in 4PT-4Vaassisted evaluation while maintain- ing the detailedness of L.L.M' generations. We release our code at htttp:://gituub.com Linxi - -H--/ SCHAO/MARRO /MARINE.
In response the pressing issue of object hallucinations in LVLMs, early attempts (Liu et al., 2023a;b; Gunjal et al., 2023; Wang et al., 2023a) focused on addressing the bias by curating high-quality datasets for fine-tuning or leveraging advanced GPT queries (Yin et al., 2023), such as GPT-4, to the generated captions. However, these meth- ods can be infeasible to implement. For instance, creating extensive, high-quality datasets for fine-tuning L.VLMs is costly and requires significant human annotation. Addition- ally, relying on advanced GPT models for post-processing is expensive and can raise privacy concerns, especially in sen- sitive fields like medical imaging. Most importantly, these approaches do not address the intrinsic causes of object hallucination in LVLMs.
'Equal contribution. 'Department of Computer Science, Cor- nell University, Ithaca, NY, USA 'Department of Computer Sci- ence, University of California. Los Angeles, CA. USA "Scoool of Data Science and Society, UNC, Chapel Hill, NC, USA. Corre- spondence to: Quanquan Gu <qgu@cs.ucla.cdu>.
In this paper, we investigate the intrinsic causes of object hallucination in LVL.Ms. Specifically, these deficiencies may stem from the three main components of the LVLMs: 1) insufficient visual context provided by the visual en- coder (Zhang et al., 2023b), 2) distortion or loss of visual information during the projection from vision to text space,
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Alignment (LVLM)
Task:
Describe this image in detail,
Vision Encoder (LVLM)
Tokenizer
Input Image
000000 00000
This image shows
focusing on the
visible objects
Guidance Model
In this image
LLM Decoder
keyboard
magazine: 0.7
mouse::
maggaznee
and
Guidance Model
y log(
telephone: (I-7) log(
magazine:
cup: magazine
Image- Grouaded Guidance
cup:
cup:
telephone:
Vision Toelbux
Control guidance in the lugit space
Figure 1. Illustration of MARINE framework, which introduces vision toolbox with one or multiple guidance models to enrich the visual context of the original LVLM. The output logits are controlled to place more importance on the guided generation with the guidance strength y.
and 3) inherent hallucinations common in general language models. To address the first two -VpM-fcccific causes, we introduce Mitigating hallucinAtion via image-g mnage-gRounded guIdaNcE (MARINE). MARINE mitigates hallucination is- sues arising from the visual encoder and information distor- tion during cross-modal alignment by leveraging external guidance from image- gaouge-goonded models, such as object de- tection models. Our approach leverages the inherent advan- tage of these image-grounded models, which are specifically designed and trained for more detailed visual information ex- traction. These models provide higher quality, fine-gra ane-grained visual encoding compared to the standard visual encoders in LVLMs, which are primarily optimized for grasping the overall context of an image. Furthermore, we integrate the models into text descrip-
results demonstrate that, comparison with state-of- the- art algorithms, MARINE exhibits further reduced hallucina- tion, as measured by popular hallucination metrics such as CHAIR (Rohrbach al., 2018) and POPE (Li et al., 2023b), as well as GPT-4V's evaluation. These results confirm that MARINE can effectively mitigate object hallucinations with- out requiring additional training resources or access to pro- prietary LLMs. To summarize, our contribution are listed as follows:
We introduce MARINE, a universal framework and ag- gregating toolbox of image-gron grounded visual models to guide the generation process of LVLMs. MARINE lever- ages the intrinsic advantages of these visual models in providing the detailed information of the input image and help mitigate the hallucinations in LVLMs.
guidance from mmage-groundee tions, allowing the LVLM to process the information with-
Through extensive evaluations on various datasets, we demonstrate that MARINE consistently outperform the baselines in hallucination mitigation while maintaining overall performance across multiple tasks (image caption- ing. VQA).
out requiring additional alignment procedures. As a result, MARINE is training-free, API-free method that addresses object hallucination inference time by targeting its two root causes.
As shown in Figure 1, MARINE incorporates one or more image-gran groge-igounding models to enrich the visual con- text of LVLMs. The guidance are then aggregated as prompt input the LLM decoder to improve the re- sponse quality. Empirical evaluations are conducted on five widely-recognizer LVLMs across benchmarks including MSCOCO (Lin et al., 2014), LLaVA-QA90 task (Liu et al., 2023d), A-OKVQA (Schwenk et al., 2022), and GQA (Hud- son & Manning, 2019). We present results based on guid- ance from a aggregated source DEtection TRansformer (DETR) (Carion al., 2020) and RAM++ (Huang et al., 2023b). We also include ideal results based on ground truth object oracle, denoted MARINE-THue -ruth.. Our experimental
MARINE provides a favorable trade-off between latency and accuracy, with the lowest computational overhead compared to existing baselines, which positions MARINE as practical and scalable solution for real-world applica- tions without significant computational cost.
2 Related Work
2.1
Object Hallucination in Large Vision-Language Models
The hallucination issue in Large Vision-Language Models (LVLMss) (Liu et al., 2023d; Zhu et al., 2023; Ye et al., 2023; Dai et al., 2023a; Gao et al., 2023) has drawn significant attention, as highlighted by studies (Li al., 2023b; Wang
2

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
et al., 2023b; Zhou et al., 2023; Fu et al., 2023; Lovenia
nations.
et al., 2023). Notably, different from textual LLMs,, LVLMs are prone to a unique type of hallucination called 'object hallucination' (Rohrbach et al., 2018), where the model falsely perceives the presence of non-existent objects in images. Efforts to address this problem in LVLMs include fine-tuning approaches using vision-language datasets (Liu et al., 2023b; Gunjal et al.. 2023), as well as GPT-assisted methods such those by Zhai et al. (2023). Notably, Yin et al. (2023) proposed training-free approach using GPT- 3.5 for hallucination correction.
3 Preliminaries
Generative language models. Let po denotes LLM pa- rameterized by 0. Consider a sequence x = [ Iv] [ as the imput prompt, where each 2, is a token from a predefined vocabulary. The L.M then generates the response sequence y = [yI,.... .mm] by sampling from the conditional proba- bility distribution po(-Ix), where 3/t denotes individual token for S t S m. The conditional distribution pe [x)x can therefore be expressed as po x) = I po Po y<)) where Y< = [V,, W,,..... Vt-1] for / > 1 and is empty for t = 1. In the case of LVLMs, visual tokens v = [uI,..., mk]] are additionally included. These tokens are generated from a pre- pre-trained visual encoder and mapped into the token space through linear projection. The conditional distribution of output y given the visual tokens and textual prompt is expressed as po py (yly,, x) = IIe= Po (y: V,, Vcc),, where Pe is approximated by LVLMs
Concurrently, Leng et al. (2023) introduced Visual Con- trastive Decoding (VCD), a technique that applies noise to image inputs and penalizes logit outputs of these cor- rupted images. Huang al. (2023a) enhanced beam-search decoding with the Over-trust Penalty and Retrospection- Allocation Strategy (OPERA), which penalizes over-trust and refines token selection based on previous outputs. HALC (Chen et al., 2024) employs adaptive focal-contrast decoding to encourage LVLMs to focus on line-grained vi- sual information, while using a computationally intensive beam search algorithm. In addition, BRAVE (Kar et al., 2024) introduces new architecture that combines features from multiple vision encoders. While not directly targeting hallucination, it shares the key insight of leveraging diverse visual signals to improve grounding.
Guidance in generative models. The process of guided generation involves getting the output y conditioned on imput x, which encodes the desired properties of the output y. This guidance can be generally added to the model by two distinct approaches: classifier guidance (Dhariwal & Nichol, 2021) and classifier-fre: guidance (Ho & Salimans, 2021). As a top-level view, both methods formulate the conditional probability distribution of output y conditioned on guidance x as
2.2 Controllable Generation
Controllable text generation (Prabhumoye et al., 2020; Hu & Li, 2021; Zhang et al., 2023a) has emerged as a vital research domain, focusing on the generation of natural sen- tences with controllable attributes such as persona (Prab- humoye et al., 2020; Hu & Li, 2021: Zhang et al., 2023a) and politeness (Niu& & Bansal, 2018; Madaan al., 2020). Among the various approaches, fine-tuning has been rec- ognized as the most straightforward approach, achieved either through full fine-tuning (Li & Liang, 2021; Ouyang et al., 2022; Carlsson et al., 2022) or integrating tunable adaptors (Lin et al., 2021; Ribeiro et al., 2021). While fine- tuning has been effective in a wide range of applications, it is also expensive in computation as the size of L.LSs is growing tremendously. Recently, there has been a develop- ment on controllable generation with diffusion models (Li et al., 2022; Lin et al., 2023b), extending to controllable text-to-image generation (Yang et al., 2023). Particularly. the use of classifier guidance (Dhariwal & Nichol, 2021) and classifier-fre guidance (Ho & Salimans. 2021) has be- come prominent in refining the quality of generated outputs. Most recently, Sanchez et al. (2023) applied classifier-fre guidance to language models in the single-modal setting to improve their performance at inference time. Our ap- proach methodologically resembles classifier- free guidance for LVLMs' text generation, while specifically addressing the multi-modal context and focusing on reducing halluci-
p(y X po(y)p(xly))
(3.1)
where py is the original generative model and p(xly) is the posterior distribution of x given y and y is the guidance strength. In the classifier guidance, the posterior distri- bution p(x in (3.1) is replaced by classifier (xly)) parameterized by Φb which requires additional training step and calculating Vx log (xly).. The classifier-fres free guid- ance, on the other hand, removes the necessity of the pa- rameterized classifier fa- Instead, according to the Bayes rule, the posterior distribution can be approximated by po (xly) xx po (ylxx)//p(5)
where po(y|x) is the gener- ative model when taking x as prompt input. Plugging this back into (3.1) yields the guided distribution that can be approximated by
x)())x)) Po x pe (y))-1
(y))) & poly).
po(y)7
As result, the guided LLM Po places more importance on the prompt during generation with the increasing value of 7, thereby producing texts that better align with the desired behavior from the prompt (Sanchez et al., 2023).
4 Method
The existing architecture of LVLMs is composed of visual encoder, visual and textual domain alignment layer, and
3

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
the LLM itself. Therefore, besides the inherent language pri- of LLMs (Biten et al., 2022), object hallucination may arise from (I) deficiencies in the visual encoder provide insufficient visual information (Zhang et al., 2023b) and (2) distortion or loss of visual information during the projection from vision to language space. To mitigate object hallu- cinations, we introduce MARINE, a framework containing two major components address the previous challenges: (1) introducing additional visual information from set of vision models and (2) using the additional aggregated visual features to guide the LVI.M's generation. In Figure 1, we present the framework overview.
I-th token is therefore sampled from the logit space by
c, x, Tog v,, c,,, = log (, generation on the additional image-grounded guidance acts as a controllable gate. Only objects with relatively high probabilities in both branches could appear at top when sampling. Specifically, setting y recovers the origi- nal LLM generation without control guidance and setting y produces the LLM generation entirely based on the control. Meanwhile, for E (0, 1), MARINE yields a combination of the original generation pe (y v. and the generation conditioned on the guidance po (y/v, /V, c, x). This strikes a balance between a better ability to follow instruc- tions to generate high-quality answers and the increased accuracy and detail image descriptions. The formulation therefore shares resemblance to the classifier-free guidance introduced for LLMs (Sanchez et al., 2023), which places importance on the textual prompt itself to better align the LLM generation with user intention in the single-modal setting. We summarize MARINE in Algorithm 1. In detail, NEE aggregates the collected visual information {c:), ) using function Aggr., which can be small language model for information aggregation (Lin et al., 2023a).
+ (1 - 7) log (y|v,, x, y
This linear combination of logits implies that the conditional
4.1 Visual Guidance from Image-Grounded Features
To introduce image-grounder guidance to mitigate hallu- cinations, our approach integrates additional object detec- tion models, which differ from the visual encoders used in LVLM that are usually pre-trained from CLIP (Rad- ford et al., 2021). This integration leverages object de- tection models to extract detailed visual information from images. Upon acquiring extra visual information from dif- ferent grounded models, we aggregate and translate the collected information into textual information. This ag- gregation can be done by the language model (Lin et al., 2023a) or rule-based algorithm (Bird et al., 2009). Such an information aggregation is effective and efficient, as it climinates the necessity of fine-tuning the alignment layer while retaining the rich information encoded by various of image grounding models. We subsequently employ a sim- ple prompt "focusing on the visible objects in this image:" and concatenate it with the aggregated object information, denoted as the guidance prompt c.
Algorithm Mitigating hallucinAAion via Image-gRoundee guldaNcE (MARINE)
1: Input: LLM parameter 0, input prompt x, visual tokens
v from LVLM's 's original vision tower
2: Input: auxiliary visual tokens (c, M from M image
grounding models, guidance scale y
4.2 Guided Text Generation with Visual Information
3: Initialize empty output y = 0]-
4: Aggregate visual information as textual prompt c
We tackle the object hallucination problem of LVLMs by placing importance additional grounded informa- tion. In addition to the visual tokens v extracted from the original LVLM and textual prompt x, we extract the auxil- iary visual tokens c from the additional guidance models. The generation of the 1-th token in the output y of our classifier-free guided LVLM po is expressed as
| 5: for L = 0, I,.,,. do
6: Construct unconditional input X = [v. x. yel.
7: Generate unconditional output logits using LLM:
g(t)
uncond log poggpo(2 ( (i2cond)
8: Construct conditional input xondl = [v, c, x, ye..
(4)
Generate conditional output logits using LLM: g(t)
ye, Vo(,)V.x.y<.)>-11
po(vIlvc,x,yedde &
(4) log pa (xod))
10: Update output logits g(t) = e + (1 - 20
uncond 11: Sample token ye from logit space denoted by (())
where e denotes our control guidance and y is the control strength. The sampling of output generation is given by
12: Let y = [y, v.]-
13:
end for
folylv,c, c, x) = ImmPooun (, yer)
14: Output: y.
x [m poiu c.). pe(uilvx..ee) poly|v,c,x)"
5 Experiments
In this section, we evaluate MARINE in mitigating object hallucinations across various LVLMs, showing that it out- performs state-of-the-as methods on established metrics across different question formats.
po V, x)
We can further view MARINE in the logit space, where the
4

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
5.1 Experiment Setup
GPT-4V-aided Evaluation (Yin et al., 2023). The GPT- 4V-aided evaluation compares the outputs of two LVLM assistants using GPT-4V as judge. In this evaluation, we utilize the LLaVA-QA90 task (Liu et al., 2023d) (in- cluding conversations, visual perceptions, and complex reasoning tasks) and additionally consider the image cap- tioning task.
Models. To demonstrate the broad applicability of our approach across different LVLM architectures, we apply and evaluate MARINE to widely-used models including LLaVA (Liu et al., 2023d), LLaVA-v1.5 (Liu et al., 2023c), MiniGPT-v2 (Chen et al., 2023), mPLUG-Owl2 (Ye et al., 2023) and InstructBLIP (Liu et al., 2023c). To address the object hallucination problems in text generation, we incor- porate the DEtection TRansformer (DETR) (Carion al., 2020) and RAM++ (Huang et al., 2023b) as the additional vision models for guidance.
Consistent with Li et al. (2023b), we randomly sampled subset of 500 images from MSCOCO (Lin et al., 2014) dataset for CHAIR evaluation. For the POPE evaluation, we created 3000 questions across three datasets-500 images each from MSCOCO, A-OKVQA (Schwenk et al., 2022), and GQA (Hudson Manning, 2019). For the GPT-4V- aided evaluation, we utilized 90 questions from the VLaVA- QA90 task and randomly selected 50 MSCOCO images for image captioning task.
Guidance from multiple sources. Our framework's com- patibility with various vision models allows for the incor- poration of multiple sources to enhance precision and ro- bustness. By considering object-level information from DETR and RAM++ simultaneously, we generate guidance that reflects consensus across these models. This approach significantly improves the accuracy and reliability of the guidance provided the LVLM.
Baselines. In addition to comparing with the performance of the original LVLM sampling method, we also consider the following popular methods for mitigating hallucinations.
Datasets and evaluations. In alignment with established evaluations from previous studies (Dai et al., 2023b; Yin et al., 2023), we assess our method using the following metrics:
Greedy-Decoding which adopts the greedy sampling strategy, by generating tokens with the highest posterior probability to address hallucinations arising from.
LURE (Zhou et al., 2023), which identifies and masks potentially hallucinated words and fine-tune a MiniGPT4 model to rectify object hallucinations in the generated descriptions.
Caption Hallucination Assessment with Image Relevance (CHAIR) (Rohrbach et al., 2018). It involves prompting the LVLMs to generate description for the input im- age, and then comparing this generation with ground truth
Woodpecker (Yin et al., 2023), which leverages GPT-3.5 to correct hallucinations in LVLM generation with five steps toward the correction.
objects present in the image. CHAIR quantifies hallucina-
tion both at instance level and sentence level, respectively defined as CHAIR, and CHAIRs:
VCD (Leng et al., 2023), which distorts the image inputs to impose penalties on logit outputs.
(hallucinated objects}|
OPERA (Huang et al., 2023a), which penalizes logits to mitigate over-trust in beam-search decoding and adjusts token selection.
CHAIR,
[all mentioned objects}|
CHAIR [captions with hallucinated ]bjects}|
Lastly, the performance of MARINE improves in correlation with the advancement of the control guidance extractor used. Consequently, to demonstrate the potential upper bound of MAR ANE'S''s s performance, we consider a version utilizing a ground-truth oracle extractor, which we denote as MARINE- Truth. Further details on model architectures, datasets and evaluation metrics are deferred to Appendix A.
{all captions]
In addition these metrics, incorporate instance- level Recall score in our evaluation to evaluate whether the descriptions accurately include the necessary visual content from the image:
Recall (non-hallucinated objects}
[all existing objects]
Hyperparameter setting. The hyperparameters for our method are fixed across tasks, with key settings including a guidance strength of 0.7, score threshold for DETR 0.95, detection threshold for RAM++ of 0.68, and greedy sampling approach with random seed of 242.
Polling-based Object Probing Evaluation (POPE) (Li al., 2023b). POPE formulates a binary classification task by prompting LVLMs with questions such as "Is there a keyboard in this image?" to answer "yyss" or "no". We specifically focus on the adversarial setting, which is considered the most challenging setting. Results for the random and popular settings are detailed in Appendix C.
5.2 Results
Experimental results on object hallucination metrics (CHAIR and POPE) are presented in Table I and 2. Overall, MARINE achieves superior performances across different LVLM architectures and evaluation metries.
We report the accuracy and Fl score of the LVLMs' re- sponses, and the proportion of "yes"" answers.
5

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 1. Evaluation with CHAIR score across multiple I.V.M architectures comparing our method with several baselines. We report C CHAIR, and the recall score. The bold numbers indicate the best results among the methods evaluated and the underscored numbers represent the second best results. We show MARINE- Truth as a reference performance MARINE.
Method
LLaVA
LL
MiniGPTv2
mPLUG-Owl2
InstructBLIP
Averuge
CHAIR
Cs
C₁
4
R1
Cs
C₁
4
Rt
Cs
4
C₁
R1
Cs
Cr
4
R1
Cs
4
C,
4
Rt
C3
Ci
4
R1
Greedy
26.6
10.5
47.4
8.8
4.6
41.1
8.2
4.2
41.1
6.2
3.4
38.8
5.0
3.2
33.2
11.0
5.2
40.3
LURE
33.8
11.6
5488
38,9
11.2
56.3
36.2
11.4
54.6
33.9
10.8
55.9
38.1
12.1
54.5
36.2
11.4
55.2
Woodpeeker
19.5
44.3
8.5
4.5
38.4
7.5
4.5
37.0
8.0
4.3
37.5
8.0
6.2
32.6
10.3
5.7
38.0
VCD
28.1
11.0
46.6
7.3
4.1
40.8
6.8
3.9
38.2
5.9
3.4
37.7
2.4
1.5
33.7
10.1
4.8
39.4
OPERA
22.4
9.9
43.6
11.0
6.7
40.2
9.2
5.0
41.3
5.8
3.2
38.4
4.6
2.7
38.0
10.6
5.5
40.3
MARINE
17.8
7.2
50.8
6.2
3.0
44.3
11.8
4.9
49.7
4.2
2.3
41.4
2.2
1.3
36.3
8.4
3.7
44.5
MARINE-Truth
19.6
5.1
79.0
6.0
2.5
55.3
12.6
3.8
70.5
3.8
1.7
48.0
3.0
1.8
35.9
8.9
2.9
57.5
Table 2. Evaluation with POPE score in adversarial setting across multiple LVL.M architectures comparing method with several
baselines. We report the POPE accuracy (%), Fl score (%) and the yes ratio (5) The ideal yes ratio for a non non-biased biased LVLM is 50%. The bold numbers indicate the best results among the methods evaluated and the underscored numbers represent the second-bess results. We
show MARINE- MARINE-Truthha as a reference performance of MARINE.
Method
LLaVA
LLaVA-v1.5
MiniGPTv2
mPLUG-Owl2
InstructBLIP
Average
POPE
Acc f
f
Yes
Acc 1
Fl
Yes
Ace
F1
Yes
f
F1
f
Acc
1
Yes
Acc 1
f
Yes
Greedy
51.8
67.4
97.7
79.4
81.6
61.6
82.7
81.7
44.5
72.5
77.5
72.4
79.8
81.4
58.6
73.2
77.9
67.0
LURE
Woodpecker
77.5
77.6
50.5
80.5
80.6
50.5
79.5
77.8
42.5
77.5
76.9
47.5
79.0
78.6
48.0
78.8
78.3
47.8
VCD
54.6
68.5
94.0
78.2
80.7
62.8
81.4
80.2
44.1
72.3
77.0
70.5
79.7
80.9
56.7
73.2
77.5
65.6
OPERA
51.7
67.4
98.0
77.5
80.1
63.2
82.9
81.9
44.3
70.3
79.1
84.6
79.8
81.4
58.6
72.4
78.0
69.7
MARINE
66.9
72.9
72.3
85.0
84.3
45.7
83.0
82,9
49.4
82.8
82.7
49.2
81.7
79.4
38.8
79.9
80.4
51.1
MARINE-Truth
75.6
80.1
72.3
92.0
92.5
57.0
86,9
88.3
62.5
93.4
93.8
56.2
93.8
93.8
51.0
88.3
89.7
59.8
such as LURE and Woodpecker are less effective across model variants.
Table Results of GPT-4V-aided evaluation. The accuracy and de- tailedness metrics are on scale of 10, and a higher score indicates better performance. The symbols x and v indicate performance metries without and with our method, respectively.
Importantly, MARINE achieves performance comparable to MARINE -Truth, a variant that uses ground-truth object labels as guidance. This finding suggests that aggregating signals from multiple visual models offers a compelling alternative to manual supervision to reduce hallucination.
L.
PPIUG Ow12
Task
Metries
1
5.8220 10
5,94
6.
6.35 10 at
LLAVA-QA90
Detail
4.59
4.59
5.16 5.06
Acc 7
5 2:27
6.11
7.9720.2
2.63
Image Captioning
Results on POPE. POPE is designed to assess object- level grounding in LVLMs by testing their ability to answer yes/no questions about visual content. We focus on the ad- versarial setting, which presents challenging negatives and helps expose hallucination and biased answering tendencies.
Detail
4.39
4.36
5.74joa:
6.19 10
Results on CHAIR. CHAIR is widely adopted bench- mark for evaluating caption hallucination in LVLMs, com- paring generated descriptions with ground-truth object anno- tations. It captures object-level precision through CHAIR, (instance-level) and CHAIRs (sentence-level), and we fur- ther report Recall to assess content coverage.
In Table 2, MARINE consistently outperforms all baselines, with average improvements of 6.7% in accuracy and 3.5% in F1 score over the original model outputs. Compared to the second-best method, Woodpecker, MARINE still maintains 1. 1.1% gain in accuracy and a 2.1% gain in F1.
Table 1 shows that MARINE consistently outperforms exist- ing approaches on all major metrics. It achieves the lowest average CHAIR, and CHAIRs scores and ranks second in Recall, reducing hallucination without sacrificing coverage. Compared to the second- best method, MARINE improves CHAIR by 1.7 points and CHAIR, by 1.1 on average. The gains are particularly strong on LLaVA models, where hal- Jucination drops by up to 8.8 points. In contrast, methods
Beyond accuracy, MARINE also reduces the overconfident bias often seen in LVLMS' outputs. This is reflected in a more balanced "yes" ratio (closer to 50%, reflecting a 15.9% shift towards unbiased answers). This shift suggests that MARINE produces more trustworthy predictions by reduc- ing the tendency toward overconfident affirmative responses.
6

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table POPE results across three datasets. We report the average score under random, popular, adversarial settings. The detailed POPE results can be found the appendix C. The bold numbers
BLEU_1
LLaVA mPLUG-OW12
SPICE
BLEU22
0.6
indicate the best results. The ideal yes ratio for a non-biased LVLM
is 50%.
MavA
mPI. JIGGDut
Dataset
WANRINE
Accuraey
1
F1 1
Yes(%)
Accuracy
1
F1 1
Yes(S))
51.2
68.5
95.5
76.7
80.4
68.2
CIDE
EU
MSCOCO
72.2
76.4
66.9
N5.5
85.0
46.5
51.8
67.5
97.9
69.6
76.5
78.5
A-OKVQA
<
64.3
72,8
M0.2
82.0
83.5
57.2
GQA
x 52.0
67.6
97.8
73.7
78.7
72.6
ROUGE
BLEU_ 44
/
62.5
71.8
$1.8
80.1
80.6
51.1
Following Yin et al. (2023), this GPT-4V-assisted evaluation provides a qualitative perspective that complements the numerical met- rics of CHAIR and POPE, offering a more comprehensive assessment of model performance. As shown in Table 3, GPT-4V consistently assigns higher accuracy with equal detailedness scores to models enhanced by MARINE, high- lighting its ability to produce more precise and detailed de- scriptions, which demonstrates the robustness of our method in real-worle visual tasks. The evaluation prompt is detailed in Appendix A.5.
Results on GPT-4V-aidee evaluation.
Figure 2. MARINE maintains or improves overall text quality on general metrics. Solid lines indicate models with MARINE, while dashed lines indicate the original models. Higher scores indicate better textual similarity to the reference outputs.
increases decoding time by just 1.98x, the lowest among all baselines. This demonstrates that MARINE achieves the most favorable trade-off between latency and accuracy, which makes it suitable for real-world use. Detailed settings are provided in Appendix A.6.
5.3 Ablation Study
Why incorporate multiple image-grounded models? Different image-grounded models excel at capturing dif- ferent aspects of visual information cermmttio---mme detect objects precisely, while others offer broader, fine-grained context. To understand whether combining these complementary signals leads to better guidance, we conduct an ablation comparing DETR and RAM++ individually versus in com- bination (Table 6). All variants are evaluated under the same decoding setup to ensure a fair comparison.
Additional results on other vision-language tasks. To further evaluate the generalizability of our approach beyond object hallucination and the MSCOCO dataset, we extended our evaluations to additional datasets including A-OKVQA and GQA and included more general caption quality met- rics. As shown in Table 4, the POPE results demonstrate that our method consistently mitigates hallucinations across various datasets with different image distributions. Figure 2 presents a comprehensive evaluation of the image caption- ing task on MSCOCO and ALaVA-QA99, a comprehen- sive VQA dataset, using metrics including BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), CIDEr (Vedantam al., 2015) and SPICE (Anderson et al., 2016). These results demonstrate that, although our method primarily targets hal- lucination mitigation, it maintains the overall performance of LVLMs on broader tasks, with no significant trade-offs in caption or VQA quality.
DETR allows for highly accurate object detection, while RAM++ excels in extensive recognition tasks, contributing fine-grained visual concepts. Their combination yields con- sistent improvements CHAIR metrics, suggesting that aggregating multiple visual perspectives is important for effective hallucination mitigation.
What is the best way to integrate guidance from multi- ple models? When aggregating the outputs from multiple mmaage-grondding models, the combination method can sig- nificantly affect guidance quality. We compare two strate- gies: taking the intersection or the union of detected objects.
Lateney analysis Many existing approaches mitigating object hallucination rely post-generation correction mod- els (Zhou et al., 2023; Zhai et al., 2023; Yin et al., 2023), external object detectors (Yin et al., 2023), or complex de- coding strategies (Huang et al., 2023a; Leng et al., 2023), all of which introduce substantial computational overhead. To assess the practical efficiency of MARINE, we evaluate its latency compared existing basclines on LLa as shown in Table 5.
As shown in Table 7, the intersection -based approach consis- tently outperforms the union, significantly reducing halluci- nation. This suggests that enforcing agreement across mod- els leads to more precise and trustworthy guidance, while umion-based aggregation may introduce noisy or spurious information. The detailed experimental setup and prompt templates are provided in Appendix A.
Our measurements include the time required for additional forward passes through external vision models. These mod- els contribute only marginal latency relative to the cost of autoregressive decoding in LVLMs. In general, MARINE
How does control strength affect generation? To under- stand the impact of guidance strength in our decoding setup,
7

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table Inference latency comparison. We report both the latency and the ratio to the latency of greedy decoding of the original L.V.M
model.
Greedy
LURE
Woodpecker"
VCD
OPERA
MARINE (nurs)
Training Cost
0
10min on A100 80G
Inference Latency(molkkm)
26.3 (x1.0)
179.9 (x6.84)
94.5 (x3.59)*
53.4 (x2.03)
185.1 (x7.0)
52.2 (x1.98)
'Woodpocker requires GPT API key access and the latency may depend on OPENAT API.
Table 6. Ablation study comparing the performance of combining DETR and RAM++ models versus using individual vision models. This approach leverages multiple object detectors to provide more reliable and robust object-level guidance, resulting in superior performance CHAIR metrics.
0.10
0.50
CHAIRS
0.20
Kocall
0.45
0.02
0.03
Model
LLaVA
LLaVA-v1.5
mPLUG-Ow12
Guidance Strength
Strength
Guidance Strength
CHAIR
Cs 1
Cr
4
Cs
Cy
4.
Os
4
Cr
LLaVA
PPLUG-OWl2
LLaVA-v1.5
Greedy
26.6
10.5
8.8
4.6
6.2
3.4
Ensembling Models
Figure 3. Ahlation study on the effect of guidance strength (r) on the performance of LLaVA, LLaVA-v1.5 and mPLUG-OW12 using CHAIR metrics, with y ranging from to 1.
MARTNE
17.8
7.2
6.2
3.0
4,2
2.3
Single Models
MARINE-DETR only
27.6
10.5
4.3
5.3
2.7
Query: Is there chair in the image?
MARINE only
29.0
6.6
3.7
5.2
2.8
LLaVA: Yes, there is white chair in the image.
Table Effect of Integration Methods for Image-Grounding Mod-
LLaVA MARINE: No, there is chair in the image. The only object present is white bird.
els.
Model
LLaVA
LLaVA-1155
mPLUG-Owl2
Query: Generate short caption of the image.
CHAIR
Cs 4
C₁
Ca
C
Cs
C₁
L
LLaVA-v1.5: 1: kitchen with a micrewave, coffee maker, and toaster.
Greedy
26.6
10.5
88
4,6
62
34
(ours)
3.0
LLaVA-v1.5 w/ MARINE: kitchen counter with
MARINE-union
30.4
9.7
5.4
277
4,8
2.7
microwave, coffee maker, and laptop.
we vary the control weight 7, which balances the influence between the original LVLM generation and the generation conditioned on external mmage-grounded guidance.
Figure 4. Hallucination mitigation examples by our proposed MAR NE across multiple tasks. Hallucinated objects generated by the LVLM are highlighted in red.
Figure shows that increasing guidance strength from 0 to 1 leads notable decrease in CHAIR scores. This trend suggests that higher guidance strength makes LVLMs rely more on image-grounded features, thereby enhancing their ability to produce accurate deseriptions. It's crucial to note that, although some models exhibit optimal performance at guidance strength of 7 L, excessively strong guid- ance can adversely affect the models' ability adhere to provided instructions. Experimental evidence is detailed in Appendix B.5. This observation highlights the neces- sity of having a balanced guidance strength that ensures high-quality, accurate outputs while adhering closely to the given instructions. Based on our findings, we recommend guidance strength within the range of > E (0.3, 0.7) as the most effective for maintaining this balance.
ance framework in the multi-modal setting, MARINE effec- tively and cost-efficiently reduces the hallucinations of five widely- used LVLMs, as assessed by various metrics across different tasks. The inherent compatibility of the MAR TNE with various vision models and projection functions further underscores its flexibility. In contrast to post-generation correction methods, MARINE strikes a balance between ef- ficiency, citructioo-followiing ability and effectiveness in reducing object hallucinations.
Limitations and future work. While MARINE has demon- strated impressive performance by utilizing guidance from amage-groundee models, there remains potential for fur- ther improvement through the integration of advanced ag- gregation methods, such multi-agent debate (Du et al., 2023), into the MARINE framework. Additionally, al- though MARINE is specifically designed to mitigate object hallucination, which is the most significant issue in LVLMs, extending its application to address other types of hallucina- tions in both L.MS and LVLMS across broader range of benchmarks would be highly advantageous.
Conclusions, Limitations and Future Work
In this paper, we introduced a training- free and API-free framework MARINE to mitigate object hallucination in LVLMs during its text generation process. Leveraging a pre-trained object grounding vision encoder for a novel guid-
8

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Acknowledgments
transformers. In European conference on compuler vision, pp. 213-229. Springer, 2020.
We thank anonymous reviewers for their helpful comments. Part of this work was done while WZ was PhD student at UCLA. WZ and QG supported in part by NSF grants DMS-2323113, 13, CPS-2312094 IIS-2403400, and the re- search fund from the UCLA-Amazon Science Hub. WZ was also supported by the UCLA dissertation year fellow- ship. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
Carlsson, F,, Öhman, Liu, F., Verlinden, S., Nivre, J., and Sahlgren, M. Fine-graines controllable text generation using non-residual prompting. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6837-6857, 2022.
Chambon, P., Bluethgen, C., Langlotz, C. P., and Chaudhari, A. Adapting pretrained vision-language foundational models to medical imaging domains. arXiv preprint arXi::22 2022.
Impact Statement
This paper introduces research aimed at advancing the field of Large Language Models. We confident that work will contribute to significant social benefits, particularly by enhancing the accountability of LLMs through the re- duction of hallucinatory outputs. Our proposed method, MARINE, holds the potential to improve the fairness of LLM interactions by effectively reducing biased halluci- nations. By mitigating hallucinations, MARINE has the potential to offer a positive social impact by ensuring that LVLMs generate more accountable responses. Despite this merit, MARINE cannot address prejudicial biases inherent in LLM prior knowledge, which could be focus of future work. To the best of our knowledge, we have not identified any negative effects associated with our research that merit highlighting in this discussion.
Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krish- namoorthi, R., Chandra, V., Xiong, Y., and Elhoseiny, M. Minigpt-v v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint ArXiv:2310.09442 2023.
Chen, Z., Zhao, Z., Luo, H., Yao, H., Li, B., and Zhou, J. Halc: Object hallucination reduction via adaptive focal- contrast decoding. arXiv preprint arXiv:2403.00422 2024.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L.,, Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
References
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-2377 2022.
Cho, J., Hu, Y., Garg, R., Anderson, P., Krishna, R., Baldridge, J., Bansal, M., Pont-Tuset, J., and Wang, S. Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-to-image generation, 2024. URL https::/..............tt arxiv. orgg httgg//bb//21...1/80 18235.
Anderson, P., Fernando, B., Johnson, M., and Gould, S. Spice: Semantic propositional image caption evaluation, 2016.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purper
languagee models with instruction tuning, 2023a.
Bazi, Y., Rahhal, M. M. A., Bashmal, L., and Zuair, M. Vision-languase model for visual question answering in medical imagery. Bioengineering, 10(3):380, 2023.
Dai, W. Liu, Z., Ji, 7., Su, D., and Fung, P. Plausible may not be faithful: Probing object hallucination vision- language pre-training. In Proceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, pp. 2128 21400, 2023b.
Bird, S., Klein, E.,, and Loper, E. Natural language process- ing with Python: analyzing text with the natural language toolkit. O' O'Reilly Media, Inc.", 2009.
Biten, A. F., Gómez, L., and Karatzas, D. Let there be clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pp- 1381-1390, 2022.
Deng, Y., Lu, P., Yin, F., Hu, Z., Shen, S., Zou, J., Chang, K.-W., and Wang, W. Enhancing large vision language models with self-training on image comprehension. arXiv preprint arXiv:2405. 2024.
Dhariwal, P. and Nichol, A. Diffussonn models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn,
answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55(12)11-388 2023.
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mor- datch, I. Improving factuality and reasoning in lan- guage models through multiagent debate. arXiv preprint arXiv:2305. 14325, 2023.
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT, Parekh, Z., Pham, H., Le, Q. Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and lasiuae representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916 PMLR, 2021.
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L.., Wang. X., Huang. T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19369, 2023.
Jing. L., Li, R., Chen, Y., Jia, M., and Du, X. Faithscore: Evaluating hallucinations in large vision-language mod- els. arXiv preprint arXiv:2311.0147) 2023.
Fu, C.,, Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, 7., Lin, W., Yang, J., Zheng, X., et al. Mme: A compre- hensive evaluation benchmark for multimodal large lan- guage models. arXiv preprint arXiv:2306. 13394, 2023.
Kar, O. F., Tonioni, A., Poklukar, P., Kulshrestha, A., Za- mir, A., and Tombari, F. Brave: Broadening the vi- sual encoding of vision-language models, 2024. URL http://axxvv....rabb/
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., Li, H., and Qiao, Y. Llama-adapter v2: Parameter-efficient visual instruction model, 2023.
https: http///axvvv./. .rg. abs////b////// 2404. 07204.
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. arXiv preprint arXiv:231 16922, 2023.
Gunjal, A., Yin, J., and Bas, E. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308. 204.2330..06394 2023.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Boot- strapping language-inen -maaa pre-training with frozen im- age encoders and large language models. arXiv preprint arXh::2301.1259) 2023a.
Ho, J. and Salimans, T. Classifier-fres diffusion guidance. In NewrlPS 202/ Workshop Deep Generative Models and Downstream Applications, 2021.
Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., and Smith, N. A. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question swering, 2023. URL https: pittpa://arxiv. http://axxvvvoooo abs/ 2303. 11897.
Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-lm improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328-4343 2022.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continu- ous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, 2021.
Hu, Z. and Li, L. causal lens for controllable text generation. Advances in Neural Information Processing Systems, (4:2m44,44:4441 34:24941-24499 2021.
Huang, Q., Dong, X., Zhang, P., Wang, He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviating hallucination in
large language models via over-trust penalty and retrospection-allocation arXiv preprint arXiv:2311.11911 2023a.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.R.. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.1033 2023b.
Huang, X., Huang, Y.J.., Zhang, Y., Tian, W., Feng, R., Zhang, Y., Xie, Y., Li, Y., and Zhang. L. Open-set image tagging with multi-grained text supervision, 2023b. URL https: aariiiiv.. org/ abs/ hrg/abs/// 22100311000 15200.
Lin, C.-Y. ROUGE: A package for automatic evalua- tion of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https: / //aclanthology. org//004-011.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question
10

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Lin, S.-C., Li, M., and Lin, J. Aggretriever: A simple approach to aggregate textual representations for robust dense passage retrieval. Transactions of the Association for Computationnal Linguistics, 1 11:436-452 2023a.
Lu, J., Yang, J.. Batra, D., and Parikh, D. Neural baby talk. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7219-7228, 2018, doi: 10.1109/ CVPR.2018.00%7 8.0054..
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitmick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECcV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 62, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024.
Madaan, A., Setlur, A., Parekh, T., Poczós, B., Neubig, G., Yang. Y., Salakhutdinov,, R., Black, A. W., and Prabhu- moye, S. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meeting of the Asso- cialion for Computational Linguistics, pp. 1869-1881, 2020.
Lin, Z., Madotto, A., Bang, Y., and Fung, P. The adapter-bot: All-in-one controllable conversational model. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 16081- 10081-16083, 2021.
Lin, Z.., Gong, Y., Shen, Y., Wu, T., Fan, Z.,, Lin, C., Duan, N., and Chen, W. Text generation with diffusion language models: A pre-training approach with continuous para- graph denoise. In International Conference on Machine Learning, pp. 21051-21060 PMLR, 2023b.
Niu, T. and Bansal, M. Polite dialogue generation with- out parallel data. Transactions of the Association for Computational Linguistics, 6:373-389, 2018.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation, 2024. URL https:// /ttt:///rxiii aarxi.ivivvv//ii///aa//aaaaa org/
01291.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 31 311-318, 2002.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang. L. Aligning large multi-modal model with robust instruction tuning. arXiv preprint Sctiv:230..15666 2023a.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination large multi-modal models via robust instruction tuning, 2023b.
Petryk, S., Chan, D., Kachinthaya, A., Zou, H., Canny, J., Gonzalez, J., and Darrell, T. ALOHa: A new measure for hallucination in captioning models. In Duh, K., Gomez, H., and Bethard, S. (eds.). Proceedings of the 2024 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 2: Short Papers), pp. 342-357, Mex- ico City, Mexico, June 2024. Association for Computa- tional Linguistics. doi: 10. 10.18655/ 30. URL https: /ttr//acaaaa naacl-short.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved base- lines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023c.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tuning. In NeurlPS, 2023d.
Liu, S., Ye, H., Xing, I., and Zou, J. Reducing hal- lucinations in vasion-language models via latent space steering, 2024a. URL https::///// http//axxvvvv//// https/////ww.....aa/baaoos 2410.15772
Prabhumoye, S., Black, A. W., and Salakhutdinov, R. Ex- ploring controllable text generation techniques. In Pro- ceedings of the 28th International Conference on Compu- tational Linguistics, pp. 1-14, 2020.
Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free free method for alleviating hallucina- tion in Ivims. arXiv preprint 240. 2024b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PMLR, 2021.
Lovenia, H., Dai, W., Cahyawijaya, S., Ji, Z., and Fung, P. Negative object presence evaluation (nope) to measure object hallucination in iision-languaa models. arXiv preprint 100i:22100.05. 0000333,, 2023.

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Ribeiro, L.. F., Zhang, Y., and Gurevych, I. Structural adapters in pretrained language models for amr-to-text generation. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, Pp. 4269-42888 2021.
Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng. M., et al. Reco: Region- controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246-14255, 2023.
Rohrbach, A., Hendricks, A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 4035-4045, 2018.
Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A.. Shi, P., Shi, Y., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint arXiv:2304. 4/78, 2023.
Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., and Chen, E. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint 1xXi::2100.6044. 2023.
Sanchez, G., Fan, H., Spangher, A., Levi, E., Am- manamanchi, P. S., and Biderman, S. Stay on topie with classifier-free guidance. arXiv preprint arXiv:2306. 170002306.78066, 2023.
Yu, T., Yao. Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., et al. Rlhf-v: To- wards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024.
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. A-okvqa: A benchmark for visual ques- tion answering using world knowledge. In European Conference on Computer Vision, pp. 146-162. Springer, 2022.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Zhai, B., Yang, S., Xu, C., Shen, S., Keutzer, K., and Li, M. Halle-switch: Controlling object hallucination large vision language models. arXiv e-prints, pp. arXiv-2310. 2023.
Zhang, H., Song, H., Li, S., Zhou, M., and Song, D. A survey of controllable text generation using transformer- based pre-trained language models. ACM Computing Surveys, 56(3):1-37, 2023a.
Vedantam, R., Zitnick, C. L., and Parikh, D. Cider: Consensus-beed bassed image description evaluation, 2015.
Wan, D., Cho, J., Stengel-Eskin, E., and Bansal, M. Contrastive region guidance: Improving grounding in vision-language models without training. arXiv preprint arXiv:2403.02325. 2024.
Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W., Gao, P., et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 146244 2024a.
Wang, B.. Wu, F., Han, X., Peng, J., Zhong, H., Zhang, P., Dong, X., Li, W., Li, W., Wang, J., et al. Vige: Visual instruction generation and correction. arXiv preprint arXiv:2308. 127 14, 2023a.
Zhang. Y., Qian, S., Peng, B., Liu, S., and Jia, J. Prompt highlighter: Interactive control for multi-modal 1lms. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13215-13224 2024b.
Wang. J., Zhou, Y. Xu, G., Shi, P., Zhao, Xu, H., Yc. Q. Yan, M., Zhang, J., Zhu, J., et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint ar ar0iv:2308. 15126, 2023b.
Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain-of-the tf-thuught reasoning in language models, 2023b.
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., and Luo, P. Lvlm-chub: A comprehensive evaluation benchmark for large vision- language models. arXiv preprint arXiv:2306.09265 2023.
Zhou, Y., Cui, C.,, Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754, 2023.
Yang, L., Zheng, Z., Chen, B., Zhao, Z., Lin, C., and Shen, C. Nullu: Mitigating object hallucinations in large vision- language models via halluspace projection, 2025. URL https::// petps://arxxvse wwwpxiv.or///aaa 2412 13817.
Zhou, Y., Cui, C., Rafailov, R., Finn, C., and Yao, H. Align- ing modalities in vision large language models via prefer- ence fine-tuning. arXiv preprint arXiv:2402. 14/1, 2024.
12

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.05522 2023.
13

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
A
Experiment Setup
We conduct all of the experiments using 8 A6000 GPU with 48GB GPU memory. Each single experiment can be run on single A6000 GPU.
A.1 Model Architectures
In Table 8, we provide detailed descriptions of the LVLM architectures used in our experiments. These LVLMS respec-
tively leverage the pre-trained vision encoder of the models we listed, which are all based on the Vision Transformer (ViT) (Dosovitskiy et al., 2020) architecture.
Table 8. Details of the LVLM architectures that we used in our paper.
Model
Vision encoder
LLM
LLaVA (Liu et al., 2023d)
CLIP-L (Radford et al., 2021)
LLaMA-2-7B-Chal (Touvron et al., 2023)
LLaVA-v1.5 (Liu et al., 2023c)
CLIP-L-336px (Radford et al., 2021)
Vicuna-v1.5-7B (Chiang et al., 2023)
MiniGPT-v2 (Chen et al., 2023)
EVA-G (Fang al., 2023)
LLaMA-2-7B-Chat (Touvron et al., 2023)
mPLUG-OWL.2 (Ye et al., 2023)
CLIP-L (Radford et al., 2021)
LLaMA-2-7B (Touvron al., 2023)
InstructBLIP (Dai et al., 2023a)
BLIP-2 (Li et al., 2023a)
Vicuna-v1. (Chiang et al., 2023)
A.2 Descriptions about Additional Metrics
In Figure 2, we evaluate the text quality of the outputs generated with MARINE using general metrics as follows:
BLEU (Papineni et al., 2002) measures how well the generated translation matches the reference translations in terms of n-gram overlap.
ROUGE-L (Lin, 2004) measures the quality of machine-generatee summary by comparing it to one or more reference summaries.
CIDEr (Vedantam et al., 2015) assesses the quality of image captioning models. It focuses on evaluating how well the generated captions align with human consensus.
SPICE (Anderson et al., 2016) focuses on assessing the semantic similarity between the generated captions and reference captions.
A.3
Prompt Templates
For each query, we randomly select prompt template from the available template list, as shown in Table 9.
A.4 Details of Baselines
Specifically, the hyperparameters for LURE (Zhou al., 2023), VCD (Leng et al., 2023), OPERA (Huang et al., 2023a) are reported in Table 10, 11 and 12 respectively. We strictly followed the original implementations and default hyperparameters described in their papers to reproduce the results for each baseline.
A.5 Experiment Setting for Hallucination Evaluations
Key factors that potentially affect the hallucination evaluation outcomes, including the evaluation dataset and prompt template, LVLM's sampling strategy and batched generation techniques, and guidance strength, are detailed in this section. The
setting for MARINE and overall experiment settings are shown in Table and 14.
Experiment setting for CHAIR evaluation. We adopt the same prompt "Generate short caption of the image." as utilized by Li et al. (2023b). The hyperparameters are fixed, including guidance strength of 0.7, score threshold for DETR 0.95, detection threshold for RAM++ of 0.68, maximum token length of 64, and greedy sampling approach with a random seed of 242.
For the calculation of CHAIR metrics, we referenced the 80 object categories annotated in the MSCOCO dataset, following Rohrbach et al. (2018). Besides, we employed the synonym list from Lu et al. (2018) to align synonymous words the generated text with MSCOCO object categories. Additionally, due to the cost considerations associated with the GPT-3.5 API, we limited our analysis to 200 samples for Woodpecker correction for each model and reported the result in Table
Experiment setting for POPE evaluation. POPE is a flexible approach to evaluating hallucinations in LVLMs, which formulates binary classification task by prompting LVLMs with questions such as "Is there keyboard in this image?" to images
answer "yes" or "no". Following Li et al. (2023b), we created 3000 POPE questions across three datasets- each from MSCOCO, A-OKVQA, and GQA for the POPE evaluation. We reported the adversarial settings in Table 2, the most challenging setting, which constructs POPE questions from the top-k most frequently co-occurring but absent objects. Additionally, in Table 4, we reported the average scores under random, popular, adversarial settings across MSCOCO,
14

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 9. Details of the L.V.M architectures that used in paper.
Template Type
Prompt Template
MARINE-intersee
This image contains COBJECT.GROUNDIIN Based on this, <QUERY>
The image contains the following objects: COBJECT.GROUNDINGG Given these detected objects. <QUERY>
This image shows the following objects: <OBJECC..GROUND Using this infor-
mation, <QUERY>
The objects found in this image are: <OBJECT GRO.EDIIIGG. Considering this list of objects, <QUERY>
POPE task
This image contains only the following objects: <OBJECT.GROUNDING Do assume anything beyond these objects. Based solely on this list, <QUERY>
The detected objects in the image are: <OBJECT..ROUND ING>> Answer based only
on these objects. <QUERY> This image shows the following objects: BIBJECCT.RROUNDING. You must answer
using only the objects in this list. Given these detected objects, <QUERY> The objects found in this image are limited to: <OB COBJECT-C .ww.GGROUUDII ING>, You should rely strictly on this list of objects and make no other guesses. Based on this, <QUERY>
DOSJJCT..ROUND.II
MARINE-union
List detected objects the image: <OBJECT GROUNDING.A,
Based on the detected objects above, <QUERY>
The most prominent objects detected are:
<OBJECT.GROUNDDNG..
<OBJECT_GROUNDINN
Given these findings, <QUERY>
The following objects were detected in the image:
<OBJECT.GROUNDING
<OBJECT_GGROUDDING.A
With this information, <QUERY>
Here is a list of all objects detected in the image:
COBJECT.GROUNDIE
BRQJECT.GROUNDINCN
Do not infer or hallucinate any additional objects. Using only the detected objects,
<QUERY>
Table LURE (Zhou et al., 2023) Hyperparameter Settings
Parameters
Value
Uncertainty Threshold y
0.9
Position Threshold
0.8
Table 11. VCD (Leng et al., 2023) Hyperparameter Settings
Parameters
Value
Amplification Factor
Adaptive Plausibility Threshold
0.1
Diffusion Noise Step
500
Table 12. OPERA (Huang al., 2023a) Hyperparameter Settings
Parameters
Value
Self-attention Weights Scale Factor
50
Attending Retrospection Threshold
25
Beam Size
Attention Candidates
1
Penalty Weights
1
15

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 13. MARINK BaARiNe,,yyernnnnee Hyperparameter Settinss.
The settings are fixed depending on the question-answering tasks.
Parameters
Value
Guidance
Guidance Strength
0.7
score threshold for DETR
0.95
Detect Threshold for RAM++
0.68
Generation
Max Token Length
64
Sampling Random Seed
Greedy 242
Table 14, Batch size for LVLM generation is fixed across all experiments unless otherwisc noted. To expedite the evaluation process, we employed the batched generation. We avoid the negative impact of batched generation by adopting left padding if the LVLM does not explicitly assign the padding strategy for inference.
Model LLaVA
LLaVA-vi.5
MiniGPTv
mPLUG-OWl2
InstructBLIP
Batch Size
16
32
16
16
A-OKVQA, and GQA datasets. The full POPE results are in Tabel 16.
Similarly, we constrained our analysis to 200 samples for Woodpecker correction for each model due to the high costs associated with the GPT API. The outcomes of this analysis are detailed in Table Experiment setting for GPT-4V-aided evaluation. The GPT-4V-aided evaluation compares the outputs of two LVLM assistants using GPT-4V as a judge. We prompted GPT-4V to assess the quality of the generated outputs, scoring them out of 10 in two aspects:
Accuracy: how accurately each assistant describes the image;
Detailedness: the richness of necessary details in the response.
As shown in Table 15, the assessment prompt template we used is slightly different from that of Yin al. (2023), Specifically, we include the original question for task-orientated evaluation and exclude prompts that describe Woodpecker-speciffe output formats like object bounding boxes.
Experiment setting for ablation study. To explore different methods of integrating image-grounding models, we investigate the intersection and union detected objects, with integration based on synonyms using the NLTK package, To quantitatively assess the influence of guidance strength, we varied it from to 1, as shown in Figure 7. These quantitative experiments were conducted using the same setting as those in CHAIR evaluation. For qualitative analysis, we selected guidance strength from recommended range of y E (0.3, 0.7).
A.6 Experiment Setting on Other Vision-Language Tasks
Experiment setting for text quality analysis. For text quality analysis, we adopted 90 visual questions from the LLa QA90 task (including conversations, visual perceptions, and complex reasoning subtasks), and randomly selected 500 MSCOCO images for image captioning task. Following Liu et al. (2023d), we adpoted the response generated by text-only GPT-4 (0314) with the context captions/boxes provided. answers given by GPT-4 as references for LLaVA-QA90 task and used image captions provided in MSCOCO annotations as references for image captioning task.
In Table 17 and Table 18, we present detailed evaluation on the image captioning task for both MSCOCO and LLaVA-QA90 using metries including BLEU, ROUGE, CIDEr and SPICE. The corresponding figure result shown in Figure 2.
Experiment setting for latency analysis. We compared our method with existing baselines in terms of the trade-off between inference cost and the effectiveness of reducing object hallucinations, as shown in Table 5. For post-correction baselines such as Woodpecker and LURE, we first prompted LLaVA (1lava -llmaa- 2-7b- 22-7--chtt-a-e -ighttiiggtniigg pepnittiinn-rreiiiie iiw to generate captions and then measure the latency of generating the corrected outputs. The total latency for post-correction baselines includes both the generation and correction processes. For decoding methods such as VCD, OPERA and our method, we measured the latency of LLaVA generating captions directly.
We prompted the models with "Generate short caption of the image." on 500 MSCOCO images with a batch size of and
1ttpps:/ / thub. com http/hhotiaaa / http:////a////////////////////// 30x3. jsonl
16

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 15. Prompt template GPTdeV-aided evaluation. (question) the original instruction; (answer 13 is original response,
(answer 2) the response generated by the LVLM using MARINE.
Prompt template for GPT-4V-aidee evaluation
You are required to score the performance of two Al assistants in describing given image. You should pay extra attention to the hallucination, which refers to the part of descriptions that inconsistent with the image content, such as claiming the existence of something not present in the image.
Please rate the responses the assistants on scale of to 10, where a higher score indicates better performance, according to the following criteria: Accuracy: whether the response is accurate with respect the image content. Responses with fewer hallucinations should be given higher scores.
2. Detailedness: whether the response is rich in necessary details. Note that hallucinated descriptions should not count necessary details.
Please output a single line for each criterion, containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space. Following the scores, please provide an explanation your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
Question: (question) Assistant 1: (answer }}
Assistant 2: {answer 2}
Output format:
Accuracy:
Scores of the two answers:
Reason:
Detailedness:
Scores of the two answers:
Reason:
maximum token length of 64, without any stopping criteria, using single A6000 GPU. Then latency calculated the ratio of the number of output tokens and encoding and generation time.
17

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 16. Detailed POPE (Li et al., 2023h) results three datasets (MSCOCO (Iiin et al., 2014), A-OKVQA (Schwenk et al., 2022),
GQA (Hudson & Manning, 2019)).
Dataset
Type
Model
w/MARINE
Accuracy f
Precision
Recall
f
Yes(%)
X
51.8
50.9
99.5
67.4
97.7
LLaVA
66.9
61.7
89.1
72.9
72.3
Adversarial
X
72.5
65.5
94.9
77.5
72.4
mPLUG-Ow12
V
82.8
83.4
82.0
82.7
49.2
X
52.4
51.2
99.8
67.7
97.4
LLaVA
71.3
63.8
88.9
75.6
67.5
MSCOCO
Popular
75.8
68.7
94.9
79.7
69.0
mPLUG-Owl2
85.6
88.4
82.0
85.1
46.4
58.3
54.5
99.7
70.5
91.4
LLaVA
78.5
73.4
89.3
80.6
60.8
Random
81.8
75.2
94.9
83.9
63.1
mPLUG-Owi2
88.1
93.4
81.9
87.3
43.9
50.0
50.0
99.5
66.6
99.5
LLaVA
56.3
53.6
94.3
68.3
88.1
Adversial
62.5
57.3
98.1
72.3
85.6
mPLUG-Ow12
74,4
68.8
89.3
77.7
64.9
50.1
50.1
99.8
66.7
99.7
LLaVA
63.0
58.0
94.5
71.9
81.6
A-OKVQA
Popular
X
69.1
62.1
97.9
76,0
78.9
mPLUG-Ow12
82.5
78.8
89.1
83.6
56.5
55.4
52.8
99.8
69.1
94.4
LLaVA
73.7
66.7
94.7
78.3
71.0
Random
77.2
69.2
98.2
81.2
71.0
mPLUG-Ow12
89.2
89.2
89.3
89.2
50.1
X
50.3
50.1
99.8
66.8
99.5
LLaVA
54.4
52.5
93.8
67.3
89.4
Adversial
68.4
63.0
98.2
75.6
79.8
mPLUG-Ow12
76.0
73.6
81.2
77.2
55.2
X
50.1
50.0
99.8
66.7
99.7
LLaVA
58.7
55.1
94.3
69.5
85.5
GQA
Popular
70.6
63.8
94.9
76.3
74.4
mPLUG-Ow12
77.6
75.6
81.3
78.4
53.8
X
55.7
53.0
99.8
69.2
94.1
LLaVA
74.3
67.3
94.8
78.7
70.5
Random
82.0
75.2
95.5
84.1
63.5
mPLUG-Ow12
86.8
91.5
81.3
86.1
44.4
Table 17. Performance on general metrics for the image captioning task, including BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004).
CIDEr (Vedantam al., 2015) and SPICE (Anderson al., 2016) scorcs(5)).
Model
w/MAR TNF.
BLEU (t)
BLEU22 (t)
BLEUL3 (t)
BLEU_4
ROUGEJ (t)
CIDEr (t)
SPICE (t)
14.06
3.72
0.08
16.77
LLaVA
18.59
9.96
5.47
3.04
26.02
0.21
20.58
X
39.91
25.16
16.57
11.24
36.26
1.05
26.82
mPLUG-Ow12
39.51
24.37
15.93
10.70
36.01
1.03
27.42
B Additional Experiments
B.1
Additional Baselines
To further contextualize the effectiveness MARINE, we conducted additional experiments comparing our approach to
baseline that employs carefully engineered prompts designed to reduce hallucination. Specifically, we used the following prompt:
18

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 18. Performance on general metries for the I.LVVA- task, including BLEU (Papineni al., 2002), ROUGE-I (Lin, 2004),
CIDEr (Vedantam et al., 2015) and SPICE (Anderson et al.. 2016) scores(9)).
Model
w/MARINE
BLEU.I (t)
BLEU (t)
BLEU (t)
(1)
ROUGEL (1)
CIDEr (t)
SPICE (t)
21.02
12.91
8.79
6.41
32.30
0.93
31.36
LLavA
23.37
14.39
9.59
6.83
33.81
0.99
31911
X
44.50
28.57
19.58
14.43
40.24
1.46
40.51
mPLUG-Ow12
V
45.82
28.87
19.24
13.70
38.54
1.29
38.70
Table 19. Comparison against carefully engineered prompts.
Method
LLaVA
LLaVA-v1.5
mPLUG-Ow12
CHAIR
C, 4
C, 4
Recall
C, 4
C 4
Recall f
C,
C
Recall f
Original
26.6
10.5
47.4
8.8
4,6
41.1
5.0
3.2
33,2
Direct Prompting
27.2
11.0
46.4
19.6
8.3
52.3
9.0
5.1
42.0
Prompts as Additional Guidance
37.4
10.5
50.4
12.6
5.9
44.6
6.6
3.9
40.4
MARINE (ours)
17.8
7.2
50.8
6.2
3.0
44.3
4.2
2.3
41.4
Table 20, Experiments on dynamic guidance strength based on confidence scores on CHAIR metrics.
Method
LLaVA
mPLUG-Ow12
CHAIR
C, J
C, L
Recall
C,
C, 4
Recall
Guidance Strength
7.2
50.8
2.3
41.4
Dynamic Guidance Strength
14.8
6.5
49.9
41.0
Table 21. Experiments on dynamic guidance strength based on confidence scores POPE metrics.
Method
I.
mPLJG-OW12
POPE
Accuracy 1
FI
Yes Ratio
Accuracy
F1
Yes Ratio
Fix Guidance Strength
66.9
72.9
72.3
82.8
82.7
49.2
Dynamic Guidance Strength
71.97
74.48
59.83
83.3
83.2
49.4
Describe the visible contents of this image in as much detail as possible without adding any information not clearly visible. Only mention objects. colors, shapes. and textures that can be directly observed in the image. avoiding assumptions about materials, functions, or contexts. If there are any uncertainties about what an object is, describe visual characteristics (e.g., 'a circular object with a smooth surface') ) without inferring its purpose or identity, Avoid creative or hypothetical descriptions, and focus on observable details only.
With two different settings:
The original input query was replaced with the prompts as described. Prompts as Additional Guidance: We incorporated the prompt as supplemental context to guide the models in generating outputs.
Direct Prompting:
As shown in Table 19, prompt-based guidance can improve recall for some models (e.g.. LLa but does not consistently reduce hallucinations across all metrics. In fact, CHAIR scores often worsen. In contrast, MARINE achieves stronger improvements across all models.
We highlight two key differences between MARINE and prompt-based approaches:
Model Dependence: Prompting methods rely heavily on the instruction-folllwing capabilities of the model. While they may reduce hallucinations slightly for stronger models (e.g., LLaVA-v1.5), they can worsen performance in weaker models (e.g., LLaVA). Additionally, prompt-based approaches may require fine-tuning to be effective (Deng et al., 2024). MARINE, by contrast, improves grounding through explicit visual signals, making it effective even without model retraining.
Generalization and Efficiency: Prompting methods often require task-specific tuning or dataset-aware phrasing. MARINE generalizes across tasks and models with minimal engineering and no fine-tuning, while offering more consistent hallucination reduction.
B.2 Dynamic Guidance Strength
We conducted additional experiments to compare fixed and dynamic guidance strength strategies using both CHAIR and POPE metrics (Tables 20 and 21).
19

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Fix Guidance Strength uses fixed guidance strength of 0.7, selected to balance hallucination reduction and instructions adherence.
Dynamic Guidance Strength adjusts the guidance strength dynamically by mapping the mean confidence score (s) of the image-grounding models to a range of (0.4, 0.8) using the formula
7 0.4 (0.8 - 0.4) (s 8min
8 Smax
8min
A higher confidence score indicates more reliable grounding, which results in stronger guidance. Empirically, we find that dynamic guidance improves performance for weaker models such as LLaVA, which are more sensitive to noisy signals. For stronger models like mPLUUGOWI12 a fixed guidance strength already sufficient to reduce object hallucinations effectively. B.3 Effect of Sampling Temperature
In our main experiments, we use greedy decoding (temperature 0) to ensure deterministic outputs and reproducible comparisons-consiae with our primary baseline (VCD) and common practice in hallucination benchmarks. To evaluate robustness under stochastic decoding, we also test with a temperature 0.6 and report mean + standard deviation Table 22. MARINE continues to outperform baseline generations across all hallucination metrics, demonstrating effectiveness regardless of sampling strategy.
Table 22. Object hallucination metrics under temperature = 0.6 sampling.
Method
LLaVA
mPI.UG-O112
CHAIR
C,
C 4
Recallf
C2 4
C, 4
Recallf
Greedy
26. 11.6
10.8
46.0
4.9_a.a
2.8
37.720.
MARINE (ours)
19.3
7.6
50.6
m.5
2.4
41.1
B.4 Memory Analysis
We evaluated the peak GPU memory usage during inference on 500 image captioning examples using the LLaVA model, with a batch size of 16 and a maximum generation length of 64 tokens. Results are reported in Table 23. Although MARINE introduces additional vision models, the overall memory footprint increases by only approximately 30% during inference-sigeee significaatl less than doubling. This is because the object detection models relatively lightweight compared to the large LLM backbone.
B.5 Further Study Guidance Strength
Figure shows how varying the guidance strength 7 affects the quality of LLaVA's output on the LLaVA-QA90 and image captioning tasks (max generation length 256).. We observe that setting 7 = 1 does not yield the best image captioning performance. In the LLa LLaVA-QA90 task, guidance strengths in the range of 0.5 to 0.7 lead to higher output quality. This observation is consistent with prior findings in classifier- free guidance literature: overly strong guidance can dominate the generation process and reduce fluency or instruction adherence.
To further validate these results, we use GPT-4V as an automatic judge to score outputs (on a 10-point scale) for accuracy and detail. The results, summarized in Table 24, show that balancing the original LVLM branch leads improved generation quality. Finally, Figure provides qualitative examples showing how excessive guidance reduce instruction alignment, often introducing unnecessary visual details into the response.
20

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 23. Peak GPU Memory Usage during Inference (GB) of MAR NN compared to greedy decoding and VCD,
Metric
Greedy
VCD
MARINE (Ours)
Peak GPU Memory Usage
23.53
20.73 (x0.88) 30.78 (x1.30)
Table 24. Results of GPT-4V-aided evaluation. The aceuracy and detailedness metrics are on scale of 10, and higher score indicates better performance. The symbols and v indicate performance metrics without and with method, respectively.
Task
Metric
(1 1)
(y = 0.7)
Accuracy
5.52
5.79
LLaVA-QA90
Detailedness
4,58
4.77
Accuracy
6.06
6.22
Image Captioning
Detailedness
5.00
5.24
LLaVA-QA90 Task
Image Captioning
40
BLEU /ROUGE
BLBU /OOGEE
388
22.5
20.0
36
15.0
34-
12.5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Guidance Strength
Guidance Strength
Bleu-1
ROUGE-L
Figure 5. The impact of guidance strength on the output text quality.
Further Analysis
C.1
Limitations of Hallucination Evaluation
While CHAIR and POPE are widely adopted for evaluating object hallucinations in vision-language models, both have inherent limitations. CHAIR depends on fixed object vocabulary and synonym list, which may miss rare or fine-grained concepts. POPE relies on the quality of segmentation tools to define ground-truth objects, introducing variability across settings.
To address these limitations, we incorporate ALOHa (Automatic Localized Hallucination) (Petryk et al., 2024), reference- based metric that evaluates hallucination at both the object level (AT.OHag) and the caption level (ALOHa). We follow the standard ALOHa setup using MSCOCO ground-truth captions and enable reference object detection for more precise and generalizable assessment. As shown in Table 25, MARINE consistently outperforms greedy decoding across all models and both ALOHa metrics.
C.2 Additional Related Work
Several recent works aim to improve grounding or reduce hallucination in vision-language models. BRAVE (Kar et al., 2024) enhances faithfulness by combining diverse visual sources, similar in spirit MARINE, but introduces additional trainable modules. MARINE achieves comparable performance with training-free free, modular design.
Other approaches focus on evaluation (Hu et al., 2023; Cho et al., 2024; Lin et al., 2024) or feature-level interventions (Yang et al., 2025; Liu et al., 2024a) to steer models away from hallucinations. Liu et al. (2024b) address texl inertia, where models generate similar outputs regardless image content. Wan et al. (2024) introduce sub-image contrastive alignment, and Zhang et al. (2024b) control generation by adjusting visual attention weights.
These methods highlight complementary strategies to MARINE's s structured, object-level guidance for reducing hallucination.
C.3 Effect of MARINE on logit distribution.
In Figure 7, we illustrate specific example that shows how MARINE influences the logit distribution of LVLMs during text generation. Specifically, MARINE is observed to selectively target the potential hallucinated tokens, reducing their original probabilities to mitigate the risk of hallucination in the generated text. For instance, in the provided example, the probability of "fork" is significantly lowered with MARINE, which would have originally resulted hallucinated object.
21

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Question: How might the art installation encourage interaction among people?
LLaVA w/ MARINE with balance guidance
The art installation, which consists of a large, colorful, and intricately painted tiger sculpture, likely to encourage interaction among people in several ways. Firstly, the vibrant colors and unique design of the liger sculpture can attract attention and draw people towards it, making them curious about its origin and significance. This curiosity can lead to conversations and discussions among passersby, fostering social interactions and sense of and prominence in the city's public space can encourage people to take a break from their daily routines and engage with the artwork. The presence of the tiger sculpture can create sense of awe and wonder, encouraging people to pause and appreciate the beauty of the artwork. This can lead to a more relaxed and enjoyable atmosphere, as people take the time to admire the art and enjoy the surrounding wwtrrnmmnt..iinnnn.aae.
the sculpture's size
GPT4 response as reference
The art installation, featuring oversized wooden bench and two massive animal art sculptures, encourages interaction among people by providing visually intriguing and inviting space to sit or gather. People naturally drawn to the unique and whimsical nature of these installations, sparking curiosity and conversation among those who encounter them. The oversized bench provides seating for multiple individuals, prompting people to sit down, rest, socialize while admiring the glant painted feline heads and other artworks on display. Additionally, the installations capture the attention of passersby, creating shared experience a sense community among those who visit the area. The artistic environment stimulates discussion and appreciation of the art, facilitating social interactions and connections among those who gather near the sculptures and oversized bench.
the sculpture's placement in the city's public space can also serve as focal point for community events, such as art exhibitions, cultural festivals, or even street performances. The tiger sculpture can become gather (reach the maximum generation length]
LLaVA MARINE with overly strong guidance
The art installation, which features a large, colorful tiger sculpture and car parked next to it, can encourage interaction among people in several ways. Firstly, the vibrant colors and unique design the tiger sculpture can attract attention and draw people towards it, making them curious about its origin and purpose. This curiosity can lead to conversations and discussions among passersby, encouraging social interaction next to the sculpture can provide a focal point for people to gather and take photos, further fostering social interaction and creating sense of community. The art installation can also serve exhibitions, festivals, or even photo shoots, which can further encourage people to interact with each other and engage with the artwork. Overall, the art installation can serve catalyst for social Interaction and community engagement, promoting a sense of connection and shared experience among people.
engagement. Additionally, the presence of the car parked
backdrop events gatherings, such art
Figure 6. This case highlights that overly strong guidance can induce the model to prioritize providing exhaustive visual details from the image, even when such details are irrelevant the specific instruction (c.g., "a car parked next to it"). In contrast, balanced guidance enables the model to maintain better adherence to the instruction while still utilizing the visual information effectively.
Conversely, standard language elements such as "various", an adjective describing the overall image context, and "with", a crucial preposition, maintain their original probabilities. This selective nature of modulation by MARINE ensures coherent and contextually relevant text generation that adheres to the instruction while effectively reducing hallucinations.
C.4 Discussion on fine-tuning methods
The examples shown in Figure illustrate that LURE, times, fails to adhere to the given instructions when correcting LVLM generations. Despite receiving concise image descriptions generated based on instructions for short responses, LURE predominantly overwrites them with excessively long responses that contain information irrelevant to the instruction. Furthermore, LURE fails to adequately address the binary question format of POPF, as LURE fixates on extended descriptions without responding with "yes" or "no",, making its evaluation using POPE impractical. This issue can be prevalent in small-scale fine-tuning methods, where the limited variety of the specifically tailored fine-tuning dataset harms the model's performance on other tasks. In contrast, the training-free approach of MARINE demonstrates effective mitigation of hallucinations across variety of question formats.
C.5 Extended Analysis in Ablation Study
Additional experimental results explore the score threshold of object grounding features, which are examined across LLaVA, and mPLUG-OW12, with findings presented in Figures 9, and 10.
This variation is achieved by implementing four confidence thresholds (0.5, 0.7, 0.9, and 0.95) in the DETR model predictions (with MAR PIRNNE-TT Truth serving as an ideal reference), where higher thresholds correspond to lesser, yet higher-
22

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Table 25. AI.OHa hallucination scores (all values are in %). MARINE improves over greedy decoding across models and metries.
Method
LLaVA
LLaVA-v1.5
mPLL'G-Ow2
ALOHat
ALOHHo 1
ALOHa
ALOHoo
ALOHaT
ALOHog 1
Greedy MARINE
40.1
70.1
61.9
83.1
70.2
87.0
48.7
76.1
66.7
85.6
72.9
88.2
Generate a short caption of the image.
+
original probs
control probs
ouptut probs
0.15
table laden with various dishes, includ- ing plate of scambled eggs, bowl of fruit, and a bowl of yogurt. There is also spoon and a fork present.
0.1
LLLVV w: MARINE
0.05
A family is cnjoying meal together on dining table, with various dishes including bowl of cake and cup of dessert. A spoon, a cell phone, and cuuu are also present on the table.
d
plate
bow
fork
(a) An example of image description where the original LLaVA (b) The probability distributions at the token of the hallucinated word in the original, control, and MARINE outputs. MARINE effectively decreases the the probability of "fork".
outputs a hallucinated object, "fork".
a
various
several
multiple
the
with surrounded
featuring
consisting
which
(c) Probabilities of non hallucinated words remain the same, high- lighting MARINE's ability preserve normal outputs.
Figure 7. This sample shows how MARINE controls logit distributions to mitigate hallucinations like "fork" while preserving the probabilities of "with", "various" during generation.
Input: Is there book in the image?
LLaVA: Yes, there is book in the image, and it is placed the next to doll.
LURE:
There are two dolls in bedroom scene, with one of them being the focus of attention. One of the dolls is sitting in crib, while the other is lying in a baby carriage in the same room. A the bed behind the doll in the crib.
book can be
Woodpecker: No, there is no book in the image. The image shows two beds([0.294, 0.001, 0.999, 0.908];[0.0, 0.37, 0.658, 0.998]), and dolI([0.078, 0.439, 0.553, 0.92]) is placed on each bed. A leddy bear is next to the doll.
MARINE: No, there is book in the image. The image features doll in crib, doll bed, and teddy bear.
Figure 8. Example responses to an image-question pair. The LURE-corrected output deviates from the original question, offering irrelevant descriptions without directly addressing the query. Woodpecker hallucinates the existence of two beds while there only one bed in the figure. In contrast, MARINE maintains the original answer's style and adheres to the user's instruction while eliminating hallucination. quality, visual information. Our findings highlight two significant insights. Firstly, an increase in the quality of visual information correlates with a noticeable decrease in hallucinations produced by the LVLMs. A lower threshold, which allows for more visual information but also includes noisier content, could potentially result in an increased occurrence of hallucinations. Furthermore, lower-quality visual information is associated with enhanced Recall. This suggests that LVLMs under guidance, despite the presence of noisy visual inputs, tend focus more on the visual details (i.e., objects).
23

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
resulting in more elaborate descriptions.
0
0.0
y=
0.10
0.3
3.00
0.6-
Thuth
0.95
Puth
0.05
Wot
CHAIRs
(b) CHAIR,
Recall
Figure 9. VL's performance on CHAIR according to different score threshold of object grounding features in MARINE. We consider four confidence thresholds (0.5, 0.7, 0.9, and 0.95) for DETR to vary score threshold.
0.14
y-0.5
y=1.0
0.52
0.05
0.48
0.08
0.03
0.06
0.44
0.04
6.1
0.35
6.
0
0.9
Truts
0.55
Tran
(a) CHAIRs
(b) CHAIR,
(c) Recall
Figure APLUG-OW12 performance on CHAIR according different score threshold object grounding features MARINE.
consider four confidence thresholds (0.5, 0.7, 0.9, and 0.95) for DETR to vary the score threshold, with MARINE-Truth serving as an ideal reference.
C.6 More Case Studies
In Figures 11, 12 and 13, we present examples of the outputs from LURE (Zhou et al., 2023), Woodpecker (Yin et al., 2023) and MARINE different tasks further validate our arguments in the paper.
24

---

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Query: What is the position of the skateboard in the image?
LLaVA: The skateboard is the ground, with the rider standing it and holding onto the trucks.
LLaVA w/ MARINE: The skateboard is on the ground, with the person standing it.
Query: Generate short caption of the image.
LLaVA-v1.5: family of four, including a woman and three children, are sitting at a table with plate of food in front of them.
LLaVA-v1.5 w/ MARINE: A family of four, including two adults and two children, sitting around dining table with plates food front of them.
Figure II. Hallucination mitigation examples by our proposed MARINE across multiple tasks: LLaVA-QA90 and image captioning. Hallucinated objects generated by the LVLM are highlighted red.
Query: Generate short caption the image.
LLaVA: A black and white dog is sitting a beach, surrounded by rocks and sand. The dog is looking at its owner who is standing behind it, as if waiting for a command or attention. The scene captures the dog's loyalty and affection towards its owner. (Note: 'bird' omitted)
The scene features a black and white dog sitting on beach, surrounded by pebbles. The dog looking at its owner who standing behind it, as if waiting for attention. The background reveals vast ocean, and other birds be
LURE:
scattered throughout the
The presence of these birds gives the impression that the dog not alone, and they are enjoying the time together on beach. Overall, the image captures charming scene on the sandy beach.
Woodpecker: black and white dog([0.22 (0., 0.662, 0.551, 0.894]) is sitting on the beach([0.002) 002, 0.029, 0. 998, 0,999]), surrounded by sand. The dog is looking at its ([oner([0.712,, 0.57, 0.92, 0.684]) who is standing behind it.
LLaVA w/ MARINE:
In the image, black white dog standing on beach, looking bird.
Figure 12. A comparison of responses from baseline models and our MARINE in image description task. It illustrates MARINE'S superior ability to reduce hallucinations, in contrast to LURE and Woodpecker, which fail to effectively address hallucinations and sometimes increase hallucinated content. This example highlights the strengths of sareet-duringggeenrrate framework over post-correcion approaches, showcasing its efficiency, preservation of original style, and enhanced adherence to instructions.
25

---

Mitigating Object Hallucination in Large ViniuneL Models via Image-Grounded Guidance
Query: Generate short caption of the image.
MiniGPT-V2: no
Woodpecker: No, there is no toothbrush in the image.
MiniGPT-v2 w/ MARINE: yes
Figure 13. A comparison of responses from baseline models and MAR NE POPE "yes-or-no" task. MiniGPT-v2 provides concise response without referencing any objects. Under these circumstances, Woodpecker unable to perform corrections via GPT-3.5 due to missing visual details. MARINE however, successfully corrects the response while retaining MiniGPT-v2's style.
26


Reviews:
[{"review_id": "pj08MppnGv", "reviewer": "Reviewer_yYZG", "summary": "This paper proposes a framework (MARINE) that aggregates a VLM and traditional vision tools such as object detection and image-text alignment. Concretely, given an input image, MARINE uses vision tools as guidance models, achieved through a linear combination of unconditional and conditional logits over the vocabulary. This framework can adopt VLMs where logits are accessible and vision tools, and it does not require fine-tuning or any processing via an API for the vision tools.\n\nThis approach is primarily evaluated using two automatic metrics for hallucination detection in text (CHAIR and POPE), and it’s compared against various decoding methods. On average, across 5 different base VLMs, MARINE outperforms the baselines in terms of CHAIR and POPE metrics, with the exception of CHAIR recall. Additionally, a GPT-4V-based evaluation is conducted (following the LLaVA paper), along with assessments on other vision-language tasks (VQA) and ablation studies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful review and encouraging feedback. Thank you for recognizing the clarity and practical design of our approach, our emphasis on mitigating hallucinations, and the strong empirical support we provided. Below, we provide detailed responses to your comments:\n\n### Q1. Discuss the limitations and the scope of the metrics used in the paper.\nThank you for your suggestion and for pointing out recent developments in hallucination evaluation. CHAIR and POPE are widely adopted and reliable metrics for evaluating hallucination, and we believe they offer reliable assessments within their respective scopes. Nonetheless, we acknowledge their inherent limitations: CHAIR depends on a predefined list of object classes and synonyms, which may struggle to detect uncommon objects or very nuanced attributes. POPE’s reliability, in turn, can be influenced by the segmentation tool used to obtain ground-truth objects.\nWe will include a brief discussion of these limitations in the revised paper and also report results using ALOHa to complement our evaluation.\n### Q2. Use up-to-date automatic metrics such as ALOHa.\nThank you for pointing out the newer evaluation metrics. We will include them in discussion. We have incorporated ALOHa into our evaluation to better assess localizable object hallucinations. Specifically, we report both the object-level hallucination score ($ALOHa_0$) and the caption-level aggregated score ($ALOHa$). As shown in the table below, MARINE consistently reduces object hallucinations and outperforms baseline generations across all settings and metrics.\n\n| Method  | LLaVA-$ALOHa$ $\\uparrow$     | LLaVA-$ALOHa_0$ $\\uparrow$ | LLaVA-v1.5-$ALOHa$ $\\uparrow$     | LLaVA-v1.5-$ALOHa_0$ $\\uparrow$     | mPLUG-Owl2-$ALOHa$ $\\uparrow$     | mPLUG-Owl2-$ALOHa_0$ $\\uparrow$     |\n|-------|--------|------|--------|--------|--------|--------|\n| Greedy   | 40.1%  | 70.1%  | 61.9%  | 83.1%   | 70.2%  | 87.0%  |\n| MARINE   | 48.7%$_{+8.6}$   | 76.1%$_{+6.0}$  | 66.7%$_{+4.8}$  | 85.6%$_{+2.5}$  | 72.9%$_{+2.7}$      | 88.2%$_{+1.2}$ |\n\nNote: For implementation details, we use MSCOCO ground-truth captions as references and enable reference object detection for more localizable and generalizable object hallucinations detection. \n\n### Q2.1. Evaluation in addition to automatic metrics\nThanks for the suggestion. In Table 3 of our manuscript, we did include GPT-4v evaluation, which compares the outputs of two LVLM assistants using GPT-4V as a judge. This \"LLM-as-a-Judge\" evaluation protocol has become widely accepted as a reliable proxy for human evaluation, particularly when large-scale human assessments are costly.\n\nAlthough conducting extensive human evaluations is beyond the scope of our rebuttal timeline, we fully acknowledge its importance. We consider incorporating comprehensive human evaluations as future extensions of this research.\n\n### Q3. Is using synonym list a common practice with CHAIR?\nYes, this is standard practice in the original CHAIR paper [1] and its official implementation. Specifically, they use a synonym list from Lu et al. (2018) [2] to map words (e.g., “player”) to MSCOCO object categories (e.g., “person”).\n\n[1] Object Hallucination in Image Captioning\n\n[2] Neural Baby Talk\n\n### Q4. Suggested related work.\nThank you for your suggestion on this line of research on faithfulness evaluation for text-to-image generations, parallel to our focus on image-to-text generations. We gave the mentioned research a careful read and summarized them below. \n\nTIFA focuses on evaluating text-to-image generation by automatically generating and answering questions derived from prompts, measuring faithfulness along several categories (objects, actions, attributes). DSG (Davidsonian Scene Graph) also assesses text-to-image alignment but emphasizes a structured decomposition of prompts into atomic propositions. VQAScore introduces a VQA-based metric that better captures compositional semantics (such as object relationships and attributes), providing insights for attribute and relationship level object hallucination evaluation. Finally, Gecko addresses text-embedding tasks: it uses LLMs to synthesize query–passage pairs (and carefully select hard negatives) in order to train a compact yet powerful universal representation model.\n\nCAPTURE introduces a structured metric for evaluating detailed image captions by extracting and aligning objects, attributes, and relations across captions. We will include this in related work discussion in our next revision.", "claims_and_evidence": "The main claim of this paper is that using vision tools as guidance effectively and efficiently mitigates hallucination in image-to-text generation. This paper provides evidence for this claim (CHAIR/POPE results and latency analysis). However, I feel the choice of automatic metrics for hallucination detection can be improved (see my comments in “Methods And Evaluation Criteria”).", "methods_and_evaluation_criteria": "**Methods**\n\nThe proposed approach is clearly explained in this paper, and the motivations behind it seem reasonable. The design choices appear well-justified given the primary goals: flexibility (training/API-free), simplicity, and effectiveness.\n\n\n**Evaluation**\n\nI feel CHAIR is limited to evaluate VLMs that can generate long and detailed image captions/descriptions. CHAIR uses MSCOCO captions and their object annotations, but these captions are considerably shorter than what current VLMs can generate. POPE relies on a segmentation tool to obtain ground truth objects, and some negative sampling methods could make a single question too easy (e.g., random negatives could be too easy). \n\nAlthough hallucination detection metrics are beyond the scope of this paper, the scope of the metrics used should be clearly explained. Newer automatic metrics for hallucination such as ALOHa (https://aclanthology.org/2024.naacl-short.30.pdf) and Visual Fact Checker (https://arxiv.org/pdf/2404.19752) have been introduced. Including results from these newer and more robust metrics would strengthen this paper. In addition to automatic metrics, performing human evaluation to verify their robustness would also be helpful.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See my comments in the “Methods And Evaluation Criteria” section.", "relation_to_broader_scientific_literature": "A line of research on image-text alignment might be related. It would be nice to mention some key papers as related work (e.g., TIFA, DSG, Gecko, VQAScore). Also, newer metrics for detailed image captions such as CAPTURE (https://arxiv.org/pdf/2405.19092) could be related.", "essential_references_not_discussed": "See the “Relation To Broader Scientific Literature” section.", "other_strengths_and_weaknesses": "- Overall, this paper is well-written and easy to follow. Supplementary materials typically provide additional information to clarify ambiguous points in the main text.\n- Just to clarify my position, my only concern is the choice of automatic metrics. I had hoped that this paper would explain the limitations and scope of those metrics or use more up-to-date ones.", "other_comments_or_suggestions": "See my comments in “Methods And Evaluation Criteria”", "questions_for_authors": "- A.5 says “Besides, we employed the synonym list from Lu et al. (2018) to align synonymous words in the generated text with MSCOCO object categories.” Is this a common practice with CHAIR? If not, how does this affect the final numbers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "piofeM5fLH", "reviewer": "Reviewer_SDtF", "summary": "The paper presents a novel method called MARINE to reduce hallucination in large vision-language models (LVLMs).\n\nThe method can be applied to LVLMs without any training. When auto-regressively generating individual tokens, logits are computed twice: once with the normal LVLM input (\"unconditional\"), and once with an augmented \"conditional\" input containing tokens from visual guidance models (DETR and RAM++). The logits of the unconditional and conditional inputs are then combined for sampling the next token ( similar to classifier-free guidance in diffusion models).\n\nThe paper compares the method with 5 LVLMs against 5 different baselines and shows improvements of both CHAIR and POPE scores, as well as an improvement of caption metrics. MARINE is also less compute intensive than the baselines methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful and constructive review. We appreciate your recognition of MARINE’s effectiveness and comprehensive evaluation. We provide detailed responses to your questions below:\n\n### Q1. Effect of sampling temperatures.\nIn our paper, we opted for greedy sampling (temperature = 0) to ensure deterministic behavior in LLMs, eliminating randomness and thereby facilitating more reliable comparisons. This setup is also consistent with our main baseline VCD, and is widely used in existing benchmark evaluations to ensure reproducibility. \n\nHere, we conducted experiments using temperature = 0.6 and reported the mean ± standard deviation in the table below. As shown, MARINE consistently improves object hallucination metrics regardless of the sampling strategy.\n\n| Method | LLAVA-$C_s\\downarrow$  | LLAVA-$C_i\\downarrow$  | LLAVA-$Recall\\uparrow$  | mPLUG-Owl2-$C_s\\downarrow$  | mPLUG-Owl2-$C_i\\downarrow$ | mPLUG-Owl2-$Recall\\uparrow$ |\n|----|----|----|-----|-----|-----|-----|\n| Original | 26.1 ± 1.6  | 10.8 ± 0.5  | 46.0 ± 0.8 | 4.9 ± 0.6   | 2.8 ± 0.3 | 37.7 ± 0.6   |\n| MARINE (ours) | 19.3 ± 0.8$_{-6.8}$  | 7.6 ± 0.1$_{-3.2}$ | 50.6 ± 0.2$_{+4.6}$ | 4.5 ± 0.6$_{-0.4}$     | 2.4 ± 0.2$_{-0.4}$ | 41.1 ± 0.4$_{+3.4}$ |\n\n### Q2. Questions on guidance strength. Is mPLUG-OWL2 an outlier in Figure 3?\nIncreasing guidance strength generally improves model faithfulness across all evaluated models, indicated by a notable decrease in the CHAIR. However, the optimal guidance strength varies by model. mPLUG-OWL2 serves as example of the more advanced models that begin with an inherently lower CHAIR score, suggesting it captures visual information better than LLaVA. Thus, it benefits from guidance strengths other than strictly 1.0, where the visual guidance effectively complements rather than dominates its internal visual encoder. In contrast, earlier models like LLaVA rely significantly more heavily on the introduced visual guidance, as it can largely dominate their intrinsic visual grounding capabilities.\n\nHowever, excessively strong visual guidance can overall harm a model’s ability to follow instructions accurately. This negative effect is illustrated in Figures 9 and 13 of our manuscript, where a guidance strength of 1 reduces the quality of model generations for tasks beyond visual grounding. \n\nFor simplicity and consistency, we adopted a universal strength of 0.7 across experiments. Nonetheless, tuning this hyperparameter for each base LVLM using a validation set could yield optimized results tailored to each model. We will include this discussion in our next revision.\n\n### Q3.1. Why the intersection-based method outperforms the union-based method?\nThe intersection-based method retains only visual signals consistently grounded across different vision models, while the union-based method includes all signals, even conflicting or incorrect ones. Intersection outperforming union indicates that precision is currently more critical than recall for LVLMs. In other words, intersection reduces false positives from visual guidance, whereas union increases true positives at the cost of more false positives.\nFor example, LLaVA, one of the earliest LVLMs, is particularly prone to hallucinations with complex or noisy instructions and thus benefits greatly from intersection-based methods. Newer models, such as LLaVA-v1.5 and mPLUG-Owl2, are more robust but remain sensitive to partially incorrect inputs.\n### Q3.2 Why use different prompts in MARINE-intersec and MARINE-union?\nWe evaluated MARINE-union using the exact same prompt template as MARINE-intersec, which underperformed. We hypothesize this is because the original version provides more detailed information, whereas reusing the intersec-style prompt combined reliable grounding with potentially misleading false positives, thus weakening the visual guidance.\n| **Model** | **LLaVA** | | **LLaVA-v1.5** | | **mPLUG-Owl2** | |\n|-----|-----|----|-----|------|-----|------|\n| **CHAIR** | $C_S \\downarrow$| $C_I \\downarrow$ | $C_S \\downarrow$ | $C_I \\downarrow$ | $C_S \\downarrow$ | $C_I \\downarrow$ |\n| MARINE-union | **30.4** | 9.7 | **5.4** | **2.7** | **4.8** | **2.7** |\n| MARINE-union (new) | 32.6 | 9.7 | 7.8 | 3.9 | 6.2 | 3.5 |\n*New: Same prompt template as MARINE-intersec.\n\nNote: Greedy and Intersec results are reported in Table 7 of the main paper.\n\n### Q4. More recent works included related work but not baselines.\nThe more recent works were concurrent to our project development, and thus we included them in discussion but not empirical comparison. In the following, we additionally include \n\n### Q5. Typos\nThank you for the catch. We will ensure to correct them in our next revision.\n\n### Q6. What is the \"noise intensity\" for DETR?\nDETR outputs a confidence score for each object. We filter predictions using a threshold. lower thresholds allow more noisier detections, while higher ones yield fewer, more precise results. This threshold defines the noise intensity.", "claims_and_evidence": "1. The paper claims that the MARINE method is effective for mitigating object hallucinations in LVLMs.\n\nThe paper gives good evidence that the method indeed works well compared to previous methods.\n\nTheoretically, the method is based on classifier-free guidance, which was introduced in diffusion models (Ho and Salimans, 2021) and then adapted for text sampling (Sanchez, 2023).\n\n1a. The paper always uses greedy sampling, and previous works such as (Sanchez, 2023) have sampled at different temperatures. The paper simply assumes it works equally well with greedy sampling, without referring to previous literature, experimental evidence, or theoretical considerations.\n\n1b. In Section 5.3 (Line 428) the paper states that the best guidance strength is between 0.3 and 0.7. Figure 3 shows that object hallucinations decrease with increasing guidance strength for LLaVA, but not for for mPLUG-OWL2. It would be interesting to see how this affects other models, maybe mPLUG-Owl2 is an outlier. Figure 8 in the appendix shows that increased guidance strength improves captioning metrics on captioning tasks (I'm less convinced of the value of the captioning metrics on the LLaVA-QA90 task). Then the only evidence for guidance strength 0.7 being better than guidance strength 1.0 is Table 22 in the appendix. This seems somewhat scarce evidence for this central parameter (note that with guidance strength 1.0 the method reduces to something much simpler).\n\n1c. Ιn section 5.3 the paper claims that \"intersection-based method outperforms the union\" and this is based on Table 6. I find it surprising that using a single model only *increases* sentence-level hallucinations for LLaVA models. Note that Table 9 in the appendix lists different prompts for MARINE-intersec and MARINE-union. Which models to use and how to combine their output is also a central part of the method, so I think this should be explained in some more detail with better evidence.\n\n2. The paper claims that MARINE is training-free and compares favorably with existing methods. This claim is well supported by the fact that MARINE can be applied to a variety of models without re-training (Tables 1,2) and by the inference latency measurements (Table 5).", "methods_and_evaluation_criteria": "The benchmark datasets are appropriate. The method is evaluated on CHAIR and POPE (original MSCOCO, as well as A-OKVQA and GQA) to measure the mitigation of hallucinations.\n\nAdditionally, the method is evaluated on and GPT-4V-aided evaluation (Yin, 2023), which also measures hallucinations, but at the same time gives information about the usefulness of the outputs.\n\nFinally, the method is also evaluated on captioning metrics, again with the goal to verify that the output quality other than hallucinations does not deteriorate.", "theoretical_claims": "There are not theoretical claims in the paper. Note above comments about greedy sampling and guidance strength.", "experimental_designs_or_analyses": "The method is compared with four previous methods (LURE, Woodpecker, VCD, OPERA – all from 2023). This evaluation seems robust and fair.", "relation_to_broader_scientific_literature": "The work is well anchored in existing literature about object hallucinations in LVLMs and controllable generation.\n\nSection 2.1 mentions some more recent work from 2024, but all baselines (Section 5.1) are from 2023. Why is the work not compared to newer methods?", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths and weaknesses are discussed in above sections.", "other_comments_or_suggestions": "Typos:\n1. Line 102 (right column): \"studies(Li\"\n1. Line 137: \"2023a)and\"\n1. Line 223: \"InstructBLIP (Liu et al., 2023c)\"\n1. Line 382 (right column): experimental evidence for guidance strength is provided in Appendix B.4 (not C)\n1. Line 682: \"ROUGH-L\"\n1. Lines 360-362: missing spaces before \"(\"\n\nQuestions:\n1. Line 262 (right column): what is the \"noise intensity\" for DETR?\n1. Line 264 (right column): why only greedy sampling? in (Sanchez, 2023) different sampling temperatures were explored, but the method was never applied with greedy sampling\n1. Lines 716-748 (Table 9): why use different prompts in MARINE-intersec and MARINE-union?\n\nNits:\n1. Lines 289: consider mentioning that these results are on MSCOCO – this would make it clearer how Table 2 relates to Table 4\n1. Line 1110: awkward placement of orphan line", "questions_for_authors": "See above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FlOJL0bBg", "reviewer": "Reviewer_R69G", "summary": "The paper proposes the MARINE method for mitigating object hallucination in LVLMs. The method uses results from external object detection models and adds it in the form of an extra textual prompt into the LVLM’s generation. The method is compared with several baselines on object hallucination benchmarks, as well as on VQA and image captioning tasks. Several ablation studies are also presented in the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review and valuable feedback. We sincerely appreciate your recognition of the simplicity and thorough experimental validation of our MARINE approach.\n\n### Q1.1 Clarify the claim regarding misalignment.\nIn the original claim, by \"visual encoder,\" we referred to the LVLM's visual encoder component (e.g., CLIP), highlighting that MARINE supplements its initially insufficient visual context. The mentioned misalignment issue refers that LVLMs typically employ a trainable linear alignment layer, which can potentially lose or distort important object-level information during the mapping from visual to textual representations. In contrast, MARINE directly extracts and utilizes explicit object-level details, thereby preserving critical visual information intact. We will clarify this point explicitly in the revised manuscript.\n\n### Q1.2. Clarify the claim regarding access to advanced models.\nThank you for pointing this out. We will clarify that MARINE inherently requires access to additional vision models to enrich the visual inputs in our revision. Additionally, the \"advanced LLMs\" that we referenced refer to closed-source models that are accessible only via paid APIs (e.g., GPT-4o), while the vision models are open-source. We will explicitly state this distinction in our revision.\n\n### Q2. Could you explain the concept and importance of \"MARINE-Truth\" clearly in the main paper?\nWe consider using the ground-truth object list as a variant of MARINE and denote it as MARINE-Truth. The performance of MARINE-Truth serves as a reference to MARINE’s best performance. However, the ground-truth object list may also contain noise and therefore occasionally underperforms MARINE. We will explain this in our revision.\n\n### Q3. In your latency measurements, why did you exclude the latency of additional forward passes through external models required by MARINE?\nThank you for raising this point. To clarify, we did include the latency of additional forward passes through external vision models in our measurements. These external vision models have negligible inference overhead compared to autoregressive models (e.g., LLMs, LVLMs). To illustrate clearly the impact of including or excluding the latency introduced by these external models, we present a detailed latency comparison in the table below. Specifically, the table includes scenarios both excluding visual prompt generation latency (Offline MARINE) and including it (Online MARINE):\n|   | Greedy  | LURE  | Woodpecker*  | VCD  | OPERA | **Offline MARINE**   | **Online MARINE**  |\n|------|-----|-----|-----|-----|------|------|------|\n| **Training Cost** | 0  | 10min on A100 80G | 0  | 0 | 0 | 0 | 0  |\n| **Inference Latency (ms/token)** | 26.3 (×1.0) | 179.9 (×6.84) | 94.5 (×3.59)* | 53.4 (×2.03) | 185.1 (×7.0) | **52.2 (×1.98)** | **52.23 (×1.985)** |\n*Woodpecker requires GPT API key access, and the latency may depend on OPENAI API.\n\nAs shown, processing images via additional vision models adds only a negligible overhead to the overall latency. \n\n### Q4. Can you compare MARINE against other SOTA VLM methods, especially BRAVE?\nThank you for pointing out these related works, we will include them in discussion in our next revision. Especially, BRAVE indeed shares a very similar intuition as ours for ensembling diverse visual information sources to improve model faithfulness, confirming the motivation of our work. Here, we compare MARINE against BRAVE on the POPE benchmark. As shown below, MARINE achieves comparable performance than BRAVE, while introducing no additional trainable parameters.\n\n| Model    | Total Params | Trainable Params | Rand  | Pop   | Adv   | POPE$_\\text{avg}$ |\n|--------------|--------------|------------------|-------|-------|-------|-------------------|\n| LLaVA-v1.5 (7B)   | 7B  | 7B      | 87.3  | 86.1  | 84.2  | 85.9|\n| BRAVE    | 10.5B| 3B      | –     | –     | –     | 87.6     |\n| **MARINE (ours)** | 7B       | 0  | **87.9** | **86.5** | **86.7** | 87.0 |\n\n*Note: We report BRAVE’s POPE$_\\text{avg}$ score as stated in their paper. Their model and detailed evaluation results have not been open-sourced. For fair comparison, we adopt the same test set as LLaVA-v1.5.*\n\n### Q5: Labels in figures.\nThank you for catching these. We will update them in our next revision.", "claims_and_evidence": "The paper makes the following claims:\n\n1. “MARINE mitigates insufficient visual context provided by the visual encoder and misalignment between the vision and text domains”: I think this claim is too general. First, MARINE cannot be used with an arbitrary visual encoder but rather with object detection models. Second, it is unclear to me where in the experimental section the misalignment between the vision and text domains is investigated and which experiments demonstrate that MARINE mitigates it. \n\n2. “MARINE does not require additional training resources or access to advanced LLMs.”: This is true but it instead requires access to advanced object detection models, which should be made more clear in the paper.\n\n3. “MARINE outperforms the baselines in hallucination mitigation while maintaining overall performance across multiple tasks (image captioning, VQA)”: I believe that this claim has been demonstrated in the experimental section, though there are cases where MARINE does not outperform the baselines (eg in Table 1 and 2).", "methods_and_evaluation_criteria": "While the chosen benchmarks make sense, it would be great to compare MARINE against more powerful VLMs that similarly incorporate more fine-grained visual information, like SILC: Improving Vision Language Pretraining with Self-Distillation, ECCV 2024 and BRAVE : Broadening the visual encoding of vision-language models, ECCV 2024. Especially BRAVE is conceptually similar to MARINE in that it incorporates information from multiple encoders.", "theoretical_claims": "I do not understand what MARINE-Truth is and neither can I find any details in Appendix A as stated in the main paper. I also do not understand the point of this part. Please explain in the main paper what it is and provide a better explanation on why it is important to look at it.", "experimental_designs_or_analyses": "1. In Appendix, you state that “For decoding methods such as VCD, OPERA and our method, we measured the latency of LLaVA generating captions directly”. I think this is unfair, as MARINE additionally requires forward passes through multiple external models which in my opinion should be counted towards the latency calculation.\n\n2. See also my comment above regarding the misalignment analysis (under claims).", "relation_to_broader_scientific_literature": "Mitigating object hallucinations is an active area of research.", "essential_references_not_discussed": "See my comment above about SILC: Improving Vision Language Pretraining with Self-Distillation, ECCV 2024 and BRAVE : Broadening the visual encoding of vision-language models, ECCV 2024.", "other_strengths_and_weaknesses": "1. The novelty of the paper is a bit limited and the claims made in the introduction are too general. In particular, I would like the authors to rephrase the paper and remove any occurrences where it is claimed that information from a general visual encoder is used because this is not true (at least it is not shown in the paper). In addition, MARINE is only useful when used on images where the pretrained object detectors give meaningful outputs, which should also be stated in the paper.\n\n2. Despite the limited novelty, I find the simplicity of the method and extensive experimental evaluation a strength.", "other_comments_or_suggestions": "Please label y axis and change colors in Figure 3, 8, 14 and 15 as the lines are hard to distinguish.", "questions_for_authors": "Please see my questions in the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N3WBGrSvUu", "reviewer": "Reviewer_GyxS", "summary": "This paper proposes the MARINE framework to address the object hallucination issue in Large Vision-Language Models (LVLMs). This framework introduces visual guidance from image-grounded models to effectively reduce hallucinations during inference. Experiments show that MARINE outperforms baseline methods on multiple LVLMs and balances latency and accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and acknowledgement of our extensive experiments and the overall clarity and structure of our paper. We detail our response as follows.\n\n### Q1: Using the DETR trained on MSCOCO may be unfair.\nMSCOCO (train) is a widely-used open-source image-caption dataset frequently utilized for pre-training various vision encoders, including CLIP, the backbone of current LVLMs (e.g., LLaVA). While all vision encoders leverage data that includes MSCOCO training split, the difference in how to process and leverage these information determines model performance. This variability is evident in MARINE’s superior performance compared to LVLMs utilizing only the CLIP encoder. Additionally, our manuscript's Table 6 illustrates instances where MARINE with only DETR underperforms compared to MARINE employing only RAM++. Nonetheless, aggregating information from multiple visual encoders (as in MARINE) consistently achieves the highest performance.\n\nMoreover, the vocabulary derived from MSCOCO effectively encompasses frequent objects common across diverse natural-image datasets, demonstrating strong generalizability to other evaluation data. **This is confirmed by the substantial improvements in POPE results on A-OKVQA shown in Table 15**, highlighting our method's capability to generalize effectively and outperform baselines beyond MSCOCO.\n\n### Q1.1: Comparison with VCD.\nWe acknowledge that VCD is a valuable work, but our approach addresses the hallucination problem from a different and effective perspective. VCD aims to reduce LVLMs’ over-reliance on language priors from LLM pre-training data by contrasting distorted visual inputs. In contrast, MARINE focuses on hallucinations arising from insufficient visual context. Thus, VCD and MARINE approach hallucination from complementary angles, and integrating both methods has the potential to achieve further performance improvements.\n\n### Q2: Relevant works on feature steering for hallucination reduction.\nThank you for pointing to these relevant lines of research on feature steering. We summarize them as follows and will include the discussion in our next revision.\n\nNullu identifies a “HalluSpace” by comparing truthful and hallucinated features, then projects model weights to the null space of those hallucination-prone directions, reducing object hallucinations with no extra runtime cost. VTI (Visual and Textual Intervention) learns “shift vectors” by analyzing how vision features change under corruption and how text features differ between hallucinated and correct outputs. It then applies these shifts at inference to stabilize LVLMs and reduce hallucinations. \n\n### Q3: More in-depth explanation of how this method differs from direct prompting.\n\nWe clarify key conceptual and empirical differences between MARINE and prompting-based methods, which we believe directly address this concern:\n- Direct prompting relies solely on the model’s textual instruction-following capabilities, which can exacerbate hallucination issues in less capable models (e.g., LLaVA). Conversely, MARINE directly enhances visual understanding by integrating additional visual information, making it effective even for weaker models without incurring training overhead.\n- Prompting methods cannot introduce new visual data and thus remain constrained by the original vision model’s capabilities. MARINE introduces novel visual information through an improved vision encoder, fundamentally enhancing the model's observational accuracy.\n- Direct prompting and additional prompt methods require careful crafting and tuning specific to each task or dataset. MARINE, however, exhibits strong generalization capabilities across diverse models and datasets, effectively reducing hallucinations and improving reliability without additional training or manual prompt optimization.\n\nAppendix B.1.1 further elaborates on these distinctions. In particular, we evaluated the direct prompting baseline using a highly detailed instruction explicitly guiding the model to describe only observable visual characteristics\n\nAs shown below, MARINE consistently outperforms the prompting baseline by significantly reducing hallucinations while maintaining or improving recall. Although prompting can improve recall in some cases, it often worsens hallucination metrics. MARINE achieves better overall reliability across all models.\n\n| Method  | LLAVA-$C_s\\downarrow$ | LLAVA-$C_i\\downarrow$ | LLAVA-$Recall\\uparrow$ | LLaVA-v1.5-$C_s\\downarrow$ | LLaVA-v1.5-$C_i\\downarrow$ | LLaVA-v1.5-$Recall\\uparrow$ | mPLUG-Owl2-$C_s\\downarrow$ | mPLUG-Owl2-$C_i\\downarrow$ | mPLUG-Owl2-$Recall\\uparrow$ |\n|-----|------|------|-----|------|-----|-----|-----|-----|------|\n| Original  | 26.6  | 10.5 | 47.4 | 8.8 | 4.6 | 41.1 | 6.2 | 3.4 | 38.8 |\n| Direct Prompting | 27.2| 11.0| 46.4| 19.6 | 8.3  | **52.3**  | 9.0  | 5.1  | **42.0** |\n| **MARINE** (ours)  | **17.8**  | **7.2**  | **50.8**  | **6.2** | **3.0** | 44.3| **4.2** | **2.3** | 41.4|", "claims_and_evidence": "Most claims are supported by clear evidence. The authors conduct extensive experiments on five LVLMs using multiple metrics like CHAIR, POPE, and GPT-4V-aided evaluation. They compare MARINE with various baselines and perform ablation studies. \nHowever, the experiments are limited to specific datasets and tasks, which can be problematic. I will give my reason in the Methods And Evaluation Criteria section.", "methods_and_evaluation_criteria": "Overall, the paper is well-written and easy to follow. The technical routing makes sense to me. My primary concern is implementing the vision models, such as DETR. As far as I know, both CHAIR and POPE contain the samples selected from MS COCO, and using the DETR  trained on MS COCO can surely improve the method's performance. In this case, the results on CHAIR and POPE can be somehow unfair. Note that the compared method, VCD, does not introduce additional information and achieve comparable performance. \n\nCan the author provide more explanation about this point?", "theoretical_claims": "The theoretical claims are right.", "experimental_designs_or_analyses": "See Evaluation Criteria.", "relation_to_broader_scientific_literature": "The paper's key contribution to MARINE in mitigating object hallucination in LVLMs builds on using knowledge from vision models. MARINE offers a training-free and API-free approach. By leveraging image-grounded models, the root causes of hallucination are addressed well.", "essential_references_not_discussed": "I think most of the listed related works are essential for understanding this paper's contributions. While there are some other works in Large vision language models (LVLMs) that seem to apply feature steering to mitigate hallucinations, such as: \n\n[1] Reducing hallucinations in vision-language models via latent space steering, 2024.\n\n[2] Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection, 2024.\n\nThese are suggested to be discussed in related works.", "other_strengths_and_weaknesses": "Generally, the paper is well-written and easy to follow, and the results seem to be good.\n\nUsing 2D features from pre-trained models to prompt downstream tasks such as 3D detection and 2D few-shot detection is not a new concept. It would be beneficial to have a more in-depth explanation of how this method differs from direct prompting.", "other_comments_or_suggestions": "I am willing to adjust the score if the issues are satisfactorily addressed.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes a method to reduce hallucinations in large VLMs. The work received initially positive scores (3,3,3,4). The authors submitted a rebuttal, which helped clarify some of the comments provided by the reviewers. Eventually, the reviewers fully leaned towards accepting the work based on the novelty of the idea, the clarity of the presentation and the effectiveness of the provided results. The recommendation from the AC is to strong accept this work for the conference.
Decision: Accept (spotlight poster)