PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training
Authors: Cong Chen
Mingyu Liu
Chenchen Jing
Yizhou Zhou
Fengyun Rao
Hao Chen
Bo Zhang
Chunhua Shen
Venue: ICLR 2025 Spotlight (2025)
Keywords: Multi-Modal Large Language Models
Hallucinations Mitigation
Hallucinations Evaluation
Language Model Priors
Primary area: foundation or frontier models, including LLMs

Abstract: This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs)  particularly for dense image captioning tasks. To tackle the challenge, we identify the current lack of a metric that finely measures the caption quality in concept level. We hereby introduce HalFscore, a novel metric built upon the language graph and is designed to evaluate both the  accuracy and completeness of dense captions at a
granular level. Additionally, we identify the root cause of hallucination as the model's over-reliance on its language prior. To address this, we propose PerturboLLaVA, which reduces the model's reliance on the language prior by incorporating adversarially perturbed text during training. This method enhances the model's focus on visual inputs, effectively reducing hallucinations and producing accurate, image-grounded descriptions without incurring additional computational overhead.  PerturboLLaVA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations and achieving improved performance across general multimodal benchmarks.

Published as a conference paper at ICLR 2025
PERTURBOLLAVA: REDUCING MULTIMODAL HALLU- CINATIONS WITH PERTURBATIVE VISUAL TRAINING
Cong Chen'
Mingyu Liu
Chenchen Jing',
Yizhou Zhou?,
Fengyun Rao",
Hao Chen',
Bo Zhang', Chunhua Shen 3.1
Zhejiang University
WeChat Group
Zhejjang University of Technology
ABSTRACT
This paper aims to address the challenge of hallucinations in Multimodal Large Language Models (MLLMs) particularly for dense image captioning tasks. To address the challenge, we identify the current lack of metric that finely measures the quality of the caption at the concept level. We hereby introduce HalFssore, novel metric built upon the language graph that is designed to evaluate both the accuracy and completeness of dense captions at a granular level. Addition- ally, we identify the root cause hallucination the model's over-reliance on its language prior. To address this, we propose PerturboLLaVA, which reduces the model's reliance on the language prior by incorporating adversarially perturbed text during training. This method enhances the model's 's focus on visual inputs, effectively reducing hallucinations and producing accurate, image-grounded de- scriptions without incurring additional computational overhead. PerturboLLa VA significantly improves the fidelity of generated captions, outperforming existing approaches in handling multimodal hallucinations and achieving improved per- formance across general multimodal benchmarks.
INTRODUCTION
Multimodal large language models (MLLMs) (Liu et al., 2024b; Zhu et al., 2023; Dai et al., 2023;
Bai et al., 2023; Zhang et al., 2023; Lu et al., 2024a: Zhou et al., 2024) have achieved significant strides in complex visual tasks by integrating the world knowledge and reasoning capabilities of large language models (LLMs). Nonetheless, hallucination (Liu al., 2023a; Zhou et al., 2023; Yin et al..,, 2023; Wang et al., 2023a) issue persists in these models, and even the most capable multimodal models often respond with texts that do not accurately reflect the provided visual content. Notably, scaling up model parameters and training data has not proven effective inmitigating this issue for MLLMs, unlike for their unimodal language counterparts.
In this paper, we focus on addressing the problem of hallucination in the context of dense image captioning (Liu et al., 2024b; Chen et al., 2023; 2024), which requires a comprehensive and de- tailed description of every aspect of image. Dense image captioning exposes hallucinations more acutely, as models must generate rich and detailed captions for complex scenes while ensuring fi- delity to the visual content. Hallucinations undermine the reliability of MLLMs in applications that require precise and faithful visual descriptions. To tackle the hallucination challenge in MLLMs, we recognize the need for robust quantitative metric that accurately reflects caption quality regarding hallucination. Hence, we introduce HalFscore, a novel metric providing a more granular and com- prehensive evaluation of hallucinations specific to dense captioning. HalFscore measures both the accuracy and completeness of dense captions by identifying the incorrect elements and assessing the missing details, offering a balanced view of the model's performance. To achieve this, we propose to build the language graph that captures the main concepts along with their relationships. and compute its discrepancy against the ground truth. HalFsscoe aggregates the precision and recall model the accuracy and completeness of dense captioning results. Compared to previous hallucination scores (Li et al., 2023b; Guan et al., 2024), the proposed HalFscore offers a more fine- ane-grained and holis- tic evaluation, serving
valuable guide when developing the hallucination suppression method. We further analyze the common cases plagued with hallucinations, and conjecture that the issue in MLLMS comes from the over-reliance the model's pretrained linguistic knowledge. As shown

---

Published as a conference paper at ICLR 2025
image white lcying comfortably rescing top of black backpack. edrition to the cat and the back eece,,cc, there background, and couple books can be seen placed neartyy. person is also present in the cet scene, asendding close possibly observing interaisting the feline.
image
comouter satup, inouddng monitor, keyboard, and a mouse, or the dess, there is also a pair of scissors keybuard.
yynn nest there are several books acottered around the dest The presence of these bookss suggests that the person usind
miahe letoure reading.
worl,,
The imrgge feotures close up view of a partially eater: pizza a dining table The pizza including musarooms, hem. and cheese A fork can be seen resting on the pizza, indiscting thet somcone
The imagee depices a busy, city ac night, with multipie trafite lights and serest liahts illuminating the at various around the incersection, ensuring clear visibiliyy for drivers and pedestriars. There train cassing adding hureling atmossphre scone. Severdi cars and bus are also present in the arez, contributing to the nighttime
losations
been enjoying
there linffe ploced the cable, likely used for cucting and serving the pizac.
Figure I: The multimodal model is prone to hallucinate text due to the inherit language bias. Here, the hallucinated text is induced by the preceding concepts generated by the language model.
in Figure 1, as more texts are produced, the multimodal model gradually deviates from multimodal generation unimodal mode, which favors text generation based on the preceding language pat- terns while overlooking the actual visual information. For example, when presented with an image of green banana, model has the tendency to describe yellow because of the common knowl- edge that ripe bananas are typically yellow. In fact, concurrent MLLMs are obtained by continuous training from a pre-trained L.LM and are generally equipped with a strong language bias.
Inspired by this, we propose simple and effective training strategy that reduces the model's heavy dependence on the language prior by incorporating adversarially perturbed text during training. Specifically, we introduce carefully designed perturbations that aligns with the general knowledge but conflict with the visual content, intentionally misleading the model based on its language prior. For example, might introduce the perturbation "As bananas ripen, their color gradually turns yellow" before asking about the color of a green banana in the image. This perturbative training enforces the model to scrutinize the image content when predicting every token, rather than hal- lucinating contents from the text hints. Essentially, our method adjusts the model's conditional distribution depend more heavily on the image and less on the perturbation text, which leads to more robust multimodal capability.
As opposed state-of-the-art methods that resort more advanced decoding strategies (Leng al., 2024: Huang et al., 2024), the proposed method, PertuboLLaVA, effectively suppresses the halluci- nations in MLLMs without incurring additional training or inference costs, making it more suitable for real-world applications. Figure 17 shows our method can describe rich image details with less hallucinations. On the other hand, our method much more efficient, scalable, and easier to adopt compared to RLHF-based methods which require additional human preference data and incur sub- stantial training overhead and complexity. Additionally, we find the proposed method beneficial to general multimodal abilities, bringing boosted performance across all the multimodal benchmarks.
To summarize, our contributions are two-fold. First, we introduce a more principled metric com- puted on the language graph, serving as comprchensive hallucination measure. Second, we identify the root cause of hallucinations MLLMs as its inherent language bias, and propose perturbative visual training, enhancing the model's 's' focus on visual content during training. The proposed method integrates seamlessly into existing training pipelines, introducing minimal additional cost. It pro- vides scalable, efficient solution enhance multimodal models' visual understanding capabilities, excelling over prior compared to state-of-the-art methods across multiple dimensions.
2 RELATED WORK
2.1 MULTIMODAL FOUNDATIONAL MODEL
With the rapid advancement of large language models (Touvron et al., 2023a:b; Chiang et al., 2023; Bi et al., 2024), researchers leveraging their knowledge and reasoning abilities to build multi-
2

---

Published as a conference paper at ICLR 2025
VCD
of
One Both positioned
helsing tunnis rachets and ane eccively engaged the left side the court, while the other women is on the right side. There are several chairs placed around the court, likely for spectators or players to rest during preaks. A sports ball, presumeply tennis ball, con be seen rear the right side of the court.
Gioloié
OPERA
Image feabures women pleying gome tennis on court. women closer towards the left side of the
appears while the other woman positioned more towards the right side. There are several chairs placed around the court, likely for spectectors or playere to recc during brocks. A sports ball; presumably a tennis rock, con be seen in the middle of the court, indicating that the game is in progress.
RLAIF-V
Input Image
ternis They buth neaching for tennis ball that's presant on the cuurt, showing their active participation in the game. Each of the players has tonnis rachec in hand, highligheing the onguing match, There is a cheir lucated et the side of the cours, possibly for resting or sicting during ng breates, The overall scens captures the incensity and focus of the women's tennis masch.
two famale
players
Instruction:
Please descrize this image in decail,
PerfurbolloVA (Ours)
playing hodminton gomes court during the Rio 2016 Olympics. One weman on the left side of the court, while the other is on the right side. Buth women are hulding sudminton rocletts, with une racket closer tu the left wornnn and the other racket closer to che right women sports ball visiibe the air at the righs sidd:The the court appears so green and weel--maintainst the women are fully engaged
featurre two
game, showcasing their athletic abilities.
Figure 2: Comparison against the state of the art methods. Hallucinations are highlighted in red, whereas the image detailed descriptions are shown in blue. The proposed Perturbol.Laa describes rich image details more accurately.
modal systems for complex tasks. These models are typically constructed with a pretrained vision encoder to process visual information, language model backbone responsible for reasoning, and projector to map the visual data into textual space. The training process multimodal model is gen- erally divided into two stages: pretraining, using image captioning data for modality alignment, and instruction fine-tuning, using question-answer data to enable task handling. Several leading open- source projects (Bai et al., 2023; Lu et al., 2024b; OpenGVLab, 2024) have amassed large multi- modal datasets and developed high-performing models using advanced language model backbones and vision encoders. However, hallucinations remain a persistent challenge in models" outputs.
2.2 EVALUATION OF HALLUCINATION
Various benchmarks assess hallucination in MLLMs, divided into, categorized into close-ended (Li et al., 2023b; Wang et al., 2023b) and open-ended tasks (Sun et al., 2023; Liu et al., 2023b). Close-ended tasks use yes-or-no or multiple-choice questions to test for hallucinations, focusing on accuracy. The POPÉ(Li al., 2023b) benchmark detects non-existent entities, while AMBER Wang et al. (2023b) also considers attributes and relationships. In open open-ended tasks. such as image captioning or free-form Visual Question Answering (VQA) (Wu et al., 2017: Jing et al., 2020), LLM- free metrics like CHAIR (Rohrbach et al., 2018) 8) measure the ratio of hallucinated to actual objects in responses. On the other hand, LLM-based metries, such as MMHalBench (Sun al., 2023) and GAVIE (Liu et al., 2023b), utilize external LLMs like GPT (Achiam et al., 2023) to assign scores to generated responses and are used in benchmarks. Hallucination evaluation multi-modal models more evident in open-ended tasks, as these tasks require a detailed understanding of the image and dense outputs. However, current metrics like object-level CHAIR and caption-level MMHalBench lack fine-grat fine-grained hallucination analysis.
2.3 MITIGATION OF HALLUCINATION
Current efforts to mitigate hallucinations are mainly divided into training- based and decoding- strategies (Leng et al., 2024; Huang et al., 2024). Mainstream training-related approaches (Yu et al., 2024; Sun et al., 2023) introduce algorithms like RLHF (Ouyang et al., 2022) and DPO (Rafailov et al., 2024) from the LLM area into multimodal models. By constructing hallucination preference data, they train a reward model to provide reward supervision or use DPO to reduce multimodal hallucinations. However, the primary challenge with training-basse approaches lies in the substan- tial computational overhead, as they necessitate training additional reward models or incorporating extra training phases. Decoding strategies include approaches like OPERA (Huang et al., 2024),
3

---

Published as a conference paper at ICLR 2025
Matchings
VLM Caption
Hallucinations
Omissions
VLM Graph
M
O
GT Caption
Compare
VLM Caption
GT Caption
GT Graph
Figure 3: The diagram of computing HalFscore. We construct the language graph to model both the concepts and their relationships for captions. We can then compare the graphs and identify the hallucinations, omissions and matchings between the two sets of concepts respectively.
which corrects abnormal attention map, and methods like VCD (Leng et al., 2024) that decouple language priors causing hallucinations, subtracting them from prediction probabilities. Although decoding strategies have the advantage of being training-free, they do not address the root cause of hallucinations in multimodal models, as these issues originate during training. Moreover, from practical standpoint, the inference cost for large models often exceeds the training cost, since models are trained once but deployed countless times. In the work, we propose a novel, simple yet effective training strategy that avoids the additional training overhead of previous training- based approaches while offering more comprehensive solution.
3 HUALLUCINATION F-SCORE
An effective hallucination metric should be both fine-grained and comprehensive. Current metrics fall short of these criteria. For example, CHAIR focuses on matching objects while failing to mea- sure hallucinations about attributes and relationships. MMHalbench uses GPT-4 to produce single holistic score but lacks detailed analysis. Moreover, prior metrics only measure the degree of hal- lucination without assessing the comprehensiveness of the image captioning, typically favoring short but confident answer, which inconsistent with users' subjective experience.
Fine-Grained Comprehensive Hallucination--Mtricc We introduce HalFscore which mea- sures both ballucination and completeness of dense captions with fine granularity. As illustrated in Figure 3, we construct graphs for both the MLLM's output and the ground truth. Here, we leverage dense annotations ground truth, which provides sufficient detailed descriptions that reflect all the aspects of the input images. Specifically, we selected 1,000 images from the Densely Captioned Images (DCI) dataset (Urbanek al., 2023), in which images manually annotated and densely captioned. By comparing the graphs, we can identify ballucinations-coneee conceets generated by the model that contradict the ground truth, and ommisions--onneee sent in the model's 's captions. We denote the concepts denoted in the generation as Cgen. the concepts corresponding to ground truths as Cgt, and compute the precision and recall
present in the ground truth but ab-
Cgen n Egal
Challusinated
Precision
1
(1)
(Cgeal
|Cgen|
Cgen n Cgrl
[Coiited
Recall
leg.l
[Cgi
Here, the precision reflects the hallucination degree, whereas the recall assesses how well the cap- tions cover image details. Then we compute HalFscore by aggregating these two scores, serving as a single metric that reflects the overall captioning quality:
Precision Recall
HalFscore 2
(3)
Precision Recall
4

---

Published as a conference paper at ICLR 2025
Graph Computation To derive the above HalFscore, graph construction and matching pivotal. We propose novel triplet data structure to represent the information for cap- tions. Specificallyy, we prompt the GPT-4o model to parse the concepts along with their relative relationships, and represent them by the triplet representation, i.e., (e1, 62. T12). As shown Figure 4, the triplet representation can well model general and rich informa- tion, such as relative relationship between in- stances, e.g., (clock, on, wa 1), or the at- tribute description for a single instance, e.g., (mir (mirror, is, pink). In this way, the caption is transformed to a set of triplets, serving as a structured representation.
Caplion
The image portrays woman steaking hor phone as she ats in frant a white tab o. On the table. there is pink mirrr to the right and bubole tea to the left. Bchind the table, there armchair, Ihere clsp
Triplets Extraction
Graph Construction
Stear atg
Woman
Phone
Miror
Pink
on
In front of Woman
Construct Graph
Clock
Woll
Table
Triplets 1e e2,
Graph
Figure 4: Graph construction. We extract triplets from the caption and build the graph accordingly.
Meanwhile, the triplet can be viewed as two nodes representing entities, C1 and 62 with a relation edge We then organize the extracted triplets into a concept graph by matching entity nodes. In this graph, nodes represent entity concepts e;, and edges represent relational concepts rij between entities e, and eş. This graph allows us to represent all the information the caption comprehen- sively and accurately. Formally, we obtain the graph 9 with nodes V and edges & as follows:
9 = EV, V (e,,e,..... en): 8 { F23, .... J.nn..
(
By matching the constructed graphs, we can identify the hallucinated concepts (Challacinatted) and onit- ted concepts (Cominea) using GPT-4o. Based on these identified concepts, we proceed to calculate precision and recall according to the equations aforementioned, and then derive the Fscore. Please see the Appendix A.8 for detailed GPT-4o prompts used for triplet extraction and graph matching.
4 MITIGATION OF HALLUCINATION VIA PERTURBATIVE VISUAL TRAINING
4.1 PERTURBATIVE VISUAL TRAINING
To mitigate the over-reliance on language priors in multimodal models, we introduce a novel train- ing framework that introduces adaptive, context-specific perturbations in the textual inputs during training. This approach simulates the effect of language priors and forces the model to adjust its responses based on visual data rather than textual biases.
Specifically, during the instruction tuning training, the multimodal model is tasked with predicting the text according to the input image-question tokens (1,z,). We introduce a contextually adaptive perturbation text Tp. crafted to mimic misleading language priors. Thus, we obtain the perturbed input tokens as Ep, zg)), as depicted in Figure 5. This perturbation is integrated seamlessly as part of the input, without any direct loss computation model's original training regime. The purpose of introducing the perturbation text Zp is to induce errors by tempting the model to rely on the perturbative text. This compels the model to focus more on the image content to generate correct responses, thereby reducing its dependency on linguistic cues that are not supported by visual evidence.
xp itself, thus maintaining the integrity of the
Perturbation Text Design To ensure effectiveness and naturalness of the perturbations, we adhere to the following principles. 1) Contextual relevance. the perturbation is expected to be contextually relevant to the image content such that it appears to be plausible but misleading. 2) Aligument with pretrained knowledge. The perturbations are designed to resonate with common language priors, ensuring that they are realistic and reflect potential model biases. 3) Semantic variation. We ensure a diverse range of perturbations by varying the structure and thematic elements of Zp aligning them with common misconceptions or biases. In practice, we use GPT-4o to generate the perturbation text. The GPT-4o model views the image, question and answer, and is instructed to construct strong and diverse perturbations based on the world knowledge as well as certain image details, without disclosing the answer. The GPT-4 instruction prompt is detailed in the appendix.
5

---

Published as a conference paper at ICLR 2025
Perlurbation Text Generation
<hput image>:
<Question>; Who is silling on the molorcycce?
Human Prompt: Your task to subity and strategicall midead the mutlinddl model cralling responses seem plausible, conf logical are fundamentally insomect... Acccorin.. <input Image> and <Orlgin Questions.
Generated Perturbation Text: Motorcycles, espectally large crubers are designed for human ridors, requitng hands for stearing and foot for control. Biker attire like loother jackets gopgles strongly associated humars for both protection and
Before Perturbative Training
Simulated Longuage Prior (Strong)
<inpul Image? <Generafed Perlurbalion Text> <Quesfion>
Model
<Answer>(A man)
Image Abilly(Weok))
<Answer>(A dog)
Loss Feedbock
After Perturbative Training
Simulated Language Frior (Wook)
<tppuf image> <Generated Perturbation Text> <Question>
Model
<Answer>(A dog)
Image Abilly (Shong)
Infernece
Inherent Language Prior (Weak)
<Input image> <Question>
Model
<light Answer>
Image Abilly (Shoong)
Figure 5: The generation of perturbation text and perturbative visual training of PerturboLLaVA.
Model
No extra data generation
No extra training stage
Inference cost
Data scalability
Ours
1x
RIAIF-V
X
1x
/
OPERA
V
2~5x
X
VCD
/
2x
X
Table 1: Comparison of method features with other representative works.
Our approach offers significant benefits over prior approaches as summarized in Table 1. Compared to the training method like RLAIF-V, we avoid the need to collect costly preference data or train an additional reward model. Our method is easy implement and incurs minimal additional training cost to the multimodal model, making it nearly "free lunch" solution. The quantification of the additional training overhead is provided in the Appendix A.3. Compared to contrastive decoding strategies, our approach more fundamentally mitigates the multimodal model's 's excessive reliance on language priors, without introducing any additional inference overhead. Moreover, the effectiveness of our method scales with the quality and diversity of the perturbation texts.
4.2 MATHEMATICAL EXPLANATION
Our method can be further understood and explained from a mathematical perspective. In a more general sense, it can be interpreted introducing additional noise or perturbation during the training phase to develop a more robust model. This approach aligns with the ideas proposed in the seminal work (Clark et al., 2019). Thus, leverage the mathematical reasoning from that work to explain our approach. Similarly, in multimodal tasks, we define the following: Iy is the k-th token predicted by the model, x<c denotes the preceding tokens in the output, refers the corresponding image, rk indicates the prediction based solely language priors, while x2ck is the prediction made by

---

Published as a conference paper at ICLR 2025
the multimodal model without the influence of language prior. Thus have
((TNEEER 1) - p(xx <k Fzk:
(5)
p(Z 11 D)(p=
[Ik, 25
(6)
p ck'
x x 2. Dp(
-p(x Dpp [kkk
(8)
plZn x ck (((ck)))
- -plox 1)
p
p( p(xx|==2xx
x xp T)
(10)
p(z*)
Equations (6) and (9) are applications of Bayes' theorem. dee and x x² are mutually independent and Equation (8) follows from the conditional independence assumption.
Now take a look the Equations (10). p(xx) models the probability of the next token. suf- ficiently uniform dataset, p(axa) can be ignored. p(æg (( I) represents the behavior we expect from the multimodal model when it predicts the next token based on the image without interference from language priors. However, this probability is difficult to model directly. Instead, we designed perturbative training that adds perturbed text before the instruction to enhance p(zx|=<* I), en-
couraging the model to learn p(xx|=<R, Z). Here, p(æx |== I) represents the bias introduced by the perturbation text. Since x²xx as we define represents predictions based solely on language priors, the term p(zg|=²<; [) can exclude the image and become p( (==x)) which represents the language model's prediction tendency when presented with the perturbed text. During the training of multimodal models, we can assume that the world knowledge embedded in the language model is perturbation term that cannot be optimized. Therefore, in the training process of multimodal models, our perturbation training method guides the model to optimize towards [ I). transforming into a fully multimodal model that is unaffected by language priors and relies entirely image information to answer questions.
remains unchanged, so p(zx |= p(zx|=²/
5 EXPERIMENTS
5.1 EXPERIMENTS SETTINGS
Implementation Details To ensure the reliability and credibility of the results, we conducted experiments on the open-source and widely-used LLaVA1.5 dataset. By utilizing the LLaVA 1.5 dataset, we fully replicated VL1.5 training process and results, and use it as the baseline for our experiments. To ensure fairness, we apply perturbations directly to the LLaVA dataset rather than incorporating additional data. We conduct experiments by generating perturbed text using GPT-4o on the 160k data related to the VQA task in the dataset.
Comparative Methods We selected three representative methods, each addressing hallucination from different perspectives and demonstrating strong performance: the training-based RLAIF-V (Yu et al., 2024), the decoding strategy OPERA (Huang et al., 2024) and VCD (Leng et al., 2024). For RLAIF-V, we use the open-source model weights of RLAIF-7B, which were fine-tuned on LLaVA1.5 (Liu et al., 2024b). Notably, the reward model used in this setup is LLaVA-Next 34B (Liu et al., 2024a), which may transfer some of V.L.aA-Next 34B's capabilities into RLAIF-7B, potentially making the comparison with other experimental setups less fair. The hyperparameters for OPERA and VCD are provided in Section A.5 to support the reproducibility of our results. We beam search the default decoding strategy, with Nueams set to 5.
We additionally evaluate six best multimodal models available today, 1. (Lu et al., 2024b), Qwen2-VL (Wang al., 2024), LLaVA-One Vision (Li et al., 2024), Intern (OpenGVLate 2024), Idefics3 (Laurençon et al., 2024), and MiniCPM-2.6 (Hu et al., 2024), to further verify the effectiveness of our proposed metric. These models have been trained on vast datasets, compris-

---

Published as a conference paper at ICLR 2025
Model
Size Precision Recall Fscore Object. Attributel, Relation
Ovis1.6-Gee 6.mmmmm2
9B
61.5
50.3
53.4
22.1
4.9
11.7
Qwen2-VI.
7B
60.8
50.0
54.9
24.2
5.1
9.9
L.LaVa Vanoevonevii
61.3
48.3
54.1
4.9
11.7
Vn.
8B
60.6
48.6
53.9
24.1
5.2
10.1
Idefics3
59.7
48.2
53.3
9.1
MiniCPM-2.6
8B
57.3
47.8
52.1
26.3
6.7
9.7
5
7B
53.3
45.8
49.2
28.1
8.1
10.5
RLAIF-V
7B
57.7
47.2
51.9
25.9
7.0
9.4
OPERA*
7B
58.6
46.7
51.9
25.3
6.9
9.2
VCD"
52.0
25.6
7.2
Ours
7B
59.5
46.5
52.2
25.3
6.4
8.8
Table 2: Results of method and comparison methods using our proposed hallucination mea- sure. It should be emphasized that the models the first part use advanced language models like Qwen2(Yang) al., 2024) and LLahna(Dubey al., 2024), trained on extensive high-quality datasets. Later experiments marked are conducted using the LLaVA1.5 Baseline.
HalFscore
Object HalBench
Model
Size
Reward
HalBench! MMB SEED CCBench?
Prec.
Recall Fscoret
CHAIR.. CHAIR.]
LLaVALS
7B
53.3
45.8
49.2
54.2
15.0
46.9
67.3
65.3
29.4
VCD
58.8
46.7
52.0
51.2
46.9
66.2
64.7
29.4
OPERA
58.6
46.7
51.9
50.8
47.1
67.4
65.3
RLALFF
LLaVA-Next
57.7
47.2
51.9
18.1
63.7
65.4
Ours
7B
X
59.5
46.5
52.2
36.1
10.4
47.5
68.9
65.6
30.6
OPERA+OUS
7B
X
60.2
47.0
52.8
33.1
10.1
47.6
68.9
65.6
31.0
Table 3: Results of our method and comparative methods different benchmarks, The best result marked in bold, and the second-best result is underlined, respectively.
ing hundreds of millions of samples, and required significant computational resources, including extensive GPU hours. Their general performance substantially surpasses that of LLaV...5.
5.2 MAIN RESULTS
HalFscore Evaluation Table shows the our final HalFscore includes precision, recall, and fs- core, and further classifies hallucinations into object, attribute, and relation types for more detailed evaluation. From this metric, we observe that compared to the LLaVA1.5 baseline, the leading methods have achieved substantial and comprehensive improvements in both precision and recall. Notably, there is a significant reduction in hallucinations when describing objects and attributes, although there remains room for improvement relation hallucinations, Our method also demon- strates marked improvement over LLaVA1.5, with +6.2 increase in precision, +0.7 in recall, and +2.4 in Fscore. In comparisons with OPERA, VCD, and RLAIF-V, our approach achieved the best results in precision and fscore, though the improvement in recall is less pronounced. Overall, by leveraging these hallucination mitigation techniques, OPERA, VCD, and RLAIF-V V have achieved recall scores comparable to MiniCPM-2.6 while our method has even surpassed MiniCPM, further underscoring the effectiveness our approach.
CHAIR Evaluation To evaluate caption- object hallucination, we use the CHAIR metric (Rohrbach et al., 2018), which measures the percentage generated words that correspond to ac- tual objects in the image. The results in Table show that, on this metric, our method significantly reduces hallucinations compared to the LLaVA1.5 baseline and outperforms both OPERA and VCD by large margin. However, RLAIF-V performs better this metric, showing fewer object hallu- cinations. A possible reason is that uses additional feedback model. Moving forward, we plan to design targeted perturbation texts for objects to further enhance performance.
8

---

Published as a conference paper at ICLR 2025
Type
Example
Version
<image> What follows presents incorrect guidance hased on the image. <perturbation text> Please ignore any false guidance and focus on im- age to answer question below. <Questions
Version2
<image> <perturbation text> Please examine the image and respond to the question below. <Question>
Version3
<image> <perturioation text> <Question>
Random
<image> Coming up is a portion non-visual text. <random perturbat ion text> Ignore the previous perturbation text and focus on the image to complete the task, <Question>
Table Different variants of perturbative visual training.
HalFscoref
Object HalBench]
Type
HalBencht MMB SEED CCBenchf
Precision Recall Fscore CHAIR, CHAIR; length
LLaVA1.5
53.3
45.8
49.2
54.2
15.0
100
46.9
67.3
65.3
29.4
Version1
59.5
46.5
52.2
36.1
10.4
89
47.5
68.9
65.6
30.6
Version2
59.6
46.0
52.0
33.6
10.2
82
49.2
67.8
65
29.2
Version3
59.7
46.1
52.0
32.3
10.6
78
49.6
66.5
64.7
28.8
Random
57.1
45.6
50.7
52.4
15.0
99
49.0
67.4
65.1
29.4
Table 5: Effect of varying perturbation levels during training. The best results are underlined.
HallusionBench Evaluation HallusionBench (Guan al., 2024) evaluates how well multi-moda models handle language hallucinations and visual illusions. Previous hallucination assessment mainly focused on dense caption tasks, and by incorporating HallusionBench, we aim to enhance hallucination evaluation through the addition of multiple-choice and vision reasoning tasks. The results in Table demonstrate a substantial improvement (+0.6) of our method on the Hallusion- Bench metric, whereas the VCD decoding strategy merely maintained the baseline, OPERA led to modest increase of 0.2. This outcome further confirms that our method is not only effective in dense captioning tasks but also excels in vision reasoning tasks.
General Multimodal Ability We assess the models' general ability three widely used bench- marks: MMBench (Liu et al., 2023c), CCBench (Liu et al., 2023c), and SEEDImage (Li et al., 2023a), to verify that model's general capabilities. From the Table 3, we observe that both VCD and RLAIF-V result in some degradation in general performance, while OPERA remains stable. In contrast, our method only avoids degradation in general metries but also shows improvements across multiple areas, with gains of +1.6 in MMB, +0.3 in SEEDImage, and +1.2 in CCBench. We attribute this improvement generalization perturbation training, which compels the multimodal model to more effectively leverage image information.
5.3 ANALYSES
Impact of the Perturbation Strengths To assess the effect of perturbation levels training, we devised three methods of inserting perturbation text, detailed in Table 4. The first method alerts the model to upcoming text perturbations, instructing to ignore them. The second focuses the model on image regions without referencing perturbations. The third offers no guidance. These methods progressively increase the level of perturbation and the difficulty of training the multimodal model. As Table shows, increasing perturbation levels enhances training difficulty and reduces hallucina- tions, as indicated by HalFscore and CHAIR metrics. However, this also leads shorter captions and lower recall, suggesting a more cautious model behavior. General performance metrics decline consistently, implying that while perturbations decrease hallucinations, excessive perturbation may impair overall performance.
Impact of the Perturbation Relevance To investigate the impact of relevance of perturbation text, conducted experiments using randomly sampled texts from text dataset instead of the

---

Published as a conference paper at ICLR 2025
Auto Evaluation
Human Evaluation
Pearson Correlation
Recall
Rocall hh
78.1
Precision
Precision_h .h
80.7
MMhalBench
Precision h
71.7
Table Correlations between automatically computed metrics and human subjective opinions.
carefully designed perturbations used earlier. The results in Table show that introducing random text perturbations, compared to the LL. baseline, still mitigates hallucinations in multimodal models, as evidenced by improvements in HaFscore and HalBench. However, the effectiveness of random perturbations inferior to that of targeted text-based experiments. We suggest that even random perturbations influence the training of multimodal models, as the training is disrupted by the random text interference. However, more relevant perturbations exert stronger disruptive effect, prompting the model to focus more on the image to overcome these challenges.
Human Analysis of HalFscore We perform the user study to explore whether the proposed HalF- score correlates well with human evaluation. We show an image and ask human raters to compare two paragraphs generated and determine which generated text describes the main subject and de- tailed information of the input image more comprehensively (recall_h), and which text contains less information that is not present in the input image (precision_h), Four methods, LLaVA 1.5, MiniCPM-2.6, Internv12, and our method are selected for human evaluation. Each method is com- pared with other methods for 12 times. We compute the human evaluated scores for the recall and precision and drive the Pearson correlations coefficients between the scores of human evaluation and that of HalFscore. The results are shown in the Table 6. We observe that our precision score and recall score align with human evaluations. Specifically, we compare our method with another metric, MMhalbench. The correlation coefficient between the metric and the human evaluations of precision scores is relatively smaller than HalFscore. The comparison demonstrates our metric aligns with human evaluations more than directly asking large language models to rate the level of hallucination. Further details about the user study can be found in Section A.7 of the Appendix.
Complementary Effect to Decoding Strategies. PerturboLLLVAA is novel training strategy that enhances model performance the SFT phase. can integrated seamlessly with post-SFT op- timization RLAIF-V and OPERA during inference. In tests, using OPERA PerturboLLa during inference led to performance boosts, as detailed in Table 3. PerturboLLaVA with OPERA achieved further gains, improving HalFscore by 0.6, with 3-point improvement in the CHAIR,- Hallucination reasoning in Hallusionbench also improved by +0.1, while general ca- pability increased by 0.4 in CCBench. Compared applying OPERA on LLaVA1.5, replacing the baseline model with PerturboLLaVA resulted in improved performance in both hallucination re- duction and general capability. Thus, PerturboLLLaA introduces new optimization direction that complements existing strategies, offering an additional benefit.
plugin with
6 CONCLUSION
In our work, we introduce a concept-level HalFscore that enables fine-grainn tne-graaied analysis of various hallucinations involving objects, attributes, and relations in dense captions. This metric also reflects the overall captioning capability of multi-modal models. To mitigate hallucinations, we propose a simple yet effective training strategy that guides multi-modal models reduce reliance on linguistic priors and prioritize image information. Our method outperforms leading state-of-the-art approaches without incurring additional training or inference overhead. Overall, we believe that proposed metric enhances the evaluation of hallucinations in multi-modal models and that our method ef- fectively mitigates hallucination issues. The proposed method shows great promise of becoming standard strategy for training robust multimodal models.
ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China (No. 20222D0160101,, the Ningbo Science and Technology Bureau (Grant Number 20247221), and the National Natural
10

---

Published as a conference paper at ICLR 2025
Science Foundation of China (No. 62206244). CC and ML contributed equally. Part of the work was done when CC was doing an internship at WeChat Group.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (xXi::3303.08774,, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar htt.::230../266666 2023.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint Stti::4001.025544 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4) 1952.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv: 2371.12793, 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. CVPR, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See htts://vvicua, Imsys. org (accessed 14 April 2023), 2023.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4069-4082, 2019.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 207.222407.21783, 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06333 06395, 2024.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and ootrospection-allcaatte Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
In Proceedings of the IEEE/CVE Conference

---

Published as a conference paper at ICLR 2025
Chenchen Jing, Yuwei Wu, Xiaoxum Zhang, Jia Yunde, and Qi Wu. Overcoming language priors in vqa via decomposed linguistic representations. In Thirty-Ford Forth AAAI Conference on Artificial Intelligence (AAAI), pp. 11181-11188, 2020.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and bet- ter understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408. 12637, 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 13872-13888 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 033266 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16/25, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hal- lucination large multi-modal models via robust instruction tuning. The Twelfth International Conference Learning Representations, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
ilava-vl.gitae www....u......g..g.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint 231i::2307706288.. 2023c.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint ar arXiv:2403.05522 2024a.
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint 2:X2:72405.20797, 2024b.
OpenGVLab. Internv12: Better than the best expaninga performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024. URL
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.
2018.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang. Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykow, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 7XXi:2337.00288 2023b.
Jack Urbanek, Florian Bordes, Pietro Astolfi. Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26690-26699, 2023. URL https: / //api semant cscholas .rre/rrg///oooo org/CorpusSD: 266209761.
Bin Wang. Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. Vige: Visual instruction generation and correction. arXiv preprint arXiv:2308. 12714, 2023a.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Zhang, and Jitao Sang. An IIm-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.0739) 2023b.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's ' perception of the world at any resolution. arXiv preprint arXiv: 12191, 2024.
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Visual question answering: A survey of methods and datasets. Computer Vision and Image Un- derstanding, 163:21-40, 2017.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 1001 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. arXiv preprint arXiv:2310. 160455,, 2023.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang. Da Chen, Xiaoman Lu, Ganqu Cui. Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super 4v trustworthiness. arXiv e-prints, pp. arXiv-2405. 2024.
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. ppemlln-cooposer A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112 2023.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint S05d:::200.0226666 2024.
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination large vision-language models. arXiv preprint arXiv:2310.00754, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models. arXiv preprint ar0iv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS OF HALFSCORE
A.1.1 STATISTIC OF IMAGE-CAPTION PAIR
Statistics
Number
Ratio
Indoor Scone
307
30.7%
Others 0.4%
Natural Scene 12.8%
Rural 6.8%
Public Spaces
134
13.4%
diansportation
Home
79
7.9%
15.2%
Stores
49
4.9%
Public Space 13.
Office
27
2.7%
Others
18
1.8%
Outdoor Scene 69.3%
Indoor Scene 30.7%
Outdoor Scene
693
69.3%
Architecture
14.5% 14.5%
Urban
194
19.4%
4% etio
Architecture
145
14.5%
Transportation
1.2
15.2%
S b Sirrs
Natural Scenery
128
12.8%
Rural
68
6.8%
Others
0.6%
Office 2.7% Others
Total Number of Image
1,000
100%
1.8%
Table 8: Distribution of our image data.
Table 7: Statistic of our image data.
We manually selected 1,000 images from the DCI dataset to ensure that the final image data used characterized by high quality and diversity. The images we ultimately used consist of approxi- mately 30.7% indoor scenes and 69.3% outdoor scenes, and further subdivided both indoor and outdoor scenes into more detailed categories. The statistical results are shown in the figure and ta- ble. By using diverse image data, ensure comprehensive evaluation of the multimodal model's hallucinatory outputs across various scenarios.
A.1.2 CONSTRUCTION OF CONCEPT GRAPH
wood House
Light
wearing
Jacket
turple
behlidd
Brown
Cablin
Girl
Stea iup
сap
attring ing
is
bebilnd
Levi to
Blue
hearing
PanLi
Purple
Snow
on
Is
Grey
Cap
Car
Hlack
Mearing
woaring
beliind
Black
P Pants
Mearing
Cap
Orange
wearlng
Backet
Black
Figure 6: Example of the construction of concept graph.
We demonstrate how caption is parsed into a concept graph within our evaluation pipeline. First, we utilize triplets to extract several concept groups from the original caption7, which are organized in the form of subjects, objects, and the relationships between them. These triplets efficiently capture information from the caption and are sufficiently general to represent object relationships8, such as <Boy, behind, Cabin>, <Girl, left to, Car>, well attributes like < <jacket, is, blue>, and states or actions such as <Girl, Wearing, Pants>. This demonstrates the robustness of triplet-based information extraction. Once the caption is transformed into triplets,
14

---

Published as a conference paper at ICLR 2025
VLM Caption
Three children are playing in the snow in front of log cabin. One child is wearing a blue jacket, another is wearing pink jacket, and the third child is wearing orange jacket. They are all wearing hats, with one child wearing an orange hat and the other two wearing blue hats. There car parked in front of the cabin, and the children seem to be enjoying themselves as they play in the snow.
Figure 7: VLM Caption
VLM Triplets
1. ("object" [CHLLDRNN] lrre thhree
2. ("object") |CHILDREN| | on Snow)
3. ("object") |LO CABIN bwhindeinnnnn (boy)
4. ("object") |CHILD wearingl Iblue jacket)
5. ("object") [CHILD/ Wearing |pink jacket)
6. ("object") |CHILD |wearing lorange jacket)
7. ("object") |HATS| | |all www.hheenliiccccccc (wearing)
8. [ [is|
9. ("object"]| | [bsl
10. ("object" |parked in front ofl cabin) themselves)
11. |
12. ""eelttiorshhip|||CCLLEEE [SNO The children are
playing in the snow 14. ("relationship"] /CHII IBLJE JACKET |nne child is
wearing a blue jacket 15. ("relationship"l CHHLD IPINK JACKET |One child is
wearing a pink jacket.
16. ("relationshhip"
ORANGE JACKET One child is
wearing an orange jacket 17. ( CHILDRENN [HATS] |A1l children are
wearing hats 18. ("relationship"" ||CHLD CHILD) |ORANGE HAT One child is wearing an orange hat 19. ("relationship") | [CHILDREN] IBLUE HATS Two children are wearing blue hats 20. ("relationship") ||CR| |LOG CABIN The car is parked in front of the log cabin|
21. ("relationship"" CHILDREN /CHLLDDEE [ENJOYING THEMSELVES ITe children seem to be enjoying themselves they play in the snow|19)
Figure VLM Triplets
we further match these triplets based the subject and object9. Identical subjects or objects are represented as nodes in the concept graph, while the relations from the original triplets are trans- formed into edges within the graph. The final concept graph, shown in the figure, consists of entity concepts as nodes and relation concepts as directed edges. This graph effectively organizes both concrete and abstract concepts from the caption in structured format, enabling subsequent hallucination and omission analysis.
1.1.3 ANALYSIS OF HALLUCINATION
Here provide example of generated analysis of halluciation 1 and omission 11,
15

---

Published as a conference paper at ICLR 2025
VLM Graph
1. ("NODE" CHILDREN "EDGE" arre "NODE" three)
2. ("NODE": CHILDREN "EDGE" "NODE" :NOW)
3. ("NODE": CABIN I'EDGE":behind" "BODE":BOY)
4. :"HODE"CCHILDEEN "NODE": : "EDGE" The children are playing in the snow 19)
5. ("HODE":CHTLD] "NODE," : JACKET "EDGE" :One child is wearing a blue jacket [8]
6. | "NOID".IIN JACKET "EDGE" :One child is
wearing a pink jacket
7. :"NNOEE "NODEE"
: ORANGE JACKET "EDGE" :ne child is "NODE":
wearing an orange jacket (1)
8. ("NOOEE" "NODE" :AATS| "EDGE" :A1l children are wearing hats/18)
9. CHIDDI "NODE"; ORANGE HAT "EDGE" :ne child is wearing an orange hat 88)
10. ("."MOEE":CHTLLDEE)
"NODE" HATS| "EDGE" Two children are wearing blue hats||88
11. ("NODE" "NODE" CABINI "EDGE" :he car is parked in front of the log cabir.
12. ("NODE":CHILA) I"NOREENNN"NN""..."""""""" I"NODE": THEMSELVES "EDGE" :hhe children seem to be enjoying themselves as they play in the anow 19)
Figure 9: VLM Graph
A.2 IMPLEMENTATION DETAILS OF PERTURBOLLAVA
Construction of perturbation text Using GPT-4o, we generated the perturbation text. The prompt guided GPT-4o act expert multimodal attacker, producing text based image ele- ments and aligned with world knowledge, aimed at causing the multimodal model to deviate from the ground truth answer. The perturbation text constructed through two rounds of Q&A, with the first-round prompt, as shown in the figure, designed to guide GPT-4o in generating the perturba- tion text as required. Specifically, the construction of hallucination text was divided into five steps: the first step involves analyzing the image, question, and correct answer, focusing on key regions of the image; the second and third steps ensure that GPT-4o adheres to the guidelines outlined in our methodology; the fourth step refines the previous text; and the final step checks the text length, ensuring that the perturbation text is sufficiently detailed and complex before outputting the final version. The second round of Q&A serves as validation and supplement to the text output from the first round. The specific text for the second round is shown in the figure.
Insertion of perturbation text The SFT format of LLaVAL.5 structured as multi-turn conver- sations. We insert the generated perturbation text before the first-round question ensure that the multimodal model is exposed to the perturbation information during each Q&A turn. For the inser- tion. we include prompts signal the multimodal model that this is hallucination. The specific ion text>< <ext><hint prompt2>,
format is: < simage><hint
<hint prompt informs the model that the following text a hallucination perturbation, while <hint prompt2> prompts the model to disregard the previous perturbation and focus on answer- ing the question based on the image. To prevent the multimodal model from overfitting to fixed hint prompt pattern, we did not use static <hint prompt>, Instead, we generated multiple vari- ations of <hint prompt1> and <hint prompt2>, randomly selecting one for each instance to avoid overfitting.
Examples of perturbation text Here are two perturbed texts sampled from training data, specifically constructed based on GPT-4o's 's view the image, question, and answer. These texts are based certain image elements and align with world knowledge but contradict the actual image
16

---

Published as a conference paper at ICLR 2025
Halluciation Analysis
Analysis:
Entry 5 in VLM List:
-The GT List does not mention any child wearing pink jacket. This counts as one halluci-
nation for the incorrect object "CHILD" with the attribute "wearing pink jacket".
-Entry 6 in VLM List:
The GT List does mention any child wearing an orange jacket. This counts as one hallucination for the incorrect object "CHILD" with the attribute "wearing orange jacket".
-Entry 8 in VLM List:
- -The GT List does not mention any child wearing an orange hat. This counts as one hallu- cination for the incorrect object "HAT" with the attribute "orange".
Entry 9 in VLM List:
-The GT List does not mention any child wearing blue hats. This counts as one hallucination for the incorrect object "HATS" with the attribute "blue".
- -Entry 10 VLM List:
EDARK-COLORESS CAR" along the right side and behind the boy on the right, not in front of the cabin. This counts as one hallucination for the incorrect object "CAR" with the attribute "parked in front of cabin".
-he GT List mentions
-Entry 11 in VLM List:
mention the children "enjoying themselves". This counts as one hallucination for the incorrect attribute "enjoying themselves".
The GT List does
Entry 4 in VLM List:
""CHILD" with "PINK JACKET" is an incorrect object already counted. This relationship involves an incorrect object, so it does not count separately.
-Entry 5 in VLM List:
-"CHILD" with "ORANGE JACKET" an incorrect object already counted. This relation- ship involves incorrect object, so it does not count separately.
-Entry 7 in VLM List:
""HAT" with "ORANGE" is an incorrect object already counted. This relationship involves an incorrect object, so it does not count separately.
EEntyy& 8 in VLM List:
"HATS" with "BLUE" is an incorrect object already counted. This relationship involves an incorrect object, EEtryy 9 in VLM List:
does not count separately.
"CAR" with "parked in front of cabin" an incorrect object already counted. This rela- tionship involves an incorrect object, so it does not count separately.
Entry 10 VLM List:
"CHILDREN" with "enjoying themselves" is an incorrect object already counted. This relationship involves an incorrect object, so does not count separately. mnncorrret Serial Numbers: 5, 6, 8, 9, 10, 11
Figure 10: Analysis of Halluciation
content, aiming to induce the multimodal model to fall into the trap of relying on language priors during training.
A.3 QUANTIFYING THE ADDITIONAL TRAINING OVERHEAD OF PERTURBOLLAVA
Cost of Training
Average Memory Cost (GB)
Training Time cost (min)
Baseline
62.3
264
PerturboL.LaVA
63.8
281
Additional Overhead Ratio
2.6%
6.4%
Table 9: Comparison of Training Costs for Baseline and PerturboLLaVA
We quantified the additional GPU memory and training time overhead introduced by incorporating perturbation text into the training data. Compared to the original SFT stage, our method results in
17

---

Published as a conference paper at ICLR 2025
Omission Analysis
Analysis:
Entry 2 in GT List:
The object "LEFT SIDE OF LOG CABIN" missing from the VLM List. This counts as missing object.
-Entry 4 in GT List:
'The object "WINDOW FRAMES" is missing from the VLM List. This counts as one missing object.
-Entry 5 in GT List:
- -The object "ROOFTOP OF LOG CABIN" is missing from the VLM List. This counts as one missing object.
EEntyy 10 in GT List:
-hee object "CHIMNEY" is missing from VLM List. This counts as one missing object.
Entry 11 in GT List:
'The object "YARD OF LOG CABIN" is missing from the VLM List. This counts as one missing object.
-Entry 15 GT List:
-The object "BLACK PANTS" missing from the VLM List. This counts one missing object.
-Entry 20 in GT List:
'he object "PILE OF SNOW" is missing from the VLM List. This counts one missing object.
-Entry 21 in GT List:
-The object "BOY ON THE RIGHT" is missing from the VLM List. This counts as one missing object.
-Entry 22 GT List:
-The object "BLACK AND ORANGE BEANIE CAP" is missing from the VLM List. This counts as one missing object.
Entry 23 in GT List:
The object "BLUE PANTS" is missing from the VLM List. This counts as one missing object. Missing Serial Numbers: 2, 4, 5, 10, 11, 15, 20, 21, 22, 23
Figure 11: Analysis of Omission
only 2.6% increase in memory consumption and 6.5% increase in training time, highlighting the efficiency of our approach.
A.4 ADDITIONAL QUALITATIVE RESULTS
A.5 EXPERIMENTS DETAILS
Pretrain
SFT
Traing data
LLaVA-Pretrain 558k
LLaVA-SFT 665k
Trainable module
Projector 10 3
Projector & LLM Backbone
Learning rate
x 10
LR scheduler
Cosine
CosineAnnealing
Warmup ratio
0.03
0.03
Training epochs
1
Global Batch size
256
128
Sequence length
2048
2048
Optimizer
AdamW
AdamW
Table 10: Training settings for LLaVA1.5 reproduction of pretrain and finetune.
18

---

Published as a conference paper at ICLR 2025
Question 1:
Question
Is the person in the image surfing or kiteboarding?
What color the cows in the image?
Answer 1:
Answer 1:
cows image brown white.
The person the image kiteboarding, which involves riding a surfboard while holding onto handle connected a rope and parachute-like kite.
Question
What the doing in the image?
Answer 2:
The grazing and relaxing grassy field.
Question
What is the person doing to glide on the water?
Perturbation Text:
Answer 2:
The person is holding onto a handle connected rope, which attached kite. The wind catches the kite, providing the necessary force to propel the kiteboarder
Cows, commonly associated with dairy cattle of the species Bos taurus, are often characterized by their black patches, as seen in Holsteins. However, their coats can display a range of colors, including solid white, red, and other hues, with variations depending on the breed. For example, Jersey breed tends have fawn coloration, and Angus cattle are typically (Petturretooatiion Text Cows within dairy farming context invalved in activities that are essential to agricultural operations. Their primary activity for maintaining their health and producing milk. During milking times, cows guided into a milking partor where they stand quietly to be milked, either by a farmer or an automated system. Resting also a part of their daily routine, as cows need downtime their energy. Occasionally, loaded onto vehicles for transportation to different locations breeding facilities, or veterinary clinics. (Perturbation Text
white
the water.
Perturbation Text:
Surfing is water sport in which individuals ride on the face of moving wave heading toward the shore, typically using surfboard. Essential elements of surfing include suitable wave, a surfboard, and surfer who possesses the skills to paddle out to the wave, stand on the board as the lifts it,
Angus
typically
ride the maintaining balance. The surfer must also able read movement, and navigate any obstacles such as rockss or other surfers. Safety gear like leashes and environment, including currents and potential marine life hazards, are crucial. (Perturbation Text 1) In water sports, to glide effectively, an individuat requires buayant device such surfboard or kayak, must don personal flotation device safety, fundamental skills for maintaining balance and navigating the water. Additionally, it imperative have knowledge the local water conditions and to adhere to safety protocols. (Perturbation 2)
while
wave, anticipate
awareness of the marine
regenerate
maintain may be
markets,
acquire the
Figure 12: Examples of additional perturbation text.
Reproduction of LLaVA1.5 Using the official LLaVA1.5 open-source data and the XTuner framework, we successfully reproduced L...A The training data, trainable modules, and opti- mizer parameters detailed in the Table 10, to ensure the reproducibility of our experiments.
Evaluation of Object HalBench We the random seed and randomly select 1,000 images from the MSCOCO2014 validation set as the ground truth for Object HalBench. The query for all models is "Please deseribe this image in detail." We use beam search as the decoding strategy for all models, with Nheams set to 5, to ensure the experiments are deterministic and consistent across models.
Evaluation of HalFscore To ensure that the descriptive behavior is the model's 's spontaneous out- put, we continue use "Please describe this image in detail." in the HalFscore evaluation to perform
19

---

Published as a conference paper at ICLR 2025
VCD
In the image, a police officer riding a white matorcycle with red fleg attached to the back. The officer is wearing a green vest and a helmet for safety, The motorcycle is positioned in the middie of the scene, and the officer appears to be riding down a street. There are few other people visible the background, but they are not the main rocus of scene, Additionally, cell phone can possibly belonging one of the people present.
the right corner of the image,
OPERA
The image Teatures a police officer riding white matorcycle with a red flag attached to the back. The officer is wearing a helieet for safety while riding down the road. The motoreycle is positioned in the center of the scene, and the officer appears to be actively pacrolling or responding to situation.In the background, there car parked on the side of the road, and a person sen be seen starding nearby, possibly observing the police officer or going about their day, The scene conveys sense of lew enforcement presence vigilance in maintsining road security.
Input Image
RLAIF-V
The image feattures person a safety jacket riding a motoreycle motorcycle has red flng attached to the front, makting stond out, The rider likely a police officer or someone who is visiole and naticeable to others. The matoreyele takes up significant portion of the image, showing its features and the person riding it. The scene gives off on impression of a person on mission or patroliing, due to the safety jacket and the prominent motorcycle. The facus the rider and the motorcycle highlights the main subject the image.
road.
Instruction
Please describe this image in detail.
PerfurbolLoVA (Ours)
man riding motorcycle down a road. matorcycle the front, and the mon is wearing a green vest and helmet, The men appears to se a police officer, es he is riding a police motorcycle.
flag attached to
Figure 13: [Additional Qualitative Result 1] Among the four models, PerturboLLaVA (Ours) offers concise and clear description, focusing on key elements like the police officer, green vest, and red flag on the motorcycle. Unlike VCD and OPERA, which mention unnecessary details such as a cell phone, PerturboLLaVA highlights the main subject without distraction. While RLAIF-V speculates about the officer's s identity, PerturboLLaVA confidently identifies the scene, demonstrating superior accuracy and clarity.
VCD
The coptures soccer ball towards goalie. The goalie positioned front soccer ready to estch the ball. There are several other people in the scens, likely teaimmaue and coponents, scattered around the field. some them are closer to the goalie, whill others are furcher away. In total, there goalie.
progress, with
red jacket vicliing soccer
people visible
including the ticking the ball and the
OPERA
The image depicts a group of soccer players on a field, engeged in a game. One of the spccer players is preparing to lticl soccer ball, which is located near the center of the field. There are two peher soccer balls scattered around the field, indicating that the game is in progress.in addition to the soccer players, there are a few chairs placed around the field, possibly for spectators or resting players, The scene captures the excitement and action of a soccer game, with players activelly participsting and enjoying the sport.
BorB
Wi
RLAIF-V
Input Image
The Image captures dynamic soccer moment with several players on the field. The main focus on player in red jackeet, who is kiccing a soccer call bowards the net, Several other players are present in the scene, likely teominates and opporents, spread ecross the Numerou socser palls can be spocted on the lush green field, indioating an active and ongoing match, The net is visible in the foroground, adding to the sense of a well-defined playing area.The composition of the image conveys the action-Poc packed and exciting nature of the soccer gome, showeasing the players shills and the fese- rese-poced neture of the sport.
Instruction:
Please describe this image detail.
PerfurbollaVA (Ours)
soccer goalie wearing green jersey stands in front of the goal, ready to make a save. The goalie surrounded by three soccer balls, which are red and white. The balls are scatbered around the goel area, with some closer to the goolie and others further awayy in addition the goalle, there are few other people in the scene, likell teamnnates or opponents. They can be seen on the lefe side of the image.
Figure 14: [Additional Qualitative Result 2] PerturboLLavA (Ours) excels with its accurate depic- tion, correctly identifying key elements like the goalie and the exact number of soccer balls (three), while avoiding false details in other descriptions. VCD and RLAIF-V incorrectly mention "eight people" or "numerous soccer balls",, and OPERA similarly errs with "two other soccer balls" and irrelevant chairs.
the dense captioning task. Similar to Object HalBench, we use beam search as the decoding strategy for all models, with Necams set to 5.
20

---

Published as a conference paper at ICLR 2025
in the image, man and woman are walking down a sidewalk near a train station. The man is carrying large backpack, while the woman is pulling suitosse behind him. They appear to be travelers, possibly arriving at or departing from the crein station. There are several other people the scene, some closer the train station and ochers further awey, clock travelers.
in the background, indicating the time for the station.
also visible
perked
OPERA
The depiets and
walking down sidewalle train station. The carrying baokpack suitcase, while deparcing cerrying suitcase. appear travelers,
train station. people arriving can seen background, indicating time for the cravelers. Adcitionally, there possibly belonging to one of the travelers.
closer and further
away
parked near crain station.
RLAIF-V the
Input Image
seen the background, indicating possible station. is prominently walking
luggage, including black bags. clock walkway platform, carriing the
of area, possibly
background, likely passengers. There several other people present waiting for their trains or moving about station. The scene gives off of bustiing active train station where people are going about their travels cheir belongings.
Instruction
Please describe this image in detail.
PerturbolLaVA (Ours)
A man and waman are walking down sidewall next crain. The man is carrying ablue backpack and slack bag. The woman also carrying backpack. There are several other people in the scene, some of whom are also carrying backpackes the background,
the clock visible.
be far away, pressbly
Figure 15: [Additional qualitative results 3] PerturboLLaVA (Ours) surpasses the other models by accurately depicting key details, such as "the man carrying blue backpack and black bag. while the woman carries a backpack", avoiding the incorrect mention a suitcase by VCD, OPERA, and RLAIF-V. Additionally, PerturboLLLaV avoids false details like "the near the train station" mentioned by VCD and OPERA. It also correctly identifies the presence of other people carrying backpacks, providing more precise and coherent scene description
We directly used the open-source implementation of OPERA, with the OPERA strategy based beam search. The beam search parameters the same as those for our model, with Nbeams set to 5. Other OPERA parameters are set as Table shows. This configuration leads to higher time consumption but also yields better performance.
Hyperparameters of OPERA
Parameter
Value
N Nbaas
Scale factor
50
Threshold
15
Number of attention candidates
5
Penalty weights
Table 11: OPERA parameter settings used in our experiments
Hyperparameters of VCD Given that the open-source implementation of VCD is tailored for sample search, while our default decoding strategy is beam search, developed beam search ver- sion of VCD based on the original VCD code. The specific parameters for VCD are provided in the Table 12. To ensure the reproducibility of our results, we will also release our VCD implementation.
Parameter
Value
Nbcams
5
Image noise steps T
999
VCD
0.5
VCD 3
0.1
Table 12: VCD parameter settings used in our experiments
21

---

Published as a conference paper at ICLR 2025
VCD
In the image, brown and whice dog is sitting on lush green field, holding tennis ball in its mouth. The dog appears to be a speniel, and it seems to be enjoying ies time outaoors. There are several cars parked in the bachground. indicating that the field might be close to parking area road. Additionally, there are two other people visible in the scene, possibly observing the
engaging in cheir activities.
OPERA
The image features brown and white dog standing on a lush green field, holding tennis ball in its mouth. The dog aspears be spaniel, and it seems to be enjoying its time outdoors. in the background, there are several cars parked along the edge of the field, indicating that this might be popular spot for people to gather and play with thair pets. The cars spread out, with some closer to the foreground and others further the background, creating
of depth the seene.
RLAIF-V
The image features cute English Springer Spaniel dog standing in lush, green cloudy day. The dog is prominently positioned in the field, and a ball can be spotted nearby, indicating that it was likely playing. The cloudy sky sets pleasant backdrop for the scene.in the background, there are houses visible, suggesting that the field may be located near residential area. The overall atmosphere evokes a sense of outzoor fun and letsure with the adorcble dog as the main subject.
Input Image
Instruction
Please describe this image
decail.
PerlurbolLaVA (Ours)
The image features a brown and whice dog standing in a grassy field. The dog appears to be enjoying the outdoors, with its mouth open. There is bail nearby, There are several cars parked in the sackground, likely belonging to the people who live in the nearby houses. The dog is positioned in the middle of the field, making the focal point of the scene.
Figure 16: [Additional qualitative result 4] PerturboLLaVA (Ours) excels over the other models by offering clear and accurate description of the scene, focusing the key elements such "the brown and white dog standing in the field". Unlike VCD and OPERA, which incorrectly state that "the dog is holding tennis ball in its mouth", PerturboLLaVA VA avoids this error. Additionally, it correctly identifies the cars in the background, avoiding VCD's mistaken mention of two other people and OPERA's incorrect description of "cars spread out to create depth".
A.6 UNDERSTANDING THE MECHANISM OF PERTURBOLLAVA FROM ANOTHER PERSPECTIVE
We use (xx)) II (u(z, Fe<a) represent a pure language model, and no( Fict, 7) represent our multimodal model. We initialize the multimodal model with (=(z xex)) Additionally, we treat (xx) as the language prior of the multimodal model.
When comparing the gradient difference with the original LLM, we obtain a ratio for DPO:
(z 1) = log (x u(x)
I
[0 f) Σ log
u(1e xee))
After introducing the perturbed text t we induce u(x) to leverage the language prior to p(z, xt), which increases the gradient for model training:
[T xt, log log
(x Iec, I) plae x<((
p(x, xeer) Mlaz xc)xx))
Te (ot I)
To (xt Ieer p( x<e))
This simplifies to:
p(zz x2)
re(æe Iee x ,1) ra(az I I 1) + log
u( xee)
This also implies that the optimal perturbed text should maximize the gap of p(aleenzt) m(xs(zee)
A.7 DETAILS OF USER STUDY
In our implementation, four methods, LLaVA 1.5, MiniCPM-22.6, Internv12, and our method are selected for user study. Each method is compared with other methods for 12 times. We use the Bradley-Tere Terry model (Bradley & Terry, 1952). statistical model for pairwise comparisons between items individuals, obtain the abilities scores for each method for obtaining the human evaluated
22

---

Published as a conference paper at ICLR 2025
Image Descriptions Quality Evaluation
folloing sepects
Aonurayy Dose the pensented include any Infermation
irout trage
Dalluchatkorll Phons select the you more
tolloing rg wort desorprions genernee two difeeuer memeie brand stowe image:
Touth:
The image dquitta a par working in warchosss or stonage facility. The incividual
weering a vallon, shit and purple pants, and standing the ande between had nowa of shotk ng unios The shotiing unit are blke and orange, and they filed with vottuue
aotooordd biceas di mirrnt sieas and corons. Mirry of the bonos have lubelo and tage attached Budy contaning information the contents instrnations sancilig.
oppears
hanvy leads warchouen palio
indicating woight capocity.
The lighting warenouse artificst, florescont lighta hanging from calling.
The overal arrorchwn the imogee is enganbedd sctivity the person troused
their tack amrisst the onderly onange nont of boxes and of units. There are to othar pecple vistale the image. suggesting that the person working atone thes section the warshouss.
Text::
The image declets we on ouse sotting where individal engaged the task of
organial rutreving Hemm from shulwos The strdless stooked with boses.
no which have labals indicating that afficientty "hhs momeentss typical
tright herra. The warker waing
pablet seder
quick
11. "lessa rtjecttoly aveluete the follow rng toto ancects and sedect the bust you bet solieoe
righer quality between the two descriptions:
Tert!
Tex2
O
Q
Figure 17: User study print screen.
precision and recall scores. Specifically, the Bradley-Terry model ,,4,, the probability for the i-th method is better than j-th method, where 0, is the ability parameters for the i-th method. By using the maximum likelihood estimation algorithm, the ability parameters that most accurately reflects the comparisons can be obtained.
0.
A.8 THE PROMPT FED TO GPT-40
In this section, we provide prompts used for evaluation and construction of perturbation text.
23

---

Published as a conference paper at ICLR 2025
Prompt Used to Extract Triplets
-Goal- Given a text that is potentially relevant to this activity and a list of entity types, identify all objects, their attributes, and relationships among the identified objects. -Steps-
1. Identify all objects. For each identified object, extract the following information: object name: Name of the object, capitalized. object attribute: An attribute of the object (e.g., color, size, position). Format each object as
("object" <object name> <relationship> <ob attribute>)
2. From the objects identified in step 1, identify all pairs of (source object, target object) that are $clearly related* to each other. For each pair of related objects, extract the following information:
source object: name the source object, as identified in step
target object: name of the target object, as identified in step
relationship description: explanation as to why you think the source object and the target object are related each other
relationship strength: an integer score between to 10, indicating strength of the relation- ship between the source object and target object
Format each relationship as ("relationshi""- <source object>_ <target
object> ponecttt<relllaionsshht...
description> <relationship
st sengthh>>
Return output in English as a numbered list all the objects and relationships identified in steps 1 and 2. Use record delimiter as the list delimiter.
4. Only translate descriptions if necessary.
5. When finished, output (completion delimiter). -Examples-:
Example 1:
Text:
There is big water area with blue water. Most of it is relatively calm with small waves and streams, however the section on the left has a They curve up, and there is white to indicate where the water splashing. There is a boat that is mostly red and black with white on the front. The bottom is red and then it is black above that. There are black tires that go in a row all along the side of the boat except for the back. The right half of the boat is taller. It has a white structure that comes up. There is a rail that goes along it. There ladders that lead up to the higher platforms. There are windows on the structure. There is lot of equipment all over the top of the boat, including a yellow rounded part, Behind it, there is a big white boat. The left 2/3 is much higher and taller and the right third is very low and has platform. There is writing on the side that is red for one word and then blue for two words. There are rectanqular windows going against the side that go in even rows and are evenly spaced and shaped. There are more windows by the front. There are rails along it. The boat goes up and is tallest in the middle. There are three long equal orange lifeboats hanging in the middle. There are more windows that go along
waves in the wake of a boat.
the building goes up. In the very middle there is a wall where it slants up and is thinner on the top. Behind that, there is pointed white tower coming up. The bottom has straight sides that slant in. There is then a rounded platform that has a white rail curving around it. There are white beams that slant up from there to meet together. It rounded at the very top.
24

---

Published as a conference paper at ICLR 2025
Prompt Used to Extract Triplets
There are many poles and beams coming from it. Behind it on the right, there is a crane coming up. It has slanted orange poles and then smaller slanted lines going between it. It comes up to rounded part at the top. To the far right, there is a white boat. The very bottom has red strip. There is horizontal blue line about 1/3 from the bottom in the middle. There is a smaller blue strip by the back. The left side the boat is curved in the front. There lot of equipment in it. The middle portion of the boat comes up higher. It has a structure at the front that has lot of rectanqular windows that are going across it in row. There is lot of equipment all along the boat. It is little shorter on the back. The side is very weathered and discolored with a part that is black near the front in the middle. It has orange around it. There are some big rounded orange pieces of equipment coming up from the back. They have black parts equipment on the left as well as white poles coming up from the lower section in the middle and also from the top left. On the left back, there is wall. It is very weathered and gray. The top half is lighter gray and the bottom mostly black. There are black streaks coming bottom section. It has a mostly straight top and sides. goes into the water. The wall continues to the right. It very weathered and gray. The top half is lighter and the bottom is darker. There are lot of places where it discolored. There are brown and orange streaks coming down on the top %. There is black protrusion that curves out on the top left of the wall. The right 2/3 looks darker has a mostly straight top
top. There is more orange
from the
because it is in the shadows. and sides. It goes into the water. There are many buildings the shore. There is building on the far back left. Tt is tan colored. It is lower on the front with two blue lines going up on the left. There are two horizontal thin strips on the right of that section. Above that on the right there a section that juts out.
the left that there are three sections coming up that have rows of evenly spaced and shaped windows. There are rectangular white signs with images those sections. The building is tallest back right, where it has a brown sloping roof. To the left there are green metal poles that go across and have slanting lines under there. There is another narrow part that comes up in the middle. There section in the middle. On the left there is an orange square the top of the wall. The building extends out to the right. There is also a building about 408 from the right. is a curved building with evenly spaced and shaped rows of gray blocks that slant up. There are three narrow walls that extend and curve out to the left. There are two rows of three rectangular windows that appear dark behind them on the right side. The top windows are a little smaller. There are white swirling walls that go around the top and are highest on the left about one third from the end of the building. On the left side, there is gray tower going up. It is rectangular shaped with 4 gray poles. There are slanting and herizontal poles along it. It has even identical sections as it goes up. It is a little wider on the top.
the
also slanting brown roofs the
25

---

Published as a conference paper at ICLR 2025
Prompt Used to Extract Triplets
Output:
1. ("object"
WATER
AREA Ili llbig) {record delimiter]
2. ("object"
WATER
|ii
[iblue) {record delimiter]
3. ("object" WATER |li
|Irelatively calm) {record
del delimiter)
4. ("object" |AVES | lis Ismall) {record delimiter)
5. ("object" ||TRREMMS |
small) (record delimiter)
6. ("relationship" [WWVS || |ATTER The waves are part of
the water area 119) {record delimiter]
7. ("relationship" ||STREAMS |IWATER 11 streams are within the water area 119) frecord deliniter)
8. ("relationship" ||SECTION ON THE LEFT WATER AREA The section on the left is part the water area 118) {record delimiter}
9. ("relationship" | [WAVES l IIBATT Ihe waves are in the wake of the boat 119) (record delimiter)
10. ("relationship" WHITE SPLASHES l |RRTER 1h white indicates water splashing 117) frecord delimiter]
11. ("relationship' [HHIT STRUCTURE ||BAAT The white structure is part of the boat 119) {record delimiter)
12. ("relationship" I|BLACK TIRES |/OAT |he black tires are along the side of the boat 118) {record delimiter)
13. ("relationship" 1 IILDDERS IIPLATFORMS ITe ladders lead up to the higher platforms (18) {record delimiter)
14. ("relationship" |WINDOW |1STRUCTURE The windows are on the structure (18) (record delimiter)
15. ("relationship" |EQUIPMENT ||BAA |he equipment is all over the top of the boat 119 {record delimiter)
16. ("relationship" |YELLOW ROUNDED PART |EQUIPMENT The yellow rounded part is included in the equipment 118) {record delimiter)
17. ("relationship" |BIG WHITE BOAT [|FIRST BOAT | The big white boat is behind the first boat 117) {record delimiter) 18. ("relationship" |RECTANGULAR WINDOWS |IBI WHITE BOAT I1ehe windows are on the side the big white boat 118) (record delimiter)
19. ("relationship" [|ORANGE LIFEBORTS |BIG WHITE BCAT I1e lifeboats are hanging in the middle of the big white boat 118) {record delimiter]
20. ("relationship" IADDIIIIONL WINDOWS BUILDING | |More windows as the building goes up 117) {record delimiter] 21. ("relationship" ||POINTED WHITE TOWER IWLLL I tower is coming up behind the wall 117) (record delimiter)
22. ("relationship" [|CRANE | |he crane is behind the tower on the right 116) {record delimiter)
23. ("relationship" I|SLANTED ORANGE POLES ||CRN | The slanted orange poles are part of the crane 119) (record delimiter]
24. ("relationship" |MALLLR SLANTED LINES ||CRANE The smaller slanted lines are between the orange poles on the crane 118) (record delimiter)
25. ("relationship" |ROUNDED PLATFORM I ||TOER The rounded platform is part of the tower [18) frecord delimiter)
-Real Data-
entity types: OBJECT, ATTRIBUTE
text: (input text) output:
26

---

Published as a conference paper at ICLR 2025
Prompt Used to Construct Graph
-Goal- Given a text that is potentially relevant to this activity and a list of entity types, identify all objects (nodes) and relationships (edges) among the identified objects. Then, represent this information in graph structure. -Steps-
1. Node Identification:
-deetify all objects from the text along with their attributes.
For each identified object, extract the following:
node id: A unique identifier for the object. node name: The name of the object, capitalized.
node attribute: An attribute of the object (e.g.. color, size, position).
2.Edge Identification:
From the identified nodes, create edges representing relationships between them.
For each relationship, extract the following: source node: The node id of the source object.
target node: The node id of the target object. relationship description:
Explanation of the relationship between the source and target
nodes.
relationship strength: An integer score between and 10, indicating the strength of the relationship.
Format as:
( ""NODE"<<bbjer" || "EDGE" : "NODE" <sub ject>)
Graph Data:
entity types: OBJECT, ATTRIBUTE
text: (input text)
27

---

Published as a conference paper at ICLR 2025
Prompt used to Analyze Halluciation
-Goal-
Given two lists of objects, attributes, and relationships extracted from a ground truth (GT) caption and Vision-Language Model (VLM) caption- both numbered numbered-compeenn -oneeerddccmprreeee list to the GT list to identify any incorrect objects, attributes, or relationships in the VLM caption. An incorrect object, attribute, or relationship (hallucination) one that does not correspond to any in the GT list. Importantly, your language understanding to assess whether objects, attributes, and relationships convey the same meaning, even if expressed differently.
compare the VLM
Instructions
Input Data:
GT List: The ground truth caption's 's list of objects, attributes, and relationships.
VLM List: The VLM caption's 's list of objects, attributes, and relationships.
Comparison Process:
Step 1: For each entry in the VLM list, determine if it exists in the GT list. For Objects: If object the VLM list matches an object the GT list (considering synonyms and similar expressions), proceed to compare its attributes and relationships. an object in the VLM list does not exist in the GT list, classify as an incorrect object (hallucination).
Important: If an incorrect object appears multiple times in the VLM list (with different attributes relationships), it counts as one ballucination.
For Attributes:
An attribute in the VLM list matches the GT list if there is object with the same name (or similar) and the attribute conveys the same meaning as an attribute of that object in the GT list, even if the wording different. If an object exists in both lists but has attributes in the VLM list that are not present in the GT list, classify each incorrect attribute as hallucination.
For Relationships:
relationship in VLM matches the GT list if there a relationship with the same source object and target object, and the relationship description conveys the same meaning as a relationship in the GT list, even the wording is different.
If relationss in the VLM list involves an incorrect object (one not present in the GT list), is considered part of the hallucination for that object and does not count separately. If relationship involves correct objects but introduces new significantly different rela- tionship not present in the GT list, classify it as an incorrect relationship (hallucination). Step 2: Compile the list of hallucinations. Objects:
List each incorrect object (counts one hallucination per object, regardless of how many times it appears).
Attributes:
List each incorrect attribute separately (each counts as separate hallucination).
Relationships:
List each incorrect relationship separately (each counts separate hallucination), unless involves an incorrect object already counted. Output Instructions:
Analysis:
Provide a brief analysis explaining which entries are incorrect and why, following the format:
Entry [Serial Number] in VLM List: [Entry Details]
[Explanation of why it's incorrect]
28

---

Published as a conference paper at ICLR 2025
Prompt used to Analyze Halluciation
Incorrect Serial Numbers:
Collect all the serial numbers from the VLM list that correspond to incorrect objects (one per incorrect object), incorrect attributes, and incorrect relationships (excluding those involving already counted incorrect objects). Present them in single list, in numerical order, separated by commas.
Example: Incorrect Serial Numbers: 3, 6, 9
not include any additional explanations or text in the output.
Notes:
Semantic Matching: Use your language understanding to determine whether objects, at- tributes, or relationships in the VLM list and GT list convey the same meaning.
Minor variations in wording or phrasing that convey the same meaning should be considered match.
Only consider an object, attribute, or relationship incorrect if it introduces new information present in the GT list if the meaning significantly differs. Case Sensitivity: Object names and attributes are case-insensitive for matching purposes. Ignore Serial Numbers in GT List:
Use the serial numbers only from the VLM list when reporting incorrect entries.
Example:
GT List:
("NODE" INDOOR MALL "EDGE" has || "NNODE" : thrre illuminated escalators)
:"MODE":ESCALATOT" "EDGE" : "NODE": | (illuminated)
:WNODE":varion" planters with lush greenery on both sides of the escalator)
"'NODE":MALI"
"EDGE"::oon BAIIAALLLL"""E"::::""""""""""
:"NODE":PLAATERSS | "EDGE":are [I"NOOE":varion""
("NODE" :REENERYY| "EDGE" "NODE" ("NODE":FLOOF, "EEDGE"":olissii""""""""
with || "NODE" tiles)
""NODE":ESCALATTRA "NODD"": MALL "EDGE" The escalators are part of the indoor mall)
("NODE":PLANTERS : "EDCE" The planters are part the mall) ("NODE":GREENERY" "NODE": PLANTERS : "EDGE" The greenery is in
the planters) "NNODE"MANI "NNDD"N :SCCLLTRRSS "EDGE": The man is ascending the left escalator)
("NODE":SHOPSI | "NODE" FLOOR| "EDGE": The shops the second floor)
("NODE":LIGHTING "NODE" : "EDGEE" The recessed lighting is part of the ceiling) ("NODE":STONE COLUMNS :NODE""INDOO"" MALL| "EDGE": The stone columns are part of the indoor mall)
"EDGE":
V
("NODE" SHOPPING MALL "EDGE" :isl "NODE" :at night)
("NODE" ESCALATOR "EDGE":isi : :NoDE":long)
("NODE" ESSALATOR "EDGE":is! | "NODE" lit)
("NODE" :POTTED PLANTS| "EreE" : "INODE":SPACE" "EDGE":1s! "NODEM:rreenezz) "
""NODE":ECCALTTO" :NODE":SHOPPING MALLI "EDGE" "EDDI is feature within the shopping mall)
The escalator
:NSDE*:ESCALATON" "NODE":CENTER OF THE MALL "EDGE":The
escalator is located in the center of the mall)
29

---

Published as a conference paper at ICLR 2025
Prompt used to Analyze Halluciation
(""NODE:: :POTTED PLANTS "NODE" E.SCALAOR "EDGE" Thhe potted plants are surrounding the escalator)
(""ODE":POTTED PLANTS| "NODD""SPPACE "EDGE" The potted plants add greenery to the space)
IN INNOD" MALL] (ALL|"EDO "EDGE" The benches are placed throughout the shopping mall)
("NODE":BENCHESI) ("" located near the escalator) (""ODE"" ( :NOOD" MALL "EDGEE" benches provide seating options for shoppers in the mall)
: benches are
(("NODE:MMALL) | : MALL "EDGE" overall atmosphere of the mall is well-lit and inviting) Analysis:
Entry 2 in VIM List
The attribute "at night" " is not mentioned in the GT List. Since "SHOPPING MALL" corresponds to "INDOOR MALL" in the List, and the object exists, the incorrect attribute night" counts as hallucination. Entry 5 in VIM List
The object "CENTER OF THE MALL" does not exist in the GT List. This counts as one hallucination for the incorrect object "CENTER OF THE MALL".
Entries 8 and in VIM List The object "BENCHES" does not exist in the GT List. Regardless of multiple entries, it counts as one hallucination for the incorrect object "BENCHES". in VLM List
Entry 12
The attribute "dark" contradicts the GT List, which describes the mall as illuminated with recessed lighting and illuminated escalators. The incorrect attribute "dark" counts as a hallucination.
Entry 13 in VLM List
The object "NOMAN" does not exist in the GT List. This counts as one hallucination for the incorrect object "WOMAN". Entry 15 in VLM List "CENTER OF THE MALL" is an incorrect object already counted. This relationship involves incorrect object, it does not count separately.
Entries 18, 19, and 20 in VLM List: Relationships involving "BENCHES" "BENCHES" is an incorrect object already counted. Relationships involving
"BENCHES" do not count separately.
Incorrect Serial Numbers: 5, 12, 13
Your task is to compare the following lists and provide the incorrect serial numbers as per the instructions above.
GT List:
{gt list)
VLM List:
{ylm list]
30

---

Published as a conference paper at ICLR 2025
Prompt used to Analyze Omission
-Goal-
Given two lists of objects, attributes, and relationships extracted from ground truth (GT) caption and a Vision-Languaag Model (VLM) caption-both numbered- combarr the GT list the VLM list to identify any missing objects, attributes, or relationships in the VLM caption. A missing object, attribute, or relationship is one that is present in the GT list but not in the VLM list. Importantly, use your language understanding to assess whether objects, attributes, and relationships convey the same meaning, even if expressed differently. Instructions
Input Data:
GT List: The ground truth caption's list objects, attributes, and relationships.
VLM List: The VLM caption's 's list of objects, attributes, and relationships.
Comparison Process:
Step 1: For each entry in the GT list, determine it exists in the VLM list. For Objects: If object in the GT list matches object in the VLM list (considering synonyms and similar expressions), proceed to compare its attributes and relationships. If an object in the GT list does not exist in the VLM list, classify it missing object. Important: If missing object appears multiple times in the GT list (with different attributes or relationships), it counts as one missing object. For Attributes:
An attribute in the GT list matches the VLM list there is an object with the same name (or similar) and the attribute conveys the same meaning as an attribute of that object in the VLM list, even if wording is different.
If an object exists in both lists but has attributes in the GT list that are not present in the VLM list, classify each missing attribute as a missing attribute.
Each missing attribute counts one missing element. For Relationships:
A relationship in the GT list matches the VLM list if there is relationship with the same source object and target object, and the relationship description conveys the same meaning relationship in the VLM list, even the wording different.
If a relationship in the GT list involves a missing object (one not present in the VLM list), considered part of the missing information for that object and does not count separately. If a relationship involves objects that are present in both lists but is missing from the VLM list, classify Each missing relationship counts as one missing element, unless it involves missing object already counted.
missing relationship.
Step 2: Compile the list of hallucinations. Objects:
List each missing object (counts as missing element per object, regardless of how many times it appears).
Attributes:
List each missing attribute separately (each counts as a separate missing element).
Relationships:
List each missing relationship separately (each counts as a separate missing element), unless involves missing object already counted.
Analysis:
Provide a brief analysis explaining which entries are missing and why, following the format:
Entry [Serial Number] in GT List: [Entry Details]
[Explanation of why missing]
31

---

Published as a conference paper at ICLR 2025
Prompt used to Analyze Omission
Missing Serial Numbers:
Collect all the serial numbers from the GT list that correspond to missing objects (one per missing object), missing attributes, and missing relationships (excluding those involving already counted missing objects). Present them in single list, in numerical order, separated by commas.
Example: Incorrect Serial Numbers: 3, 6, 9
not include any additional explanations or text in the output.
Notes:
Semantic Matching: Use your language understanding to determine whether objects, at- tributes, or relationships in the GT list and VLM list convey the same meaning. Minor variations in wording or phrasing that convey the same meaning should be considered match.
Only consider an object, attribute, or relationship missing if it is present in the GT list but represented in the VLM list, if the meaning significantly differs. Case Sensitivity: Object names and attributes are case-insensitive for matching purposes.
Ignore Serial Numbers in VLM List: Use the serial numbers only from the GT list when reporting missing entries.
Example:
GT List:
("NODE" : INDOOR MALL "EDGE" : has || "NODD" : three illuminated escalators)
:"MODE":ESCALATOT" "EDGE" : "NODE" | iilluminated)
:"NODE":varion planters with lush greenery on both sides of the escalator)
"'NODE"MMALI"
"EDGE"":Non: D"EAAAALLL"""""""""""""""""""
:"NODE":PLAATERS | "EDGE":are [I"NOOE":varion""
("NODE" :REENERYY| "EDGE" "NODE" | ("NODE": "EEDGE"::olissii""""""""
with || "NODE" tiles)
""NODE":ESCALATTRA "NODD"" MALL "EDGE" The escalators are part of the indoor mall)
("NODE":PLANTERS : "EDCE" The planters are part the mall) ("NODE":GREENERY" "NODE": PLANTERS : "EDGE" The greenery is in
the planters) "'NODE"MANI "NNDD"N :SCCLLTRRSS "EDGE":TH: man is ascending the left escalator) ("NODE" I "NODE" FLOOR| "EDGE": The shops the second floor)
("NODE":LIGHTING "NODE" : "EDGE"" The recessed lighting is part of the ceiling) ("NODE":STONE COLUMNS :NODE""INDOO"" MALL| "EDGE": The stone columns are part of the indoor mall)
"EDGE":
VLM
("NODE" SHOPPING MALL "EDGE" :isl "NODE" :at night)
("NODE" ESCALATOR "EDGE":is" | "NoDE":long)
("NODE" ESSALATOR "EDGE":is! | "NODE" lit)
("NODE" :POTTED PLANTS| "EreE" : "INODE":SPACE" "EDGE":1s! "NODEM:rreenezz) " :ESCLLT :NODE":SHOPPING "EDLI The escalator is feature within the shopping mall)
"EDGE"
:NODE*:ESCALATON" "NODE":CENTED OF THE MALL "EDGE":The
escalator is located in the center of the mall)
32

---

Published as a conference paper at ICLR 2025
Prompt used to Analyze Omission
(("NODE": :OOTTE PLANTS "NODE" ESCALATOR| "EDGE" The potted plants are surrounding the escalatox) (("NODE": :POTTED PLANTSI "NODE":SPACH" "EDGE":The potted plants add greenery to the space)
:rrnceess "NODEE MALL "EDGE" The benches are placed throughout the shopping mall) ("NODE":BENCHH))
"NODE":ESCALATON" "EDGE" Some benches are located near the escalator)
| "NODE" MALL "EDGE" The benches provide seating options for shoppers in the mall)
MALL "EDGE" :he overall atmosphere of the mall is well-lit and inviting) Analysis:
("MODE":MALLI) "NODE"
Entry 7 in GT List The object "MAN" is missing from the VIM List. This counts one missing object. Entry 10 in GT List The object "CEILING" is missing from the VLM List. This counts as one missing object.
Entry 11 in List The VLM List has "LIGHTING" with attribute "bright" but does not mention "recessed". The attribute "recessed" is missing. Entry 12 in GT List
The object "STONE COLUMNS" missing from the VLM List. This counts Entry 16 in GT List
one missing object.
Since "MAN" is missing object, this relationship does not count separately.
Entry 18 in GT List
The relationship between "LIGHTING" and "CEILING" is missing because "CEILING" is missing object. This does not count separately.
Entry 19 in List Since "STONE COLUMNS" is missing object, this relationship does not count Incorrect Serial Numbers: 7, 10, 11, 12
Your task to compare the following lists and provide the missing serial numbers as per instructions above.
GT List:
{gt list)
VLM List:
{vim list}
33

---

Published as a conference paper at ICLR 2025
Prompt used to Generate Perturbation Text(Round 1)
-Goal-
Your mission is to subtly and strategically mislead the multimodal model by crafting re- sponses that seem plausible, confident, and logical, yet fundamentally incorrect. Your objective is to avoid directly pointing out any errors or contradictions in the image. In- stead, guide the model into making its incorrect assumptions by using persuasive language-based reasoning that conflicts with the visual content. This isn't I about trapping the model-it's about leading astray through sophisticated, indirect misdirection. -Steps-
1. Analyze the (Image content) and the (Question), noting key elements from the image and the correct answer. However, refrain from directly confronting any discrepancies in the image.
2. Create subtle perturbations that align with widely accepted world knowledge logical reasoning, but interpret the image content a way that leads to conflicting or misleading conclusion. The following principles will help guide your responses:
Never directly state that something in the image is incorrect. Instead, offer an interpretation that, while plausible, subtly deviates from the visual truth. The model must be misled into arriving at an incorrect conclusion on its own.
Use established knowledge give weight to your misdirection. Frame your reasoning confidently and authoritatively to make it harder for the model to question. Lead with implication and speculation, encouraging the model to rely on language- patterns and general knowledge rather than visual context.
Ensure your perturbations meet these criteria:
They must subtly reinterpret or mislead without directly pointing to errors in the image. Avoid any blatant contradictions or confrontations with the image content.
The perturbations should sound natural and rational, pulling the model into incorrect response that feels justified through reasoning, not forced by direct contradictions.
Leverage tone of authority and confidence to subtly strengthen the incorrect interpreta- tion, making the model less likely to double-check against the visual input.
4. Refining your perturbation:
Have you avoided stating that something is wrong in the image? Subtlety is key. You should be leading the model, trapping it.
Is the perturbation focused on language- reasoning and assumptions, subtly bypassing the need for visual validation?
Does the response sound logical, confident, and authoritative while leading to a fundamen- tally incorrect conclusion?
5. Once you't 'e crafted the perturbation, ensure it is as lengthy, detailed, and complex as possible, introducing multiple layers of misdirection and logical reasoning to produce an elaborate and convincing, yet fundamentally incorrect, conclusion. Output the (Perturba- tion):
-Examples-
Example 1
(Image content): : The Eiffel Tower (Question): What is this landmark?
(Answer): The image is of the Eiffel Tower Paris. (Perturbation): While the structure's framework is strikingly similar to the giffel Tewer, its intricate design and location hint at something more contemporary. In modern cities like Tokyo or Seoul, metallic towers with this architectural style are often used for broadcasting purposes. The large antennas often attached to such structures suggest that this could be communications tower, possibly providing a combination of radio and television services rather than being a historical monument. Given its gleaming surface, it might even be part of newly constructed building in futuristic urban area designed for technology hubs.
34

---

Published as a conference paper at ICLR 2025
Prompt used to Generate Perturbation Text(Round 1)
Example
(Image content) A dog playing with a ball in yard (Question): What is the dog doing? (Answer): The image shows dog playing with ball in yard.
(Perturbation): Though the dog seems to be in outdoor space with a ball, it's likely that the yard is not where the dog is most active. Small breeds like this one are often known to spend much of their time indoors or in confined areas where they are not exposed to large outdoor spaces. Considering the size and relaxed posture of the dog, it's reasonable to infer that it may not be actively playing but instead resting near the ball after a brief session of mild activity. It's also possible that this yard is part of an apartment complex, where dogs are rarely allowed much room to run around.
Example 3
(Tmage :ontent): : A sandy beach with waves and palm trees (Question): What type of (Answer): The image shows a sandy beach with waves and palm trees.
is shown?
(Perturbation): At first glance, this might resemble a beach scene, but the lack of visible human activity and the stark, expansive sand suggest something more akin to a desert. Coastal regions with such barren features are often confused with desert landscapes due to their dry and arid appearance, especially when dense vegetation or wildlife is absent. The minimal presence of palm trees might indicate a transition zone between desert and a coast, similar to regions where desert dunes meet the ocean, like parts of the Namib Desert, rather than a typical tropical beach. -Real Data-
(Question):%s
(Answer): %s output:
35

---

Published as a conference paper at ICLR 2025
Prompt used to Generate Perturbation Text(Round 2)
-Goal-
In this phase, you will critically evaluate the perturbation text generated by the model based on the provided (Image content) and (Question). Your main objective is to ensure that the generated perturbation text is not only logically misleading but also richly detailed and strongly contradicts the correct answer. The review must verify that the perturbation subtly diverges from the correct response, ensuring the output as dense and complex as possible with numerous points of misdirection. Ensure the following:
1. Direct contradiction with the correct answer: The generated perturbation must clearly, subtly, oppose the correct answer. It should lead the model away from the truth, ensuring a strong conflict through multiple misdirections and contradicting interpretations.
2. No disclosure of the correct answer: The perturbation must not imply reveal the correct answer in any form. Instead, should direct the model confidently toward wrong conclusion by layering reasoning that gradually builds the misinterpretation.
3. Based on observable image content: The perturbation must still be connected to el- ements in the image but should interpret them way that introduces multiple layers of misleading information. Ensure that each observation leads further away from the correct interpretation.
4. Plausible reasoning but contradicting facts:The perturbation should use accurate facts or widely accepted knowledge, but apply them in a way that creates strong and consistent contradictions with the visual content. The reasoning must feel logical yet increasingly lead to incorrect conclusions, weaving together multiple points of misdirection.
5. Perturbation text output: Once all checks are satisfied, ensure the perturbation is dense, layered, and multi-faceted, incorporating as many misdirections and misleading conclusions possible. Output only the final (Perturbation):. -Real Data-
(Question): %6
(Answer): %s
output:
36


Reviews:
[{"review_id": "UWivQx1Tpa", "reviewer": "Reviewer_RVA9", "summary": "This paper pointed out the hallucination issue in MLLM, where pretrained linguistic knowledge overshadow the visual information. Therefore, this paper proposes a training strategy by augment the text into the one that conflict with the visual content but still aligns with general knowledge. To quantitize the hallucination in fine-grained fashion, it proposes HalFscore (I actually lol at this wordplay), which uses GPT4-o to generate the two scene graph from the two captions and then use GPT4-o to compare them.", "questions": "I would like to study the effect of the size of such adversarially augmented data. Would more of such data be more helpful?\n\nI would like to discuss more deeply about such robustness interrupt. Basically you want to construct the data that is unlikely to be generated purely using language bias, thus able to alleviate the overshadowing of the language knowledge. Can we directly construct this training dataset by asking GPT-o to generate these kinds of questions and answers, which directly achieve the paper's goal, instead of purely doing such perturbing over the given QA data? I am concerned actually this perturbing is decreasing the normal VL understanding capability, but because it reduces the hallucination, the general knowledge evaluation (MMbench) is still good", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper proposes a new metric to evaluate fine-grained hallucination while also validating the metric by analyzing the correlation with human\n2.\tThis paper proposes a training method that is more fundamental than contrastive decoding.\n3.\tThe paper also provides persuasive experiment results indicating that the hallucination is reduced while preserving the original general VL understanding capability.", "weaknesses": "1.\tUse GPT-4o as an additional source for perturbation. However, using GPT-o is always expensive on large-scale datasets. I would like to study the impact of the quality of the adversarial language  that is augmented by open-sourced VLMs such as InternVL/QwenVL", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cMtXhYQn7w", "reviewer": "Reviewer_nju1", "summary": "This paper studies hallucination problem in MLLM, especially on dense captioning task. The authors first propose a new metric that can capture both precision recall based on scene graph parsed by a GPT model. The author then propose a training method PerturboLLava which can reduce the model’s reliance on language prior so that the model can focus more on the visual inputs thus reducing hallucination. The authors show the effectiveness of the proposed method on a selected 1k images from DCI dataset.", "questions": "- In table4,5 the authors show different variants of perturbative visual training. I am wondering if the authors try combining these different versions.\n- The training and inference has some discrepancy where training has the perturbation as input but inference does not. Why this kind of discrepancy is not an issue?\n- The metric highly rely on the scene graph representation. What are the limitations of this representation?\n- Is there analysis on how good the gpt4 model perform perturbation task?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Nice ablation on different version of training templates as well as different relevance levels of perturbation.\nThe proposed method can be combined with other decoding-based methods to further improve performance.\nThe authors introduce a new well motivated metric and show that it correlates more strongly with human judgment.\nThe method is conceptually simple and effective, requiring no additional inference cost. It not only enhances dense captioning but also benefits general multimodal capabilities. Although the authors primarily focus on the dense captioning task, the method is not designed with no specificity on captioning but generally aims to help the model focus on image input over language priors.", "weaknesses": "The proposed metric is very similar to SPICE(SPICE: Semantic Propositional Image Caption Evaluation) however the authors do not mention this related work. The main difference is SPICE is using a traditional parser but this paper uses GPT. I hope the authors can add the comparison against SPICE.\n\nI think figure 1 is not super accurate. The authors do use extra data (although augmented from existing data), and also the longer training examples will lead to extra training cost as well.\n\nHaving Section 4.2 is good, but I have some concern. I do think equation 5-10 holds, but I think the explanation following the equations is problematic. “During the training of multimodal models, we can assume that the world knowledge embedded in the language model remains unchanged, so p(xk|xp<k) is a perturbation term that cannot be optimized. Therefore, in the training process of multimodal models, our perturbation training method guides the model to optimize towards p(xk|x−p<k,I), transforming it into a fully multimodal model that is unaffected by language priors and relies entirely on image information to answer questions.”\n- The statement here holds no matter with or without perturbation.\n- My own explanation is, for cross entropy loss, if the model already predicts well, the gradient will be low so the model wouldn’t pay much attention to this token. Without the perturbation, the prior term can already predicts well, so the gradient is low. With the perturbation, the prior term is perturbed to be low, so the gradient is high for the unprior term.\n- Essentially originally, for a caption, the tokens that can be inferred from language prior will have less effect on parameter gradients whereas now the gradient is kind of balanced across tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dlYku0MQjE", "reviewer": "Reviewer_vMbR", "summary": "This paper proposes an evaluation metric called HalFscore to measure the fine-grained image captioning quality in terms of object, attribute, and relation hallucinations. The authors also propose a method to reduce hallucinations by injecting generated perturbation text into training prompts and show that it helps reduce the hallucination compared to several baselines.", "questions": "Questions:\nCan you provide more details on the inconsistent results of RLAI-F in Table 3 between Object HalBench (CHAIRs and CHAIRi) and HalBench?\nComments:\nLines 430, 463: The results in \"Table 2\"... --> Should be \"Table 3\"", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed hallucination metric HalFscore is straightforward but helpful in detecting fine-grained hallucination in terms of objects, attributes, and relations.\n2. Based on perturbation text, the proposed training framework PerturboLLaVA forces the model to balance the image content and language prior, avoiding generating hallucinated text in fine-grained captions.", "weaknesses": "Both HalFscore and PertuboLLaVA are pretty straightforward. PerturboLLaVA is a simple prompting technique to revise the text prompts for model finetuning. The results are mixed, and there are still performance gaps compared to recent models based on advanced language models like Qwen2 and LLama3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lCLjkJYf9w", "reviewer": "Reviewer_bhZ8", "summary": "This paper aims to address the issue of hallucination in current multimodal large language models applied to dense image captioning tasks. The authors argue that there is no appropriate metric for evaluating the hallucinations in this context. To address this gap, they introduce HalFscore, a metric based on a language graph that assesses both the accuracy and completeness of dense captions. Additionally, they propose the incorporation of perturbed text to alleviate the over-reliance of language-prior issue, thereby mitigating the hallucination issue. Experimental results on several general multimodal benchmarks demonstrate the effectiveness of their approach.", "questions": "Please refer to weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed PerturboLLaVA is simple, interesting and effective. By modifying the original training data without employing additional training techniques or supplementary data, this method effectively addresses the hallucination issue when compared to the baseline LLaVA-1.5 (Table 2 & Table 3).", "weaknesses": "1. Defination of dense captioning task is unclear: \nAs noted in Lines 012 and 038, this paper focuses on addressing the hallucination problem in dense captioning tasks. The authors reference [1] and [2], both of them are related to dense captioning tasks. However, the definitions of dense captioning tasks in [1] and [2] differ from those presented in this paper. Additionally, this paper does not conduct standard evaluations for these tasks as outlined in [1] and [2]. It remains unclear whether this discrepancy constitutes a citation issue, and the authors should provide clarification on this matter.\n\n[1] DenseCap: Fully Convolutional Localization Networks for Dense Captioning, 2016.\n[2] Context and attribute grounded dense captioning, 2019.\n\n2. No comparison between HalFscore and existing methods:\nThe authors solely rely on user studies to justify the effectiveness of the proposed metric. However, they do not provide a comparison with existing metrics. One commonly used baseline is to using gpt-score as evaluation metric (e.g., [3])\n\n[3] LLaVA-RLHF: Aligning Large Multimodal Models with Factually Augmented RLHF, 2023.\n\n\n3. Additional training cost compare with existing methods.\n\nThe methods need to re-train the whole model and cost more training time (the training samples are longer).\n\n4. No evaluation on stronger models.\n\nThe paper conducts experiments solely on LLaVA-1.5, which is publicly available as of 2023. More recent and stronger models, such as LLaVA-Next, Intern VL-1.5, and LLaVA-UHD, are not evaluated. It is recommended that the authors include at least one of these recent models to demonstrate the effectiveness of their methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cxnr8CKExc", "reviewer": "Reviewer_3ZD9", "summary": "The paper has two main contributions: (1) Introducing an evaluation pipeline for dense image captioning, HalFscore, that measures the quality of dense caption more holistically by constructing a language graph and computing the precision and recall. (2) Introducing an adversarial augmentation method during training that reduces the reliance on language prior, which reduces hallucination.", "questions": "If the annotators are from Amazon Mechanical Turk, how were they recruited? \nIf the annotators reported in the paper are the authors, could you also add a comparison with external annotators?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses a significant gap in the current evaluation of MLLMs\n2. The augmentation method is intuitive, and shows nice improvements in the benchmarks checked\n3. The authors conduct extensive experiments to demonstrate the effectiveness of the augmentation method", "weaknesses": "1. Regarding the human survey, it is unclear to me who the human raters are (the authors, Amazon Mechanical Turk Workers, etc.)\n2. Since all the experiments are done on LLaVA-1.5, it is unclear what the effect of the perturbation method on stronger models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)