Automated Multi-level Preference for MLLMs
Authors: Mengxi Zhang
Wenhao Wu
Yu Lu
YuXin Song
KANG RONG
Huanjin Yao
Jianbo Zhao
Fanglong Liu
Haocheng Feng
Jingdong Wang
Yifan Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Large Language Models
Hallucinations
Reinforcement Learning from Human Feedback
Primary area: machine_vision

Abstract: Current multimodal Large Language Models (MLLMs) suffer from ''hallucination'', occasionally generating responses that are not grounded in the input images. To tackle this challenge, one promising path is to utilize reinforcement learning from human feedback (RLHF), which steers MLLMs towards learning superior responses while avoiding inferior ones. We rethink the common practice of using binary preferences (*i.e.*, superior, inferior), and find that adopting multi-level preferences (*e.g.*, superior, medium, inferior) is better for two benefits: 1) It narrows the gap between adjacent levels, thereby encouraging MLLMs to discern subtle differences. 2) It further integrates cross-level comparisons (beyond adjacent-level comparisons), thus providing a broader range of comparisons with hallucination examples. To verify our viewpoint, we present the Automated Multi-level Preference (**AMP**) framework for MLLMs. To facilitate this framework, we first develop an automated dataset generation pipeline that provides high-quality multi-level preference datasets without any human annotators. Furthermore, we design the Multi-level Direct Preference Optimization (MDPO) algorithm to robustly conduct complex multi-level preference learning. Additionally, we propose a new hallucination benchmark, MRHal-Bench. Extensive experiments across public hallucination and general benchmarks, as well as our MRHal-Bench, demonstrate the effectiveness of our proposed method. Code is available at https://github.com/takomc/amp.

Automated Multi-level Preference for MLLMs
Mengxi Zhang 1.2 Wenhao Wu², Yu Lu", Yuxin Song', Kang Rong', Huanjin Yao!. Jianbo Zhao Fanglong Liu', Haocheng Feng', Jingdong Wang', Yifan Sun
Baidu Inc. "Tiajjin University 'The University of Sydney UUniversity of Technology Sydney 'Tsinghua University "Chinese Academy of Science
Abstract
Current multimodal Large Language Models (MLLMs) suffer from "hallucina- tion", occasionally generating responses that To tackle this challenge, one promising path is to utilize reinforcement learning from human feedback (RLHF), which steers MLLMs towards learning superior responses while avoiding inferior ones, We rethink the common practice of using binary preferences (i.e., superior, inferior), and find that adopting multi-level pref- erences (e.g., superior, medium, inferior) is better for two benefits: 1) It narrows the gap between adjacent levels, thereby encouraging MLLMS discern subtle differences. 2) It further integrates cross-level comparisons (beyond adjacent- comparisons), thus providing broader range of comparisons with hallucination ex- amples. To verify our viewpoint, we present the Automated Multi-level Preference (A.MP) framework for MILMMs facilitate this framework, we first develop an automated dataset generation pipeline that provides high-quality multi-level prefer- ence datasets without any human annotators. Furthermore, we design the Multi- level Direct Preference Optimization (MDPO) algorithm robustly conduct com- plex multi-level preference learning. Additionally, we propose a new hallucination benchmark, MRHal-BBR -Bnnch. Extensive experiments across public hallucination and general benchmarks, as well as our MRHal- -eech,, demonstrate the effectiveness our proposed method. Code is available at httttt:://gtt /tttps://gihub. com/ takkoc////////////////////tt//tt amp
grounded the input images.
Introduction
Multimodal Large Language Models (MLLMs) [ 2 6 have achieved remarkable advance- ment languaeee understanding tasks, e.g., vision question answering 77. image captioning [8],
and human machine conversation. Despite MLLMs achieving significant breakthroughs, they still suffer from hallucinations IO] referring to responses that are not accurately anchored to context provided by images. This problem shrinks the performance of MI.LMs and draws consid- erable research attention. To mitigate the hallucinations, some existing methods [T]
human-machine
12
13] [4]
adopt Reinforcement Learning from Human Feedback (RLHF) methods, which collect human/Al preferences and integrate them into the MLLMs optimization process via reinforcement learning.
Existing RLHF methods have demonstrated that comparing superior and inferior responses within binary-level preference framework can improve the performance of optimized MLLMs. However, Is single comparison between superior and inferior responses sufficient for preference learning in MLLMs? Upon consideration, we find that multi-level preference framework offers greater benefits for preference learning, primarily due to two main intuitive advantages. Firstly, reducing the gap between adjacent levels helps mitigate the challenge distinguishing miero hallucinations in responses. As depicted in Fig. [ in the baseline method (i.e., binary preference), significant differences exist between the superior response A and the inferior response C. By introducing an additional medium response and shifting the focus to multiple comparisons between adjacent levels
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Three people... blue and black bux One
jocket. digitl shows T28 oestination, waich
B
B Tame peocis getting on blue bus Dise of
them weataa yallow digital display stons estimated antival time..
Two peopue are getting on tikue bos, Oze ot then iz yellow suit... . displas, which cleadly rocorts the destiaution, and antval time...
Final MILM
Final MLLM
Final MLLM
Describe this image in detail.
Multi level Preferance Dataset
(a) Baseline of RLHF
(b) Reduce the gap (c) Cross level Comparison
Figure 1: Left: Depicted are the input image, text prompt, and corresponding multi-level preference dataset. Contents highlighted in red signify hallucinations. Responses range from A C, representing varying degrees of quality from superior to inferior. Right: Illustrating the strategy for leveraging inferior responses. (a) displays the conventional RLHF baseline, which adpots the binary-leve preference. (b) To mitigate the gap between adjacent levels, we first split a single comparison into multiple comparisons by inserting extra medium responses. (c) Furthermore, we introduce the cross-level comparison to augment the dataset with more hallucination examples.
("A>B", "B>C"), as highlighted by the red arrows in Fig. [b), we mitigate this issue. Interestingly, under certain conditions, the MLLM's performance with ">CC" comparison is even worse than that with "B>C", indicating that reducing the gap between adjacent levels is sometimes more effective than enhancing the quality of superior responses. Secondly, cross-level comparisons can further enhance performance. In the comparisons between adjacent levels, the only comparison utilizing the inferior response "A>B", which may lead the model to focus more on suppressing hallucinations in response B. To address this, we introduce the cross-level comparison "A>C" (the green arrow in Fig. [c)) to provide more negative examples, thereby helping the model suppress more possible hallucinations. By integrating these strategies, we evolve the conventional binary-level preference learning into a more sophisticated multi-level preference learning framework.
However, exploring multi-level preferences for MLLMs poses significant challenges: 1) Labeling expensiv
multi-level preference datasets is expensive and laborious. While some methods [T] 12 utilize
human annotators to obtain preference labels, this approach is effective for binary datasets but falls short for multi-level preference datasets. Specifically, establishing a K-level preference dataset requires human annotators to make K(K - 1)/2 comparisons. For example, with K 5. this results in 10 comparisons, significantly more than is required for binary datasets. On the other hand, datasets annotated by humans or AI often contain significant noise and bias. To investigate this, we collected preferences from both humans and GPT-4V [15] on a subset of ShareGPT4V [16), using three MLLMs to generate varied responses. Setting K to 3, we compared pairs of responses (A&B, B&C, A&C) through three comparisons. However, we observed a frequent contradictory pattern (A>B, B>C, C>A), with rates of approximately 14% and 11% in human and GPT-4V annotations, respectively, resulting in a low-quality multi-level preference dataset. 2) The optimal multi-level preference learning objective remains unclear. While multi-level preference is more beneficial for optimizing MLLMs, it introduces greater complexity than binary preference. Therefore, it requires an effective algorithm to fully utilize the knowledge embedded within multi-level preference datasets.
To overcome the challenges outlined above, we introduce innovative strategies both the data and method levels: 1) At the data level, we propose two novel methods for generating initial multi-level preference datasets without human or Al annotators. Furthermore, we implement an auto-check mechanism to further refine these datasets by evaluating the scores and accuracy of the generated responses. 2) At the method level, we introduce the Multi-level Direct Preference Optimization (MDPO) algorithm. derivative of the traditional Direct Preference Optimization (DPO) algorithm [1). The MDPO algorithm extends the capabilities of the DPO algorithm to facilitate multi-level preference optimization. Additionally, we incorporate tailored penalty term into the MDPO learning objective to ensure robust multi-level preference learning. 3) Finally, we introduce evaluation benchmark, MRHal-Bence which is the first designed specifically to evaluate hallucinations multi-round dialogues. In summary, our contributions are as follows:
Contrary to prior RLHF studies that focused solely on enhancing the quality of superior responses, our findings indicate that inferior responses can also play a crucial role reducing hallucinations under the multi-leved preference learning framework.

---

To support effective multi-level preference learning, we develop two novel methods and an auto-check mechanism, enabling the creation of high-quality multi-level preference datasets without the need for human or Al annotators. Furthermore, we design the Multi-level Direct Preference Optimization (MDPO) algorithm with a specifically crafted learning objective, allowing MILMss to robustly learn from the multi-level preference dataset.
Our extensive experiments across various hallucination benchmarks confirm the effectiveness our framework. Additionally, we have introduced HuRHal-Benche the first benchmark specifically designed to evaluate hallucinations in multi-round dialogues.
2 Related Work
2.1 Multimodal Large Language Models
Recently, the multimodal learning community has witnessed the great success of MLLMs 1
5 18), which employ a cross-modal alignment module to connect the visual encoder 19 20
and the language model [22 23 Typically, MILMs undergo standard training strategy invol involving two stages. First, to bridge the gap between visual and textual representations, the cross-modal alignment module is trained on large-scale multimodal dataset [1 24 (25), which endows the LLMs with visual-ure sauuull-ndesttndding ability. Then, MLLMs are further fine-tuned on specific visual instruction datasets (2 16 18 26 to facilitate various downstream tasks J.. Despite the significant
advancement, MLLMs still suffer from hallucinations, which decrease their performance on multiple tasks and attract increasing attention from researchers.
2.2 Hallucinations in MILMs
Hallucinations in MLLMs 9 I0 denote inconsistencies between the input image and the generated response. Unlike hallucinations in LLMs (27 288. those observed MLLMs are more complicated, which attracts more attention from researchers. Some methods [26 29 focus on reducing halluci- nations by constructing high-quality datasets, while others employ specialized mechanisms such as decoding strategies (30 31), retrieval augmented generation [32], and chain-of-thoug!e (33) to
mitigate ballucinations. However, due to the inherent limitations of cross-cntropy loss, these methods may provide insufficient guidance for modality alignment. Recently, reinforcement learning- -ased 34], leveraging techniques like DPO [17] and PPO [35), have emerged as promising direction. Yet, these methods rely on preference datasets annotated by humans AI, which are costly and susceptible to noise. Besides, they follow the traditional binary-level preference framework, which is insufficient for preference learning of MLLMs. To address these problems, we propose a novel AMP framework, utilizing a human-free multi-level preference dataset and the MDPO algorithm to guide MLLMs.
methods [2 [3] [4]
Methods
In this section, we delve into the Automated Multi-level Preference (AMP) framework. Initially, we outline two strategies for constructing an initial multi-level preference dataset, aligning with two perspectives of the scaling law [36 37]. Subsequently, we introduce the auto-check mechanism aimed at refining the initial dataset based relevant metries. Lastly, we introduce the Multi-level Direct Preference Optimization (MDPO) algorithm, featuring a novel and robust learning objective.
3.1 Human-free Multi-level Preference Dataset Generation
The quality of the preference dataset significantly influences the refined model's performance. Constructing a high-quality initial preference dataset relies on two fundamental principles. Firstly, the ranking between superior and inferior responses should be correct in most cases. Secondly, the language style among different responses is expected to be consistent. Specifically, inconsistent language styles can introduce biases that mislead the MLLM, resulting reward hacking and performance degradation I2 36. Considering these factors, we propose the Multi-size Expert Generation (MEG) and Incremental Generation (IG) strategies to build reliable preference datasets from the perspectives of model size and dataset size, respectively.
3

---

Multi-leve levii---evel Preference Dataset Generation
Dependency Analysis
palked
wille metochoue,
atso heydles, to otter
Standard Response
Extract Longest
"Mounted
Descsibo thi smage.
The inage priserss soane
Noun Chunks
matorhame are two bayclee, one siver and the other back The image presents scene of Mottohome...
willle matufhotiii wilich parteed on
stront Mounted on the back of the
moterhorm are two ligydie, one alver
Highlight Sentences
and Bhe oteer Nick..
Thee von, adomeed with and white logo on side. equipped with bien..
Generated Responses
The image shows white razpper
van.. The acurted with rei and
The image shows white (3 mppeer nppriio...
wittle logo ios side, is equipood with
tillo madi the stot...
Global Score
Noun Chunks
Expert
a white camper van The van. adomned with and white logo on stde 8 bike rock on the roof
Local Score
Sentence Expert
Instruction
Initial Multi- level Preference Dataset
(a) Dataset Generation
(b) Auto-check Mechanism
Pipeline for Constructing Human-free Multi-level Preference Dataset. We initiate the process with Multi-size Expert Generation and Incremental Generation establish the initial dataset. Then, to enhance the quality of the initial preference dataset, we introduce the Auto-check Mechanism, which calculates both global and local metrics based sentences and noun chunks, respectively.
Figure 2:
3.1.1 Multi-size Expert Generation
Scaling laws suggest that the performance of the model improves as the model size increases. Thus, logical strategy is to generate various responses using models of different sizes. For consistency in language style, it's preferable that these models stem from the same family. Specifically, we adpot LLaVA-based models, such LLaVA-2B (38], LLaVA--B, LLaVA-13B, and LLaVA-34B [3]. Leveraging the standard response in the instruction tuning dataset [29]. we procure up to responses of differing quality.
3.1.2 Incremental Generation
In Multi-size Expert Generation, our focus lies on employing models of various sizes, while Incre- mental Generation involves training datasets of different sizes. In practice, partition the entire fine-tuned dataset F = {I; P; R} into - equal parts for the K K-rank preference dataset, where I. P. and R symbolize the image, text prompt, and standard response, respectively. Then, subsets S; =
F, to fine-tune the pre-trained MILM M, yielding K 2 fine-tuned MILMs, where E [1, K 2]. Hence, the K responses generated by fine-tuned MILMs, along with the response generated by the pre-trained MILM and the standard response constitute the K-rank preference dataset. The entire process documented in Algorithm [
Algorithm 1
The Pscudocode of Incremental Generation for K-rank Preference Dataset.
Input: Image Z, text prompt P, and standard response R for fine-tuned dataset F PI annotated dataset A Pa: Rg },
Output: K-level preference dataset pre- pre-trained MLLM M.
(Ia; Pai [Rg, Ri,..., Rk-1]}.
1: Split F into K - 2 equal parts [F1, F2, .... FK-2]; 2) do
2: for (i = 1 to
3: Train M with subset S;
= F2, (Generate response R; via fine-tuned MLLM M,)
= Get fine-tuned MLLM M;;
4: R² = Pa):
5: end for
6: Ro=Ro, 1I Pa):
7: return D = (Io; Pai RRo Ri, ..., Rx Rx-1I}.
3.1.3 Auto-check Mechanism
In the aforementioned process, we devised two strategies to establish the initial multi-level preference dataset. While the rankings in this dataset are generally accurate, occasional anomalies may lead to incorrect preferences. To enhance the ranking accuracy, we introduce the auto-check mechanism.

---

First, we identify all nouns in the various responses, including terms like "motorhome", "street", etc. Note that certain nouns are deprecated (further details are provided in Appendix (A.I). Next, we analyze the dependency relationships within the sentence to extend each noun into the longest possible noun chunks. For example, "a white motorhome, which is parked on a street" would be represented (denoted by pink color in Fig.
After extracting all noun chunks, we send them into the noun chunk expert (i.e., the text encoder of CLIP [9]) to obtain text features F's {fs, f', fsy and Fc = { f62..... fax J. where M and N denote the number of noun chunks in standard and generated responses, respectively. We then calculate the similarity score as outlined in Eq.
fsm. fan.
S m, n|
m mor(S[m, :I),
(I)
[lfs..
x llfc.
where S E RMxN is the similarity matrix between standard and generated responses. s E R represents the similarity score of generated response Fa. We further introduce the accuracy metric:
[
if s[m] >T
pmm] = 00
Acc = Sum(p)/M,
otherwise
where the threshold, set to 0.85. Accuracy (Acc) reflects the completeness of the credible components within the generated response.
While noun chunks represent the consistency consistency, such as the relationships between multiple objects and the actions objects, etc. To assess global consistency, retrieve the sentences where each noun chunk is located the global representation. The relevant metrics of sentences are the same as those of noun chunks.
local level, entire sentences represent global
Finally, we compute the final accuracy and scores by averaging the local and global metrics. Among the generated responses, the one with the highest accuracy is regarded as the best. In cases where multiple responses achieve equal accuracy, the one with the highest scores is considered superior.
3.2 Multi-level Direct Preference Optimization (MDPO)
34 have demonstrated promising results in training MLLMs with human cumaan-rrefernce datasets. Encouraged by the success of these pioneers, we delve deeper into the potential of multi-level preferences. In this section, we design the Multi-level Direct Preference Optimization (MDPO) algorithm, furnishing a novel and robust learning objective.
Reinforcement learning algorithms
3.2.1 Preliminary
leverage the Proximal Policy Optimization (PPO) algorithm to align with preference data. However, the performance of this approach highly depends on the extra reward model, which sensitive to noises within the preference dataset. Besides, the last stage of PPO fine-tunes the actor and critic model with the online strategy, resulting in high computational costs and unstable procedures. To mitigate these issues, DPO [17] excludes the reward model by analytically expressing reward functions with optimal policy and initial policy Re Denote x: and y as the inputs and outputs of MLLMs, the reward function is converted into:
Prevalent methods [9
T (B(xz)
r(z, v) = - Blg
+ Bog Z(x),
(3)
Teet Tret(yla) (r)
where Z(-) is the partition function, constant. Under the Bradley- Terry model, the policy objective becomes:
CDPO(TE: TTor) = -B (a..n.n [log (r(a, yw.) r(z, v)))
To To (2) 3 log Thee (y (r)
To To( (xc
(4)
-- log Bog
(re((yulz))
where o--) represents the Sigmoid function, and Yur. and yn denote inputs, superior and inferior responses, respectively. In practice, neef remains frozen during DPO training. Thus, only the policy
model na is updated in the training process, ensuring efficiency and cost-effectiveness.
5

---

3.3 Learning Objective of MDPO Algorithm
To facilitate the multi-level preference dataset, we revise the learning objective for K K-rank with K(K 1)/2 comparisons:
K-
K-1
LMDPO Wyet) =
Σ
i= j=++1
K-1
K-1
(V(l)))) (x)
Ta (V)(zzz)
Σ
Σ
E
logo
(Blog
log
Teet (1s
Tree (3/s (x)
where the quality of response Ys is superior to V/-
During MDPO training, observed that despite the loss decreasing normally, the optimized MLLM sometimes generates certain words or phrases repetitively. This occurs because the probability of the policy model producing both superior and inferior responses simultaneously decreases. While the probability of generating inferior responses declines more rapidly, the policy model's capability to generate superior responses also diminishes, leading an overall deterioration in performance. To mitigate this risk, we introduce an additional penalty term, modifying Eq. 4 as follows:
(Nw [x) Trel (Yw x))
To (y) (x)
CDPO-P (To; Tret) (E wD log
Blog
log Trel (y x))
Th (yw x))
+ log
Fee (3 x)
With this penalty term, the probability of generating superior responses is explicitly improved. To minimize the impact of medium-quality responses, we apply the penalty term exclusively to the best response. Consequently, the learning objective of MDPO is formulated as:
CMDPO (TT;; Tyet)
CoPO
(7)
Experiments and Analysis
4.1 Implementation Details
We adpot VLaVA-11.5 [3] as our base model for all experiments, which is built upon Vicuna [22 23
and utilizes ViT-L/14 [19] the image encoder. Our training dataset contains 1k detailed captions
from ShareGPT4V [66] 4k image-text pairs from [34] 4k human-annotade
data from [2 and 2k
multi-round dialogues annotated by us (the annotated process is detailed in Appendix A.2p. forming total of 1 k training instances. For training MDPO, we employ the Adam [1 optimizer for 4 epochs and apply a peak learning rate
10-5 with the cosine decay strategy. To enhance
learning efficiency. we incorporate LoR Abbaed 42 fine-tuning, with a low-rank set to 64 for both attention and feed-forward modules. All experiments are conducted with a batch size of 16 on Nvidia A100 GPUs with 40G VRAM. Further implementation details of the Human-fin free Multi-level Preference Dataset generation are provided in Appendix
A.3 A.3
Evaluation Benchmarks
To verify the effectiveness of our proposed AMP framework, we conduct comprehensive comparisons with various baselines across several benchmarks, including QA-based hallucination benchmark POPE [9) fine-grained hallucination benchmark MMHall-Bech [11,, general benchmark L.LVA. VA-
Bench 2], and our newly developed multi-round dialogue hallucination benchmark MRHal-Bench. Specifically, POPE assesses the object existence hallucinations by prompting MLLMs to provide

---

Table 1: Comparison of conventional MLLMs and RLHF-based MLLMs across MMHal-Benchh MRHal-Bench, and LLaVA-Bench. "MEG" represents training data generated via Multi-size Expert Generation, while "IG" indicates training data produced using Incremental Generation.
MMHal-Bene Bench
MRHal-Bendh
LLaVA-Bent
Methods
Score
Hal.
Score (c/m)
Hal. (c/m))
Conv."
Detailf
Comp.
LLaVA
1.11
0.84
3.01 /3.01
0.40/0.37
85,4
74.3
96.3
InstructBLIP
78
1.80
0.62
3.00/3.00
0.39 / 0.38
83.2
67.6
90.6
LLaVA-v1.57B
2.01
0.61
3.38
0.32/0.29
80.2
75.9
89.2
DeepSEEK-V.
2.22
0.56
3.54/3.53
0.29 0.255
74.4
76.5
78.2
LLaVA-V1.6
3
2.30
0.59
3.80
0.27 / /
82.3
85.3
96.9
MiniCPM-V
45
2.34
0.50
3.31/3.31
0.39 0.34
80.8
75.6
89.2
LLa 1.5vv1..51a 3
2.44
0.53
3.58
0.29 /
81.6
75.5
95.2
Owen-VL--Chal
2.70
0.46
3.71
0.27 /0.21
81.9
77.1
92.3
LLaVA-V1.6 13B
3.04
0.43
3.73
0.30
89.2
90.3
98.3
LLaVA-RLHF
2.04
0.68
3.58
0.34 0.24
85.3
74.7
105.6
LLaVA- RL
2.53
0.57
3.26/3.27
0.45 0.38
93.8
74.3
111.4
RLHF-V 12
2.66
0.52
2.64
0.52/0.56
93.1
75.3
91.6
POVID 14
2.69
3.46/3.47
0.28 00.28
75.7
75.2
89.5
SILKIE 34
3,02
3.71/3.70
0.30
86.3
76.4
95.3
FGAIF [3
3.09
0.36
/.77
0.30/ 0.31
98.2
93.6
110.0
AMP-MEG7B
3.17
0.35
4.07 / 4.006
0.20 0.000.15
89.7
89.1
98.8
AMP-MEG1aR
3.23
0.34
4.21/4.21
0.15 0.11
94.4
91.2
95.6
AMP-IG77
3.12
0.41
4.02 4.04
0.22 0.13
90.2
85.9
99.8
AMP-IG13B
3.18
0.36
3.96/4.01
0.22 0.20
91.3
86.8
99.4
Hal.: Hallucination rate, Conv.: Conversation, Detailed: Detailed Description, Comp.: Complex Question,
c/m: cumulative/meann.
binary responses ('yes' or 'no'). MMHal-Benchh is designed to quantify hallucinations with the assistance GPT-4 43). Different from the simple questions in conventional benchmarks. MMHal- Bench contains more general, open- open-ended, and fine-grained questions. ELLVA-BBende serves as general benchmark for systematic comprehension, encompassing three categories: conversation. detailed description, and complex questions. In addition to these benchmarks, we introduce MRHal- Bench evaluate hallucinations in multi-round dialogues, covering six aspects: attribute, description, existence, counting, reasoning, and spatial relation. For further details, please refer to Appendix A.4
4.3 Comparisons with Leading Methods
We compare our method with multiple MLLMs, including two types of state-of-the-art models: General MLLMs. We include LLa VA (2). InstructBLIP 4). LLaVA-VIs, 3], and Swen-VL-Chat (6] as bigghperforring. open-sourced general models. These models are trained on extensive datasets and demonstrate promising results across various tasks. 2) based MLLMs. Our comparisons also extend to RLHF models such as LLaVA-RLHF [T], RLHF-V [2]. POVID [14) SILKIE B4, and
FGAIF 13.. Specifically, VA-RLF- employs the PPO algorithm on 10k human-prefeerece data and 72k factually augmented data for reward and policy models, respectively. RLHF-V utilizes 1.4k human-annotated, fine-grained preference data to optimize the policy model using the DPO algorithm. Both [3] and [3] apply the DPO algorithm to align MLLMs with GPT-4V preferences. POVID [4] generates hallucination examples through two strategies and also
the DPO algorithm.
The quantitative results are shown in Table T and 2 We observe that our AMP surpasses all general MLLMs across all benchmarks, highlighting the benefits of further fine-tuning with the preference dataset. Besides, our method also achieves tfate-of-the-are performance among which comes from two aspects. First, our MDPO algorithm facilitates multi-level preference learning, which enables the MILM to discern semantic granularity among different responses. Second, accurate ranking of our human-free preference dataset ensures reliable guidance for the MILM,, leading to more promising performance. We also provide some qualitative case studies Fig] For more cases, please refer to Appendix
methods,
A.5 A.5
7

---

Table 2: Comparisons on the POPE benchmark. indicates evaluations using the official model.
Adversarial
Popular
Random
Overall
Methods
FIf
Acc.1
FI1
Acc.1
FI1
Acc.1
FI1
Yes
LLaVA 1313
74.4
67.2
78.2
73.6
78.8
73.7
77.1
73.7
InstructBLIP
4
70.4
65.2
80.2
79.7
89.3
88.6
80.0
59.0
DeepSeek-VL.
44
72.2
65.4
71.3
63.8
76.4
71.5
71.7
73.3
Qwen-VL-Chal
66
80.7
83.2
81.6
84.2
82.1
84.2
81.5
36.7
MiniCPM-V
45
83.5
83.4
86.2
86.5
88.9
89.2
86.2
47.8
LLaVA-V1.57y
84.5
85.5
86.0
87.1
87.2
88.0
85.9
42.2
LLaVA-V1.51ue
84.5
85.5
86.3
87.4
87.1
88.0
86.0
42.2
LLaVA-V1.67p [3]
85.2
86.4
86.4
87.6
87.6
88.5
86.4
41.5
LLaVA-V1.6131
85.2
86.4
86.4
87.7
87.2
88.2
86.3
41.0
LLaVA-RLHF
79.5
80.7
81.8
83.3
83.3
84.8
81.5
41.8
LLaVA-RLHH 13B
80.5
82.3
81.8
83.9
83.5
85.2
81.9
39.0
RLHE-V 12
83.6
84.6
85.3
86.4
87.2
88.1
85.4
42.7
POVID*
84.0
84.7
85.8
86.8
87.7
88.5
85.8
43.6
SILKIE 34
80.3
83.0
81.3
84.0
81.6
83.9
81.1
36.1
FGAIF
13
79.9
79.6
83.7
84.0
86.7
87.0
83.4
48.3
AMP-MEG71
83.4
83.1
87.7
88.2
89.6
89.9
86.9
48.0
AMP-MEG1an
83.4
82.8
88.0
88.2
90.3
90.4
87.2
49.8
AMP-IG7B
82.3
82.5
87.0
87.8
87.7
88.3
85.7
45.1
AMP-IG13B
83.0
82.7
86.0
86.3
89.6
90.0
86.2
46.3
IMP-MEG (Oun)
people skaferm sreiee swnch!
AMP MEG (Durs)
There existence dressed
you
image description.
for Dumplings"
$ LLaVA-V15
pounds.
Image,
group people
standing on train platform dressed as animals. They are wearing costumes that resemble various animals, such as dog. cat, and bird..
LL.VA-V155
In the imago, the man is selling jerk chicken dumplings for $2.00 each.
E LLaVA-RUHF RUMF image,
* LLaVA-RUMF
Homax
Chowry
group people standing animals. They are dressed various costumes, which include..
image, indicates each jerk chicken dumpling conts two pounds.
jerk dunglings displayed sign.
Figure 3: Case studies including our AMP-MEG, LLaVA-V1.5 3. and LLaVA-RLHF 11. Halluci-
nations, correct responses are highlighted in different colors. Please zoom for the best view.
Table
Study on preference quantity.
MMHal- Bench
HiRHal-Bench
LLaVA-Bende
Settings
Scoret
Hal.j
Score (c/m)f
Hal.
Conv.t
Detailf
Comp.1
2-level preference
2.69
0.47
3.71/3.77
0.27 / 0.22
81.5
74.7
83.5
3-level preference
2.88
0.42
3.83 3.87
0.24 / 0.18
84.1
84.6
94.1
level preference
3.17
0.35
4.07
/ 4.06
0.20 / 0.15
89.7
89.1
98.8
5-level preference
2.96
0.41
3.93 3.95
0.22 0.17
88.5
84.8
92.9
4.4 Ablation Studies
Impact Preference Quantity. We explore the effects of varying the number of preferences from 2 to 5, with detailed implementation found in Appendix A.6 As indicated in Table 3 4-level preference identified as the optimal setting. We hypothesize that the diminished performance observed with a 5-level preference dataset may be due to increased hidden noise. Unless stated otherwise, all subsequent experiments are conducted on MILMM using the Vicuna-7B, trained on the 4-level preference dataset produced through Multi-size Expert Generation.

---

Table 4: Impact of the gap between adjacent levels and cross-level comparison. Preferences are
ranked from most superior to most inferior the following order: A, B, C.
MMHal--Bend
MRHal-Bench
LLaVA-Bench
Settings
Scoref
Hal.
Score (c/m)t
Hal.).
Conv.
Detail
Comp.
S>B
2.50
0.50
3.56/3.57
0.28 /0288
79.7
71.5
80.3
S>A
2.61
0.51
3.63 3.62
0.27/0.2
82.8
74.9
84.1
S>A & A>B
2.68
0.43
3.69/ 3.71
0.29/0.22
83.0
79.6
87.0
+Cross-level Comparison
2.79
0.44
3.75 2/5/3.773
0.27 /0.22
87.7
78.6
90.1
S>A & A>B B>C
2.85
0.40
3.86 / 3,90
0,24/ 0.17
90.2
81.3
92.4
+Cross-level Comparison
3.17
0.35
4.07 4.06
0,20 0.15
89.7
89.1
98.8
AsC B>C
2.33
0.57
3.37 3.37
0.34 0.34
75.7
70.4
80.1
2.45
0.51
3.50 / 3.50
0.31 /0.27
77.3
72.2
81.6
Table 5: Ablations on the human-free multi-level preference dataset using different annotations, including Al (i.e., GPT-4V), Auto-check, and initial annotations from MEG and IG.
Preference Annotation
MMHal-
MRHal-Bena
LaVA-BBnch Detailf
Dataset
Scoret
Hal.
Score (c/m)
Hal.
Conv.
Comp.t
AMP-MEG
AI
2.87
0.44
3.87 3,88
0.25
92.3
79.6
89.9
AMP-MEG
Auto Check
3.17
0.35
4.07 4.06
0.20/ 0.15
89.7
89.1
98.8
AMP-IG
Auto Check
3.12
0.41
4.02 4.04
0.22 0.13
85.9
99.8
AMP-MEG
Initial
2.79
0.49
3.69 /,771
0.29 /.222
87.1
80.1
81.7
AMP-IG
Initial
2.80
0.48
3.777
0.26 0.221
89.2
78.3
81.9
Impact of Gap between Adjacent Levels. The efffcctiveness of multi-level preference learning is partly attributed to reducing the gaps between adjacent levels. As shown in Table 4 we reduce the gap by "S>B =S>A" and "A>C =B>C", both of which result in performance enhancements.
Impact of Including More Comparisons. We further introduce more inferior responses, i.e.. "Response B' and 'Response C', by "S>A + S>A & S>B" and "S>A & S>B = S>A & A>B & B>C". The improvements depicted Table f verify that inferior responses are also beneficial for preference learning. To provide more comparisons between the best response and hallucination examples, we devise cross-level comparisons based on settings "S>A & S>B" and "S>A & A>B B>C". As illustrated in Table 4 this strategy brings extra performance improvement across multiple benchmarks, indicating the necessity of cross-level comparisons.
Comparisons with Al-Annotated Preference. Similar to reinforcement learning methods using AI feedback [46], we use GPT-4V [5] directly rank the responses generated by MEG based on their visual faithfulness and helpfulness. Table Jillussttts that training with our preference dataset yields more effective results compared the Af-anootated preference dataset. This suggests that our human-free multi-level preference dataset contains less noise. Furthermore, the performance of the MILM significantly decreases the absence of our Auto-check mechanism, highlighting crucial role in accurately refining the ranking of the multi-level preference dataset.
4.5 Comparisons with other Rank-based Preference Alignment Approaches.
We make some empirical comparisons by replacing our MDPO with the learning objectives of [4 and [48). As reported in Table 6 our MDPO surpasses these two learning objectives on all hallu- cination benchmarks, e.g.. 3.01 >> 3.17 on MMHal-Bench The superiority of our MDPO comes from two aspects. First, our MDPO mitigates the challenge of distinguishing micro hallucinations in responses. Taking 3-level preference an example, the comparisons of other methods are 'A>BC, B>C', while comparisons made by our AMP are 'A>B, A>C, B>C'. More specifically, our AMP splits 'A>BC' into 'A>B, A>C', which enables MLLMs to perceive the subtle differences between different responses. Second, our penalty term explicitly increases the probability of MLLMs generating good answers, ensuring the stability of the training process.
We also conduct some experiments about perturbation- (PB) methods. Our implementation details follows. We randomly change the noun, adjective, preposition, and numeral. We obtain

---

Table 6: Performance on three hallucination benchmarks across other loss functions (#1, #2),
MLLMs from different families (#3, DF), perturbation ceruurbation-based methods (#44, PB).
MMHal HMHhal-Bench
HRRHal-ennch
LLaVA --V-BBncchh
Settings
Scoret
Hal.
Scoret
Hal.y
Conv.7
Detailf
Comp.
47
2.96
0.41
3.85 3.82
0.26 / 0.23
84.1
81.7
88.2
r.
48
3.01
0.38
3,95 3391
0.24 / 0.19
86.2
84.3
91.9
PB
2.83
0.46
3.61 /3.52
0.33 / 0.35
78.4
75.1
81.3
MDPO
3.17
0.35
4.07 /4.06
0.20 / 0.15
89.7
89.1
98.8
MMHal-Bernd Bench
MRHHl-Beach
LLaVA-Barhh
32
95
311
4.0
90
85
80
3.0
75
27
70
10%
30%
50%
70%
100%
10%
30%
50%
70%
100%
G27-4V
mitlznet
Hunne
#UP:-4V
mchama
Figure 4: Performance on three hallucination benchmarks across different proportions of GPT- 4V/human annotations.
answers of varying quality by controlling the proportion of perturbations (10%, 30%, 50% based on 4-level preference setting). As shown in Table perturbation- based methods still inferior to our MDPO. We infer the hallucination pattern generated by random perturbation is different from the real MLLM and is thus not informative enough for preference learning.
4.6 Evaluation of the Automated Multi-level Preference Dataset
We estimate the inconsistency rate of our AMP dataset to be 2.25% (through manual evaluation on 2000 random samples). The 2.25% inconsistency rate is significantly lower than the human (14.40% inconsistency) and GPT-4V (11.95% inconsistency) annotations.
Moreover, conduct another experiment to validate the superiority of our AMP dataset. We mix the AMP and human/GPT-44 data for training the model. Fig. 4 shows that as the proportion of human/GPT-4V annotated data increases, the performance of MLLMs decreases accordingly.
5
Limitations
Our AMP framework offers more effective preference learning from the human- fenaan-free multi-level preference dataset. However, several challenges remain: 1) The quality of standard responses limits the performance of the optimized MILM.. A portion of the standard responses in our dataset comes from superior responses generated by language models, potentially containing imperceptible hallucinations. Besides, the standard response is less helpful despite its high faithfulness, further restricting the performance. 2) Although our AMP successfully reduces hallucinations and promotes the truthfulness of MLLMs, the essence of preference learning is pushing the model to bias the preference dataset, causing decrease in generalization ability. Therefore, finding balance between preference learning and maintaining the capabilities of MLLMs is yet to be explored.
6 Conclusions
In this paper, we introduce the Automated Multi-level Preference (AMP) framework, achieving promising performance on several hallucination benchmarks, which benefits from the reduction of gaps in adjacent levels and the introduction of cross-level comparison. To enable the AMP framework, we propose multi-level preference dataset generation pipeline, aiming to construct a high-quality preference dataset automatically. Furthermore, we design the Multi-level Direct Preference Optimization algorithm, which furnishes a novel learning objective to ensure robust and efficient preference learning. Lastly, we conduct the first hallucination benchmark multi-round dialogues and devise the relevant metrics, which may stimulate future research.
10

---

References
[1] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[2] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[3] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprini urXiv:2310.03744, 2023.
[4] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances Neural Information Processing Systems, 36, 2024.
[5] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. /4824, 2023.
[6] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile languge model for understanding. localization, text reading, and beyond. arXiv preprint, 2023.
[7] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433. 2015.
[8] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1166-3164, 2015.
[9] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large languaelanguage models. arXiv preprint arXiv:2305.1033: 2023.
[10] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00225
2024.
[11] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprini arXiv:2309. 14525, 2023.
[12] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint 12 12.00449, 2023.
[13] Liqiang Jing and Xinya Du. Fgaif: Aligning large vision-language models with fine-grained ai feedback. arXiv preprint arXiv:2404.00 2404005046. 2024.
[14] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modali- ties in vision large language models via preference fine-tuning, arXiv preprint arXiv:2402.11411, 2024.
[15] OpenAl. Gpt-4v(ision) system card. OpenAl, 2023.
[16] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799 2023.
[17] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
11

---

[18] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing languag understanding with advanced large language models, arXiv preprint arXiv:2304. 10592, 2023.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-8763.. PMIR, R, 2021.
[20] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale, arXiv preprint arXiv:2303. 15389, 2023.
[21] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1175-11986, 2023.
[22] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.00 092888 2023.
[24] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[25] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson. Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
[26] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.1455 2023.
[27] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1 1-38, 2023.
[28] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's song in the ai ocean: survey on hallucination in large language models. arXiv preprint vrXiv:2309.0/22
2023.
[29] Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M 3 it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint ar 276.v:2306.04487 2023.
[30] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weim- ing Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and eetrospectionnalllcattoo arXiv preprint arXiv:2311.17911 2023.
[31] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vanion-language models through visual contrastive decoding, arXiv preprint arXiv:2311.1699 2023.
[32] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multi-modal retrievers. arXiv preprint arXiv:2402.08327, 2024.
[33] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning language models. arXiv preprint 20X 2023.
12

---

[34] Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprini arXiv:2312.1 1006065,, 2023.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (:Xiv:/777.0000 2017.
[36] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforeement learning from human feedback, arXiv preprint arXiv:2204.05M0 2022.
[37] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001. .00361, 2020.
(38] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: framework of small-scale large multimodal models. arXiv preprint urX2v:2402. |4289, 2024.
[39] Ganqu Cui, Lifan Yuan, Ning Ding, Guamming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.0137,,0 2023.
[40] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[41] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 17X1v=1711.05101, 2017.
[42] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint wrXiv:2106.09683 2021.
[43] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rXiv:2303.08774, 2023.
[44] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprini wrXiv:2403.05522 2024.
[45] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 240000000000 2024.
[46] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop. Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with feedback. arXiv preprint arXiv:2309.0022 2023.
[47] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037-43067. PMLR, 2023.
[48] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990-18998, 2024.
[49] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
13

---

[50] Yash Goyal, Tejas Khot, Douglas Summers-She Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6904 6913, 2017.
14

---

A Appendix
A.1 Omitted Nouns Auto-check Mechanism
Similar to [5], we exclude some abstract nouns, e.g., "time", "effect", etc. Besides, some high- frequency but unnecessary nouns, such as "image", "photo", etc, are also deprecated. The complete list is depicted in Fig. 5
Unnecessary Nouns 'photo', 'image', 'photograph', 'picture', 'figure', 'painting'.
Abstract Nouns 'harmony', 'beauty', 'feel', 'concentration' life', 'matter', 'feature', 'effect',, 'skill', 'detail', 'emotion', 'nature', 'time', 'background' 'placement', 'perspective' 'universality', 'attention', 'features', 'essence', 'peace' 'compositonn' 'element', 'oregroundd, 'appreciation' 'atmosphere', 'emotions', 'artwork', 'subject'.
Figure 5: Omitted nouns in the auto-check mechanism.
A.2 Annotated Process of Multi-round Dialogues in Training Dataset
We introduce 2k multi-round dialogues in our training dataset. To get diverse questions, we use
GPT-4V [5] generate questions and corresponding responses, where the prompt is Fig.
However, the responses generated by GPT-4V still contain hallucinations. To get the high-quality response, we further employ Qwen-VL-Chat [6]. LLaVA-vI.5 13B (3), LLaVA-RLHI 13B [I]. RLHF-V [12] to provide three extra responses. Thus, together with the response of GPT-4V, we get 5 responses in total. Then, we send these responses to human annotators and ask them to find the optimal response. If the optimal response still exhibits hallucinations, this image-text pair will be deprecated. Through these steps, get the final 2k high-quality multi-round dialogues.
A.3 Details of Human-free Multi-level Preference Dataset Generation
A.3.1 Multi-size Expert Dataset Generation
To make the language style more consistent, we use models from the same family, including LLaVA- 2B, LLaVA-BB, LLaVA-13B, and LLaVA-34BB We leverage the greedy decoding strategy with specific parameters: beams (4), temperature (0.7), repetition penalty (I.1), and max tokens (512).
A.3.2 Incremental Generation
In the Incremental Generation Generation, obtain MILMS with varying capabilities by using train datasets of different sizes. Specifically, we use the 7B version of LLaVA ELaVA-VIL V1. as the pre-trained model and follow the training detail of [ and further fine-tune it on 30k/60k/90k high-quality image- text pairs. The whole dataset contains 10k ShareGPT4V [16], 20k Flickr30k 49). 30k VQAv2 [0],
and 30k LRV (26). Finally, get different responses, including the Ground Truth (GT), responses generated by fine-tuned MI.LMs and VLaVA-V..5.
A.4 Multi-round Dialogue Hallucination Benchmark (MRHal-Bench)
To evaluate the hallucinations in multi-round dialogue, build Multi-round Dialogue Hallucination Benchmark, simplified by MiRHal-Bench" Specifically, ARHal-Bende contains 105 multi-round dialogues, where the length of rounds ranges from 2 5, with an average length of 2.99. The questions in MRHal involve five categories where MILMS tend to generate hallucinations:
Autribute: Visual characteristics of objects, including color, shape, state, type, etc.
Description: Detailed descriptions of objects, behaviors, environments, background, fore-
ground, etc.
15

---

Y ou are an image content annotation expert, and you are seeing a single image. Design a muultiroouna conversation between you and a person asking about this photo.
The conversation format
User. ******** GPT: ********
where the user is the role of the person, and GPT is the role of you.
The answers should be in tone that a visual AI assistant is seeing the image and answering the question. A diverse questions and give corresponding answers. Only include questions that have definite answers: (1) Ask the questions that can be answered confidently. Do not ask ambiguous questions. Do not give any ambiguous answers.
(2) Y our answers may involve some facts about the image, such as news, social, science, etc.
First, your questions may include the characteristics of all the objects or elements in the image, their type, color, and style, the number of objects, the movement of the characters, the precise location, text, etc. Also include complex questions that are relevant to the content in the image, for example, asking about background knowledge of the objects in the image, asking to discuss events happening in the image, etc. Again, do not ask about uncertain details, do not imagine anything. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. Y ou can include multiple paragraphs if necessary. Y our output should only contain the conversations without any other things.
Figure 6: The text prompt for GPT-4V in the annotated process of multi-round dialogue.
Existence: Questions with absolute answers, e.g., yes/no.
Counting:
The number of specific objects.
Questions that require the model to integrate analysis based on image content or other knowledge generate final responses.
Reasoning:
* Spatial Relation:
The relative or absolute spatial relationships of objects in given images.
To obtain fair, objective, and trustable evaluation result, emppoy GPT-4 via meticulous template, Since GPT-4V also suffers from hallucinations, we use GPT-4 API and replace visual contents with category names and a standard human-generate answer, which is similar with [I]. The template will be released our code.
We design two types of metrics for evaluating MRHal-Bench, i.e., cumulative and mean metrics (denoted by c/m in Section 4). The cumulative (Metric.) and mean scores (Metric,.) are calculated by,
105
Metrice
Σ
Sum (Metric:))
Σ Len (Metric)).
(8)
105
Metricu
Sum (Metric;) /Len (Metrici) /105,
where Metrie, E R/ symbolizes the Metric i-th dialogue with j rounds.
MRHal-Benn will be available at this link
16

---

Short-form QA
How many bicycles parked under the tree?
What the of book?
AMP-MEG (Ours)
(Ours)
The nome of the book is "Proverbial Philozophy". "
There are four bicycles pared under the tree.
LLaVA-V1.5
S LLaVA-V1.5
There three bicycles pared under
The name of book "Preverbs."
LLaVA-RLHH ALHHF
The name of the book is "Practical Philosophy".
LLaVA-RLHF
There three bicycles pared under tree.
Long-form QA
AMP-MEG (Ours)
The image shows inside and the time "5443 pm The car likely in Europe, as the tomperature displayed in Colsus, There also of snovilake symbol visible, indicating that might be cold enouph form. The background of the image is not in clear view, but can be inferred that the photo was taken 2007.
dashboard, displaying a temperature reading of "orc"
LLaVA-V1.5
17:43 0.000 29. 05. 2007
The image features dightal display screen, likely from car, showing the current time and temperature.
displayed the upper left the while tomperature shown in the lower right corner The temperature
degrees Celsius, indirating cold day, the illuminated, making the numbers and text visible and to read.
display
& LLaVA- RLHF
The image features dashboard with a digital
temperature display. clock is to 17:25, indicating the time of day. The temporature display shows chilly -8 dogrees Celsius, suggesting
cold day ourside. The dashboard equipped with variety digital displays, including speadameter, fuel gouge, and other essential information for the driver the welliequipped and designed provide driver all necessary information ensure safe and comfortable journey...
Provide an in-depth description of the image, centering on the text and context.
presence these displays indicates that
Existence Hallucinations
AMP-MEG (Ours)
existence bright yellow
image.
S -1.5
Yes, there is person wearing a bright yellow helmet in the image.
LLaVA-RLHF
Yes, see person wearing bright yellow snowbcarder, as they are riingg sia lift up snowy mountain. The presence the yellow helmet that the person taking safety precautions while engaging winter Wearing helmet essential for protecting the head from potential injuries during skiing snowbearding. It can helo prevent serious head injuries, well as provide protection from cold weather conditions. bright
imape. gerson likely
activities.
you able identify wearing bright yellow helmet in the image?
for to spot person on the stopes, which can be helpful case of emergancies or when visibility is limited.
Image Caption
AMP-MEG (Ours)
The image captures picturesque scene of rural life in the swiss Alos, A red and white electric train, bearing logo the Swiss Federal Railways, seen traveling along a track runs adjacent to building that resembles mountain station. The building, with its peoked roof and white walls..
& LLaVA-V1.5
The image features red train traveling down tracks, passing by small building. which be house or station. The train appoars to be passenger train, is pulling Into the station. and
in the scene, scatternd around the area. They may be walting for the train to come to complete stop or simply observing its
people
standing
LLaVA-RLHF
The image features red train traveling down the tracks, passing by small building and mountain. The train positioned the middle the with the building located on the left side and mountain
background. There are several people visible standing near the train and others scattered around the area. A car can also be seen parked near people scene.
scene, with some
What's happening in the scene?
the train, possibly belonging to
Figure 7: Case studies terms of short-form Question Answering (QA), long-form QA, Exis- tence Hallucinations, and Image Caption, including our AMP-MEG, LLaVA-V1.5 3. and LLaVA- RLHF [T]. Hallucinations, correct responses are highlighted different colors.
A.5
Case Studies
In Fig. 7 we present some examples of our AMP alongside other MILMSs, i.e., LLAVA-VI.5 B] and [|]] for intuitive comparison. We focus on four typical scenarios. 1) Short-form
LLa LLLVA-RLI RL
17

---

Question Answering (QA), Our AMP generally provides accurate responses, such as the counting (e.g., "four bicycles") and character (e.g., "Proverbial Philosophy"", Long-form QA. Our AMP outperforms other MLLMs in terms of helpfulness and faithfulness. Specifically, our AMP accurately interprets all the valuable information in the given image, including the time, date, and temperature. In contrast, other MILMs make wrong judgment about or neglect the information in this image. 3) Existence Hallucinations. Compared with responses from other MLLMs, our AMP is not misdirected by "identify person wearing bright yellow helmet" and predicts the non-existence of this person. 4) Image Caption. For the detailed caption, our AMP captures all the significant visual components conrectly and infers the location ("Swiss Alps") from the Swiss national flag and mountains, However, other MLLMs overlook this flag and generate some hallucinations about people and cars. These qualitative results verify the superiority of our AMP framework.
A.6 Implementation Details of Optimal-Leve Experiments
In Section 4.4 we report the performance of K-level preferences, where ranges from 2 to When K is equal to 4, the refined MLLM gets the best performance. However, the performance of optimized MLLM varies from different model pools. Take = as the example, the model pool may be 'Response S&34B& 13B', 'Response S&13B&7B etc. The performance of optimized MLLM varies from different model pools. Therefore, we only report the optimal performance under each level, where the details of model pools reported in T
Table 7: The model pools for each level preference.
Settings
GT LLaVA-34B LLaVA-13B LLaVA-7B LLaVA-2B
2-level preference
3-level preference
4-vel preference
5-level preference
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide our contributions both in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We express our limitations in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, claim our experimental settings, Our code is available this link
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Our code and data is available at this link
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
crriiinnoccccc lc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiiccgguie///eeeeeee CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are described in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:/pp.....puu.ttttttcccccueee
Answer: [Yes]
Justification: Our research follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper for assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswppteeeee wwworeeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [Yes]
Justification: MRHal-Bench is available at this link
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "g6ml3w5eTT", "reviewer": "Reviewer_2sCy", "summary": "This work aims to mitigate hallucinations in Multimodal Large Language Models through preference optimization. Motivated by two limitations of binary preferences widely used in existing work, authors proposed a multi-level preference framework. The framework consists of 1) an automated dataset generation pipeline that converts each image-text pair into an image with multiple text descriptions from superior to inferior quality 2) a Multi-level Direct Preference Optimization algorithm that enumerates over all preference pairs with the standard DPO objective. Additionally, authors introduce a new hallucination benchmark, MRHal-Bench. The proposed framework has been evaluated on three benchmarks: MMHal-Bench, LLaVA-Bench, and MRHal-Bench against 5 base models and 5 preference fine-tuned model. The proposed framework achieves best state-of-the-art on MMHal-Bench and MRHal-Bench, although only improved over the second best FGAIF by a small margin. Authors also include comprehensive ablation studies on the effects of multi-level preference.", "questions": "*  **Artifacts of the using responses from different model size**: Authors mentioned that inconsistent language styles can introduce biases, how does this concern justify the choice of using various responses from models of different sizes in the same model family? Responses from smaller models clearly don't just change the factual information, but also introduce more repetition and incoherence issues (for example, see Li et al. 2023 [1]). Would some simple perturbation-based methods control style and other factors better? The questions on artifacts apply to varying dataset size as well, it would be great if authors can discuss potential artifacts.\n* **Why not use KL-Divergence for penalty** Author added a penalty term to avoid degrading the quality of the superior responses in formula (6), why use an entropy term instead of the standard KL-Divergence based shift penalty term in RLHF? Won't this penalty term allows reward hacking on the penalty?\n* Minor: in formula (5), maybe the outer loop should be 0 to k-2?\n\n[1] Li et al. Contrastive Decoding: Open-ended Text Generation as Optimization.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The application of multi-level preference alignment to the problem of mitigating hallucination in multimodal LLMs is novel.\n* Conduct a comprehensive comparison with existing preference fine-tuned multimodal LLMs and baselines on three benchmarks. Improve over existing methods by a small margin.\n* Provide an extensive ablation study of the multi-level preference term.\n\nAdditionally, automating of the multi-level preference data generation could be a potential strength as well, but currently lacks evaluation to justify its quality (see weakness).", "weaknesses": "I would like to see authors address the following weaknesses: \n\n* **Lack intrinsic evaluation of the automated multi-level preference dataset**. The quality is only implicitly justified by the improvement on the three final benchmarks (L258-L264), which makes it unclear what are the artifacts introduced in the automated data generation. Although human or GPT-4 annotation can be inconsistent sometimes, it is still good to collect some annotations to directly assess how the generated preferences align with the degree of hallucination. Similarly, the current auto-check mechanism is ad-hoc and introduces another component, i.e., CLIP, which could introduce additional errors into the system. It would be good to conduct some evaluation on the auto-check mechanism as well. \n* **Missing comparison with rank-based preference alignment approaches**: Despite being a novel application, non-binary preference alignment has been studied both theoretically and empirically in context other than hallucination in MLLMs, for example Zhu et al. 2023 [1], Brown et al. [2], Myers et al. [3],  Song et al. [4]. It would be great if this work could engage with prior literature on non-binary preference alignment, for example, discussing how does the proposed objective compare with ranking-based approach in prior work?\n* **Missing results of FGAIF on MRHal-Bench** In Table 1, FGAIF has a performance that is considerably close to the proposed methods (-0.14, +0.05) on MMHal-Bench and outperform the proposed method on LLaVA-Bench, yet it's missing results MRHal-Bench. These missing numbers could affect the comparison between the two methods.\n\nReferences:\n* [1] Zhu et al. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons.\n* [2] Brown et al. Safe imitation learning via fast bayesian reward inference from preferences.\n* [3] Myers et al. Learning Multimodal Rewards from Rankings.\n* [4] Song et al. Preference Ranking Optimization for Human Alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** Lack intrinsic evaluation of the AMP dataset.\n\n**A1.** Thanks for your advice. We provide more evaluation for the AMP dataset and the auto-check mechanism.\n\n**Evaluation of the AMP dataset.** We estimate the inconsistency rate of our AMP dataset to be 2.25% (by manual evaluation on 2k random samples). The 2.25% inconsistency rate is significantly lower than the human/GPT-4V (14.40%/11.95% inconsistency) annotations. Below is an inconsistent case in our AMP dataset, where $N_i$ denotes the $i$-th noun chunks:\n\n>**Standard Response**: A little girl ($N_1$) with a purple jacket ($N_2$) is flying a kite ($N_3$).\n>\n>**Response A**: A little girl ($N_1$) dressed in purple jacket ($N_2$) is flying a kite ($N_3$) on the lawn, surrounded by *many people* (other information, including some *hallucinations* denoted by *italic*).\n>\n>**Response B**: A young girl ($N_1$) dressed in purple clothes ($N_4$, similar to $N_2$ but not exactly the same) is flying a kite ($N_3$).\n\nResponse A accurately predicts these noun chunks and thus gets a higher score, leading to A>B. However, the actual ranking is B>A because A has some hallucinations.\n\nMoreover, we conduct another experiment to validate the superiority of our AMP dataset. We mix the AMP and human/GPT-4V data for training the model. Table 1 in the PDF document shows that as the proportion of human/GPT-4V annotated data increases, the hallucination rate rises accordingly. \n\n**Evaluation of the auto-check mechanism.** If we remove the auto-check mechanism of the sampled 2k examples, the inconsistency rate increases from 2.25% to 17.45%. It shows that the auto-check mechanism is critical. \n\n---\n\n**Q2.** Missing comparison with rank-based preference alignment approaches.\n\n**A2.** We make both conceptual and empirical comparisons between our MDPO and prior rank-based preference alignment approaches, showing that our method has several advantages, $e.g.$, higher performance. The details are as below.\n\n**Conceptual comparison.** There are two main differences between our MDPO and other rank-based preference alignment approaches. First, our MDPO mitigates the challenge of distinguishing micro hallucinations in responses. Taking 3-level preference as an example, the comparisons of other methods are 'A>BC, B>C', while comparisons made by our AMP are 'A>B, A>C, B>C'. More specifically, our AMP splits 'A>BC' into 'A>B, A>C', which enables MLLMs to perceive the subtle differences between different responses. Second, our penalty term explicitly increases the probability of MLLMs generating good answers, ensuring the stability of the training process.\n\n**Empirical comparison.** As reported in Table 2, our MDPO surpasses these two learning objectives, $i.e.$, [1], [2] on all hallucination benchmarks, $e.g.$, 3.01 -> 3.17 on MMHal-Bench.\n\n---\n\n**Q3.** Missing results of FGAIF on MRHal-Bench.\n\n**A3.** On MRHal-Bench, the mentioned FGAIF achieves 3.77/3.79 score and 0.30/0.31 hallucination rate. In comparison, our AMP is better, achieving 4.07/4.06 score (the higher the better) and 0.20/0.15 hallucination rate (the lower the better). Notably, the results of FGAIF are based on our re-implementation, because it is still close-sourced in terms of data and code. We are not confident that our implementation is completely correct. Once FGAIF is publicly available, we will update its results and add the comparison to our repo on Github.\n\n---\n\n**Q4.** Artifacts of the using responses from different model size.\n\n**A4.** Using responses from different model sizes does have some artifacts, $i.e.$, some inconsistency. However, the inconsistency rate (2.25%) is significantly lower than the GPT-4V (11%) and human annotations (14%). Please kindly refer to the responses to Q1 for estimation details and examples. \n\nMoreover, we empirically validate that using same-family models is better than using different-family models. We build a dataset that mixes the responses from GPT-4O, Qwen-2 and LLaVA. Training model with this dataset (#3 in Table 2) achieves inferior results than our strategy, $e.g.$ , 2.89 (different family) versus 3.17 (ours) score on MMHal-bench. \n\n**Perturbation-based methods** also improve the baseline ($e.g.$, +0.14 on MMHal-Bench), but is still inferior to our MDPO ($e.g.$, -0.34 on MMHal-Bench), as shown in Table 2. \nOur implementation details are as follows. We randomly change the noun, adjective, preposition, and numeral. We obtain answers of varying quality by controlling the proportion of perturbations (10%, 30%, 50% based on 4-level preference setting). We infer the hallucination pattern generated by random perturbation is different from the real MLLM and is thus not informative enough for preference learning. \n\n---\n\n**Q5.** Why not use KL-Divergence for penalty?\n\n**A5.** KL-Divergence is identical to our penalty term. The mathematical proof is as follows.\n\nAssume $\\mathrm{P}(y_w)$ is the probability distribution of superior responses, $\\pi_\\theta(y_w|x)$ denotes the probability distribution of the model generating superior responses, KL-Divergence is:\n$$\\mathbf{KL}[\\mathrm{P}(y_w)||\\pi_\\theta(y_w|x)]=\\sum_{y_w\\in\\mathcal{Y}} \\left[\\mathrm{P}(y_w) \\log \\frac{\\mathrm{P}(y_w)}{\\pi_\\theta(y_w|x)} \\right],\\tag{1}$$\nwhere $\\mathrm{P}(y_w)=[0,...,1,...,0]\\in \\mathbb{R}^{L}$ and $L$ is the length of vocabulary. Since $\\mathrm{P}(y_w)$ is 0 at all positions except for the superior word, where Equ.1 is re-written:\n$$\\mathbf{KL}[\\mathrm{P}(y_w)||\\pi_\\theta(y_w|x)]=-\\log\\pi_\\theta(y_w|x),$$\n$$\\min\\mathbf{KL}[\\mathrm{P}(y_w)||\\pi_\\theta(y_w|x)]\\Leftrightarrow\\max\\log\\pi_{\\mathrm{\\theta}}\\left(y_{w} \\mid x\\right).$$\nTherefore, our penalty term is actually KL-Divergence.\n\n---\n\n**Q6.** Minor\n\n**A6.** The outer loop is 1 to K-2. We take the responses from the pre-trained MLLM as $R_0$.\n\n---\n\n[1] Zhu et al. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons.\n\n[2] Song et al. Preference Ranking Optimization for Human Alignment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PwdwYNP5SQ", "reviewer": "Reviewer_iDRq", "summary": "In this paper, the authors develop an automated dataset generation pipeline capable of producing multi-level preference datasets without the need for human annotators. This paper introduces a novel multi-round dialogues hallucination benchmark, MRHal-Bench. Additionally, the authors design the Multi-level Direct Preference Optimization (MDPO) algorithm, which employs a specifically crafted learning objective to facilitate multi-level preference learning. Extensive experiments conducted on both the hallucination benchmark and a general benchmark demonstrate the effectiveness of this method.", "questions": "1. Assume we have 3 preference samples: A, B, C. Using the MDPO algorithm, we need to calculate the loss for AB, AC, and BC and then update the parameters. However, why do we need to calculate the loss for BC? Sample B may contain hallucinations; does this affect the model's learning of correct preferences?\n2. This paper does not enhance the visual capabilities of the model. However, in the case study, several OCR tasks and the AMP-MEG model can successfully recognize. Can the authors explain why MDPO algorithm can improve this aspect of ability?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To make the labeling of multi-level preference datasets cost-effective and efficient, this paper proposes an automated dataset generation pipeline capable of producing high-quality preference datasets.\n\n2. To narrow the gap between two preference samples in DPO and make the model more easily distinguish the differences between preference data, this paper proposes a multi-level DPO algorithm that use multi-level preference data to provide a broader range of comparisons with hallucination examples.", "weaknesses": "1. It is recommended to provide more quantitative information on the preference dataset generated by the automated dataset generation pipeline. For instance, the authors could use a subset of the dataset to demonstrate the similarity results compared to human annotators.\n2. In this paper, the authors conduct experiments on three hallucination benchmarks and only one general benchmark. To verify the more general applicability of the method, additional experiments are needed on general benchmarks such as TextVQA, GQA, and IconQA.\n3. In Table 1, the authors compare several MLLMs and RLHF-based MLLMs across MMHal-Bench, MRHal-Bench and LLaVA-Bench. However, the baseline model should be more up-to-date. Could you compare it with more current models such as LLaVA-v1.6, DeepSeek-VL, or MiniCPM-V?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** It is recommended to provide more quantitative information on the preference dataset generated by the automated dataset generation pipeline. For instance, the authors could use a subset of the dataset to demonstrate the similarity results compared to human annotators.\n\n**A1.** Thanks for raising this good discussion. We provide more intrinsic evaluation for the automated multi-level preference dataset and the auto-check mechanism.\n\n**Evaluation of the automated multi-level preference dataset.** During rebuttal, we estimated the inconsistency rate of our AMP dataset to be 2.25% (through manual evaluation on 2000 random samples). The 2.25\\% inconsistency rate is significantly lower than the human (14.40% inconsistency) and GPT-4V (11.95% inconsistency) annotations. Below is an example of an inconsistent case in our AMP dataset, where $N_i$ denotes the $i$-th noun chunks:\n\n>**Standard Response**: A little girl ($N_1$) with a purple jacket ($N_2$) is flying a kite ($N_3$).\n>\n>**Response A**: A little girl ($N_1$) dressed in purple jacket ($N_2$) is flying a kite ($N_3$) on the lawn, surrounded by *many people* (other information, including some *hallucinations* denoted by *italic*).\n>\n>**Response B**: A young girl ($N_1$) dressed in purple clothes ($N_4$, similar to $N_2$ but not exactly the same) is flying a kite ($N_3$).\n\nResponse A accurately predicts these noun chunks and thus gets a higher score, leading to A>B. However, the actual ranking is B>A because A has some hallucinations.\n\nMoreover, we conduct another experiment to validate the superiority of our AMP dataset. We mix the AMP and human/GPT-4V data for training the model. Table 1 in the PDF document shows that as the proportion of human/GPT-4V annotated data increases, the hallucination rate rises accordingly. \n\n---\n\n**Q2.** In this paper, the authors conduct experiments on three hallucination benchmarks and only one general benchmark. To verify the more general applicability of the method, additional experiments are needed on general benchmarks such as TextVQA, GQA, and IconQA.\n\n**A2.** Compared to the baseline, metrics of the MLLM fine-tuned with our MDPO are improved by 3.3 (58.2 -> 61.5) and 1.8 (62.0 -> 63.8) on two general benchmarks, TextVQA and GQA, respectively. \n\n---\n\n**Q3.** In Table 1, the authors compare several MLLMs and RLHF-based MLLMs across MMHal-Bench, MRHal-Bench and LLaVA-Bench. However, the baseline model should be more up-to-date. Could you compare it with more current models such as LLaVA-v1.6, DeepSeek-VL, or MiniCPM-V?\n\n**A3.** We present the performance LLaVA-v1.6, DeepSeek-VL, and MiniCPM-V in Table 3 (PDF document). \nAs shown in Table 3, our method significantly exceeds general MLLMs on hallucination benchmarks, $e.g.$, +0.13 (LLaVA-V1.6) on MMHal-Bench, +0.27/+0.28 (LLaVA-V1.6) on MRHal-Bench, verifying the effectiveness of preference learning and our AMP pipeline.  \n\n---\n\n**Q4.** Assume we have 3 preference samples: A, B, C. Using the MDPO algorithm, we need to calculate the loss for AB, AC, and BC and then update the parameters. However, why do we need to calculate the loss for BC? Sample B may contain hallucinations; does this affect the model's learning of correct preferences?\n\n**A4.** First, loss for BC provides more comparisons among hallucination examples, resulting in a better performance. Second, to alleviate the adverse effects caused by hallucinations in B, we only introduce the penalty term for AB and AC and exclude the penalty term for BC. \n\nWe also conduct another experiment for loss BC. As reported in Table 4 (PDF document), the loss for BC brings extra performance improvement.\n\n---\n\n**Q5.** This paper does not enhance the visual capabilities of the model. However, in the case study, several OCR tasks and the AMP-MEG model can successfully recognize. Can the authors explain why MDPO algorithm can improve this aspect of ability?\n\n**A5.** Improvement on OCR is because our AMP dataset actually contains some preference pairs that are centric to OCR task. Below is an example:\n\n>**Prompt**: What does the sign read?\n>\n>**Correct Response**: NO TIPPING THIS AREA IS MONITORED BY 24 HOUR RECORDED CCTV.\n>\n>**Response A**: NO TIPPING AREA IS MONITORED BY 24-HOUR RECORDED CCTV.\n>\n>**Response B**: NO TIPPING MONITOR 24 HOUR RECORDED CCTV.\n>\n>**Response C**: NO TIPPING.\n\nPreference learning framework can increase the probability of correct responses while decreasing the probability of other responses, which enhances the OCR ability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7x5aC2hbg1", "reviewer": "Reviewer_xo7T", "summary": "This paper presents the Automated Multi-level Preference (AMP) framework for improving MLLMs by addressing hallucination issues. The framework introduces a multi-level preference system for RLHF, aiming to enhance the learning process by providing more granular feedback.", "questions": "See weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The introduction of multi-level preferences rather than binary ones narrows the gap between adjacent levels, enabling MLLMs to discern subtle differences and integrate cross-level comparisons.\n- The automated pipeline for generating high-quality multi-level preference datasets without human annotators is a significant contribution, potentially reducing bias and noise while saving time and resources.\n- Extensive experiments across multiple benchmarks demonstrate the effectiveness of the proposed method.", "weaknesses": "- The contribution of the paper heavily relies on the preference fine-tuning algorithm, showing limited innovation beyond this aspect.\n- The method does not demonstrate significant improvements on the LLaVA-Bench benchmark.\n- The method's performance on the adversarial tasks of the POPE benchmark is moderate, suggesting a need to reconsider the impact of MDPO on model robustness and how to balance performance and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** The contribution of the paper heavily relies on the preference fine-tuning algorithm, showing limited innovation beyond this aspect.\n\n**A1.** In this paper, we introduce the Automated Multi-level Preference (AMP) framework, which involves generating high-quality multi-level preference datasets and the effective learning objective for MDPO. These novel designs ensure AMP achieve promising performance on several hallucination benchmarks. Moreover, we conduct the first hallucination benchmark in multi-round dialogues and devise the relevant metrics, which may stimulate future research.\n\n---\n**Q2.** The method does not demonstrate significant improvements on the LLaVA-Bench benchmark.\n\n**A2.** Despite a slight performance degradation on the LLava-Bench, our AMP shows significant improvements compared to both general MLLMs and RLHF methods on other benchmarks, $e.g.$, 3.09 (FGAIF) -> 3.23 on MMHal-Bench, 3.71/3.70 (SILKIE) -> 4.21/4.21 on MRHal-Bench, and 85.8 (POVID) -> 87.2 on POPE.\n\n---\n**Q3.** The method's performance on the adversarial tasks of the POPE benchmark is moderate, suggesting a need to reconsider the impact of MDPO on model robustness and how to balance performance and robustness.\n\n**A3.** The principle of RLHF is explicitly enlarging the probability of superior responses while decreasing the probability of inferior responses. Compared to the conventional autoregressive function, loss in RLHF ensures MLLMs to fit the characteristics of the dataset at the sacrifice of generation ability/robustness. In the future, we will consider integrating the advantages of autoregressive function to maintain the robustness of MLLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to the reviewers for their valuable comments. We are encouraged they found our method is "**novel**" (Reviewer 2sCy), our method "**provides a broader range of comparisons**" (Reviewer 2sCy, Reviewer iDRq), our automated pipeline is a "**significant contribution, potentially reducing bias and noise while saving time and resources**" (Reviewer xo7T). To address all the reviewers’ concerns, we provide experiments, discussions, and point-to-point responses. We will add these experiments and discussions in the final version.

Tables mentioned in the rebuttal answers are all in the PDF document. We provide Table 1 for **Reviewer 2sCy, Reviewer iDRq**, Table 2 for **Reviewer 2sCy**, Table 3 for **Reviewer iDRq**, Table 4 for **Reviewer iDRq**. The details of tables are listed as follows:

---

**Table 1.** Performance on three hallucination benchmarks across different proportions of GPT-4V/human annotations.

To assess how the inconsistency aligns with the degree of hallucination, we incorporate human/GPT-4V annotations into our automated dataset based on the 4-level setting (the optimal setting). 

**Implementation Details.** On our training dataset, the ratio of contradictory patterns in human/GPT-4V annotations is 16.9%/14.4%. To get preferences from contradictory patterns, we ask human/GPT-4V to annotate all levels directly instead of annotating them pairwise. Then, we integrate the human/GPT-4V annotated dataset into our automated dataset, with proportions of 70%, 50%, 30%, and 10%, respectively. 

**Results.** As the proportion of human/GPT-4V annotated data increases, the hallucination rate rises accordingly, which proves the advantage of our automated preference dataset.

---

**Table 2.**
Performance on three hallucination benchmarks across other perturbation-based methods (\#1, \#2), MLLMs from different families (\#3, DF), perturbation-based methods (\#4, PB), our baseline (\#5), and our MDPO.

---

**Table 3.**
Performance of LLaVA-V1.6, DeepSeek-VL, and MiniCPM-V on three hallucination benchmarks.

---

**Table 4.** 
Effectiveness of loss for BC.

Meta-review: This submission seeks to mitigate hallucinations in Multimodal Large Language Models by optimizing preferences. Motivated by the shortcomings of binary preferences, the authors introduce a multi-level preference framework. 

We appreciate the responses from authors and the discussions with the reviewers. Please add those experiments and discussions in the final version.
Decision: Accept (poster)