CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs
Authors: Jinlan Fu
huangfushenzhen
Hao Fei
Xiaoyu Shen
Bryan Hooi
Xipeng Qiu
See-Kiong Ng
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Models
Preference Optimization
Direct Preference Optimization
Hallucination
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. Recent studies have attempted to mitigate this by applying Direct Preference Optimization (DPO) to multimodal scenarios using preference pairs from text-based responses. However, our analysis of representation distributions reveals that multimodal DPO struggles to align image and text representations and to distinguish between hallucinated and non-hallucinated descriptions. To address these challenges,
In this work, we propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations.
We introduce a visual preference optimization module within the DPO framework, enabling MLLMs to learn from both textual and visual preferences simultaneously. Furthermore, we propose a hierarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations. On the Object HalBench dataset, CHiP outperforms DPO in hallucination reduction, achieving improvements of 52.7% and 55.5% relative points based on the base model Muffin and LLaVA models, respectively. We make all our datasets and code publicly available.

Published as a conference paper at ICLR 2025
CHiP: CROSS-MODAL HIERARCHICAL DIRECT PREF- ERENCE OPTIMIZATION FOR MULTIMODAL LLMs
Jinlan Fu'
Shenzhen Huangfu 1.2- Hao Fei! Bryan Hooi! Xipeng Qiu See-Kiong Ng!
Xiaoyu Shen
Natiooal University of Singapore Fuudn University
Digital Twin Institute, Eastern Institute of Technology, Ningbo
jin jinlan @annaanoonn@@ee cona@gma iimmail.aa com, stencheehhharçff@@@all..com
ABSTRACT
Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. Recent studies have attempted to mitigate this by applying Direct Preference Optimization (DPO) to multimodal scenarios using preference pairs from text-based responses. However, our analysis of repre- sentation distributions reveals that multimodal DPO struggles to align image and text representations and to distinguish between hallucinated and non- non-hallucinated descriptions. To address these challenges, In this work, propose Cross-moda Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We introduce visual preference optimization module within the DPO frame- work, enabling MILMS to learn from both textual and visual preferences simulta- neously. Furthermore, we propose hicrarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations. On the Object Hal- Bench dataset, CHiP outperforms DPO in hallucination reduction, achieving im- provements of 52.7% and 55.5% relative points based on the base model Muffin and LLaVA models, respectively. We make all our datasets and code publicly available.
INTRODUCTION
The emergence of large language models (LLMs) has demonstrated unprecedented intelligence (Chiang et al. 2023 Touvron et al. 2023 Al@ 2024), bringing us closer to achieving human- level AI. Concurrently, building on the toundation of text-based LLMs, research in Multimodal Large Language Models (MLLMs) has also rapidly surged, leading to the development of power- ful multimodal models such as GPT-4V (OpenAl 2023), BLIP2 (Li et al. 2023a), and L.L. VA (Liu et al. 2024c). The current MILMss typically integrate the visual encoder into the text-oriented back- bone LLMs through a connector to achieve the understanding of visual signals (Liu et al.] 2024c Yao al. 2024). Although MLLMs have achieved impressive results, hallucination remains sig- nificant cha challenge, where the model's 's output is not based on the visual input (Bai et al. 2024 jiang
al. 2024).
With the help of Direct Preference Optimization (DPO) (Rafailov et al. 2024) Fig. 2 (a)). text- oriented LLMs have achieved satisfactory alignment with human preferences, which can help pre- vent hallucinations and enhance their ability to meet human needs. However, the alignment tech- niques for Multimodal L.MMs (MLLMs) remain underexplored. A natural approach is to extend DPO from the text modality to multimodal contexts through multimodal DPO (Pi et al.] 2024 Sarkar et al.) 2024a) Fig. (b)). However, simply replacing text preference data with multimodal preference data
'Wor done during intersship at NUS. CCoresponding author.
https://git thub. / JWJGAI /CiPP

---

Published as a conference paper at ICLR 2025
481
whard
PC3
-16
Iollucination Rate
POPE
%
PC238
PCE
PC2
PC1
Objital
MSMtal
AMBER
(a) LLaVA
(b) LL
(c) CIIV
(d) Performance
Figure 1: Comparison of representation distributions and performance across models. Representa- tions are constructed by selecting 150 samples (images, non-hallucinated descriptions, and hallu- cinated descriptions). The image text semantics are represented using the last token embedding from the LLM. (d) is the hallucination rate (lower the better) comparison of different models on hallucination benchmarks, namely ObjHal, MMHal, and AMBER. Findings: (1) DPO struggles to align image and description representations and to effectively distinguish between hallucinated and non-hallucinated descriptions. (2) The proposed CHiP method, which incorporates both image and fine trne-graieed text preferences, achieves better alignment between images and ground trund-truth descrip- tions while increasing the distance between moundd-ruth and hallucinated descriptions. (3) CHiP outperforms DPO and original LLaVA in terms of hallucination rate.
PhiEaaz Uprazistika
Visul Profermee Optimization
Ma
My
ma
Max
max
You
3/
Hw
Hl
M(a) > r(a.a))) se)
r(a, thgn Bul 25 r(R, Plhg 8t)
riz, mir Ma) nia, Me:
r.z. Ma) riz,
(a) DPO
(b) Multimodal DPO
(c) CHiP
Figure 2: Comparison of preference optimization in different scenarios: (a) DPO, (b) Multimodal DPO, and (c) Cross-modal Hierarchical Direct Preference Optimization (CHiP). x represents the instruction. Yu denotes the response preferred by the human over yı. Mw represents the image that is more likely to generate the preferred response Vu than my * (A) and * (a) represent the segments (tokens) involved in the hierarchy reward calculation in the preferred and unpreferred responses.
insufficient to handle complex multimodal scenarios. this work, we identify the limitations of multimodal DPO by visualizing the representation distributions of both images and texts and pro- pose solutions to overcome these challenges. Ideally, for well-aligned MLLMs, the representations of an image and its ground-truth description should be close possible, while the representations of ground-truth and hallucinated descriptions should be more distant. Fig. T shows a visualization of the last token representations of image and text in the LLM (specifically LLaMA (Touvron al., ) for LL. (Liu et al. (2024b), using 150 samples (image, ground ground-truth description, and hallucinated description). Fig. (a b. d) highlights the limitations of multimodal DPO, showing its difficulty in aligning image and description representations and distinguishing between hallucinated and non-hallucinated descriptions, hindering performance improvement in hallucination evaluations.
To address these limitations, we propose Cross-modal Hierarchical Direct Preference Optimiza- tion (CHiP) (Fig. (c)), which enhances the alignment from multiple textual granularities (e.g.. re- sponse, segment, token levels) and visual preferences. Specifically, we introduce Visual Preference Optimization by constructing visual preference pairs, allowing the model to learn preferences from both text and visual modalities, aligming text and image representations more closely. Moreover, we introduce a Hierarchical Textual Preference Optimization to allow MILMSs to acquire preference in- formation at multiple granular levels, namely, response, segment, and token, enhancing their ability to differentiate between hallucinated and non-hallucinated text. To validate the efficacy of CHiP, we evaluate it on four popular hallucination benchmarks under LLaVA-1.6 1.6 and Muffin frameworks. The results show that CHiP outperforms GPT-4V significantly on the evaluated benchmarks. Moreover, on the Object HalBench dataset, based on Muffin and LLaVA-1.0 1.6 models, CHiP outperforms DPO in hallucination reduction, with performance improvements of 52.7 7% and 55.5% relative points, respectively.
2

---

Published as a conference paper at ICLR 2025
To sum up. contributions threefold:
We analyze the limitations of multimodal DPO through image and text representation distribu- tions, emphasizing its failure to achieve mooss-modal semantic alignment and distinguish between hallucinated and non non-hallucinated descriptions.
We propose CHiP address these limitations. CHiP includes hierarchical textual preference optimization module to capture fine- nne-grained (i.e., response, segment, and token) preferences and visual preference optimization module extract cross-modal preferences. We equipped CHiP with various MLLMs, and the results of multiple datasets demonstrate that CHiP reduces hallucinations and enhances cross-modee semantic alignment.
2 RELATED WORK
Multimodal Large Language Models (MLLMs) (Liu et al. 2024c Bai et al. 2023 Dai et al. 2023
He et al. 2024) play a crucial role in multimodal understanding and reasoning tasks by processing both images and text. Their development has been fueled by progress in open-source LLMs (Tou-
vron al., 2023 AI@ Meta
2024
Chiang et al. 2023) and cutting-edge image encoders (Radford
et al. 202 lb Wang et al., 2023c Li et al. 2022
However, misalignment between images and text causes MLLMs face issues like hallucinations and errors. Mitigating hallucinations is a crucial research of MLLMs. Hallucination mitigation strategies generally fall into two categories: training-freet and training-based methods. Training- free approaches (Huang et al.) 2024 Yin et al.) 2023) handle potential hallucinations by post- processing MLLMs' outputs. On the other hand, training-bast based approaches aim to reduce hallu-
cinations through instruction fine-tuning (Liu et al., 2023b Zhang et al. 2024) preference learn-
ing (Gunjal et al. 2024: ei al. 2023b Sun et al. 2023 Yu et al. 2002a Deng et al.) 2024| Wang al. 2024). For example, REVERIE (Zhang al.
(2224) is reflective instruction tuning method that incorporates rationale learning into visual instruction tuning. As for the preference learning, for example, Gunjal et al. (2024) proposed Fine-grained DPO (FDPO) and trained a fine-grained multimodal reward model based on InstructBLIP (Dai et al. 2023).
Different from previous research, we address the lisual-language preference misalignment in MLLMs by introducing a novel cross-modal hierarchical DPO (i.c., CHiP), which simultancously optimizes preferences in both the text and image modalities from a fine-grained perspective. Our approach demonstrates better alignment between the two modalities from the visualization of repre- sentation and reduced hallucination generation.
PRELIMINARIES
Direct Preference Optimization (DPO) (Rafailov et al. 2024) is primarily preference optimization method that focuses on aligning language models with human preferences without the need for explicit reward modeling or reinforcement learning. Given model to be optimized Ta. and the reference policy Te.. which is supervised fine-tuning model, the RL optimization of RLHF can be formulated as:
max (ew:Dyysssalle) [r( v)] 3ak. [oo (x)) No
Tee((y) x²]]
By maximizing the KL-constrained reward objective to obtain the optimal solution and establishing mapping between the reward model and the optimal policy, the representation of the reward function derived as follows:
(y)lz)
(xx, v) log
+ + log ((x),
)ur(y| where x is the input instruction, y is the response, 3 is constant, and (xx) is the partition function.
Given the chosen response Yw. where the evaluator preferred it over the rejected response yı, DPO expected to learn to maximize the reward difference hetween chosen (ywe) and rejected responses (y)). The preference optimization objective becomes:
LDPo = - -E ww) [log (r(x, Mw) - r(x, v())
To (z)
To (
[log a log
3 log
Trer(ywolz)
Trer(yi

---

Published as a conference paper at ICLR 2025
where DPO learns preferences based on the ranking of the entire response, and the action score be formulated as:
log (ylo))
log plule, Mee),
B:EW
where Yi denotes the i-th token of the response During DPO training, the reference model [y))) is usually kept fixed while the policy model no (y(a)) is updated.
METHODOLOGY: CROSS-MODAL HIERARCHICAL DIRECT PREFERENCE OPTIMIZATION
In this section, we will introduce the Cross-modal Hicrarchical Direct Preference Optimization (CHiP). CHiP consists of two modules: (1) Hierarchical Textual Preference Optimization, which incorporates preference optimization at the response, segment, and token levels; and (2) Visual Pref- erence Optimization, which addresses the overlooked visual information.
4.1 HIERARCHICAL TEXTUAL PREFERENCE OPTIMIZATION
Image-based responses are often long and complex, and response-level preference optimization re- lies on rough rankings of response quality without clearly identifying which segments or tokens contain hallucinations. This makes it challenging to assign credit desirable behaviors, leading to reward hacking (Laidlaw et al. (024) and the need for a large amount of labeled data. There- fore, we introduce the Hierarchical Textual Preference Optimization module to assign rewards from fine-grained.
For MILMs, each sample includes an image (m) in addition to prompt x, chosen response Yws and rejected response yn- Multimodal DPO relies on both prompt and image select the preferred response from (ww 3. Next, we will provide a detailed illustration the three levels of preference optimization for text: response-level, segment-level, and token-level.
Response-level Preference Optimization (Cppor). At the response level, DPO in the MLLMs' scenario aims maximize a (r(z, Mari Vi) r(2, Mey 31)), and the objective function can be for- mulated as:
Blog To(Nw [o(Uw|m,r) Meat Ter (m,)
(o m, Teat (yrl m, x)
Lopor = - log
$log
(5)
log (y] can be formulated as:
log (y)
Σ logglurl,, N:
(6)
where yt denotes the i-th token of the response y.
Segment-level Preference Optimization (LDpo.). Intuitively, the corrected segments, particu- larly entity nouns, play a crucial role in eliminating hallucinations and should be assigned more rewards. Following Yu al. (2024)) we assign higher rewards the segments that differ between the chosen response and the rejection response. Based on Eq. 6 the action score for the segment- level can be denoted as:
log TH "F (ylxx, m)
_ (
Σ log log m, + Σ log log m. m.sss) D:Ey
C
E:E
where Ye indicates the segments where changes have occurred. denotes the i-th token of the response y. To prevent the model from being misled into giving higher scores to longer responses, σ is used as a normalization factor, where C = |y| + 7 * |ell By substituting Eq. 7 into Eq. 5 we can obtain the objective function of segment-level preference optimization LDPOs-
1
Token-level Preference Optimization (Crow). For most previous methods, the optimization ob- jective of DPO was constructed based on sentence-level KL divergence/Eq. Ip. However, the output generated from images is an autoregressive sequence, aligning MI.LMss with human values the
4

---

Published as a conference paper at ICLR 2025
token level is natural. Finer-grained alignment only improves alignment performance but also helps the model maintain diversity (Zeng et al. 2024). Unlike response-leve optimization, which computes single reward and KL divergence for the entire response, token-level optimization eval- uates each token individually, with the cumulative token values forming the score of response. The sequential KL divergence be defined as:
Lpoz sa (3 (eqgi. (æ,, m, Mwi Teat ([=))) Baage (a,, m. yi Teat|
(8)
where sg represents the stop-gradient operator, and
Dseg (z,, m. yi Teer|(
Σ
Dru(Fre(yyla,y) (yloo (
(9)
Hierarchical Textual Preference Optimization (HDPO). includes the response, segment, and token-level preference preffreece optimization. It can be formulated as:
LHDPO LDPO, ALDPo, + >Poo
(10)
where À and y represent the weights of LDPO, and Lpok. respectively.
4.2
VISUAL PREFERENCE OPTIMIZATION
To mitigate MLLMs' over-reliance on large language models, we next introduce our Visual Prefer- ence Optimization Module. This module forces the model to make preference judgments based on visual information by constructing pairs of images with preferences as variables. Given a pair of images (me; mi). where mw allows the prompt to my Visual Preference Optimization tries to maximize a (r((z, Mar Uw) r(æ, my, Bw))) and the objective function can be formulated as:
better match the chosen response Yhe compared
( Mww (oW x)
To [mi x))
Lppo, = log Blog
[yr (m,; B log
(11)
[rr m),x)
where the rejection image my can be generated by rotating, cropping, or adding noise the chosen
image m.. The objective of CHiP is combination of the hierarchical textual [E.. 10) and visual
Eq. preference optimizations:
LCH)P = LDPO, + LDPO, + ALDPo. + +Lopok
(12)
Since the entire response and image encapsulate modality semantics, we assign weight of (fully consider) to response-level (Lppo,) and visual preference optimization (Cppo.). À and 7 (< 1) (partially consider) adjust the contributions of segment- and token-level preference opti- mizations. We refer to the model that only considers response-level and visual preference opti- mization as Cross-moda Direct Preference Optimization (CMDPO), which can be formulated as: LCMDPO = Lopo, + LDpor.
Hicrarchical textual preference optimization module tries maximizes of((z, m, (w)))r) m, v1) from different levels, while the visual preference optimization module tries to maximizes (r(x,, my Vu:) r(z, my. Vw)). The combination of the them allows MLLMs choose prefer- ences based on both fine-grained textual and visual modalities.
5 EXPERIMENT AND RESULTS
In this section, we empirically investigate the effectiveness of CHiP in reducing hallucination.
5.1 EXPERIMENTAL SETTINGS
Comparing Models. We consider applying CHiP to two different multimodal LLMs: LLaVA- 1.6 (Liu et al. 2024b) and Muffin Yu et al. 2023). For LLa. we have chosen the model size of 7B, using CLIP (Radford et al. 2021a) as the visual encoder and Vicuna- 1.5- 7B (Zheng et al. 2023 as the LLM backbone. For Muffin, ct al. 2023b) as the visual module and 13B Vicuna v1.0 (Chiang et al. 2023) as the LLM backbone, and version fine-tuned on the VQAv2 dataset (Goyal al. 2017) (released by Yu al. (2024a)).
have chosen model size of 13B, using BETТ3 (Wang
5

---

Published as a conference paper at ICLR 2025
Training Data, There several publicly available training datasets that include preference pairs for multimodal hallucinations. Here, we choose to use the RLHF-V-Datsset (Yu et al. 2024a with 5k training samples as our training dataset.
Baselines. We primarily compare CHiP with standard DPO based on the same models. While other multimodal LLMs cannot be directly compared due to differences in base models, preference data, and alignment methods, we provide these results for reference. LLaVA (Liu et al.] 2024c) Muffin (Yu at..) 2202), LRV (Liu et al. 2023a), LLaVA-RLHF (Sun et al. 2023). InstructBLIP (Dai et al. 2023). Qwen- Twen-VL-CChe -hht (Bai et al. 2023). LLaVA 1.5 2024a). HALVA (13B) (Sarkar et 2024b).
(Liu et al. 2023c). RLHF-V YYu et al.
Benchmarks and Evaluation Metries.
Object HalBench (ObjHal) (Rohrbach et al.) 2018) is a widely used benchmark for evaluating object hallucination. To improve evaluation stability, the benchmark includes 8 diverse prompts and tested on 300 instances. Metrics: Following Yu et al. (2024a); Wang al. (2024), we report both the response-leve hallucination rate (R.) and mention- hallucination rate (M.)
MMHal-Benchh (MMHal) (Sun et al., 2023) is a question-answerigg benchmark that covers
question categories and 12 object topics. Metrics: It uses GPT-4 to assess response quality
(Ova.) )) and hallucination rates (R.).
HallusionBench (Guan et al. 2024) evaluates visual illusions and knowledge hallucinations, fea- turing 346 images and 1129 questions. It was the GPT4-assisted evaluation. Metrics: Question Pair Accuracy (qA). Figure Accuracy (fA), and All Accuracy (aA).
AMBER (Wang et al. 2023a) was designed to be evaluated without LLM assistance. Fol- lowing previous works (Wang et al. 2024), we only consider the generative tasks. Metries: (a) CHAIR (Rohrbach et al. 2018) (CHAIR); (b) Object coverage of responses (Cover); (c)
Response- level hallucination (Hal); (d) Human cognition hallucination (Cog).
Implementation Details. We train the Muffin (13B) (Yu et al., (023) and LLaVA-1.6 (7B) (Lu
al. 2024b) with CHiP for epochs, with learning rate of 5c-7 and batch size of 32. For the training time, LLAVA-1.6 took about three hour to train with CHiP on 4 H100 GPUs, while Muffin took approximately five hours. Hyperparameter: Since our training dataset is RLHF-V dataset Yu
et al. (0024)). we followed Yu et al. (2024a) to set the hyperparameter 3 = 0.5 and followed (Zeng et al.] 2024) to set 7 0.1 for token-level preference optimization. As for the weight of segment-level preference optimization, namely X, we set to X = 1 and À = 3 for the Muffin and LLava dataset set (Sec. 5.3). How to Identify Hallucinated Segments? ? Our training dataset, RLHF-V contains both pre-correction (hallucinated) and post-correction (non-hall -alluecccinntee) descriptions. We enumerate all segments longer than two tokens in rejected responses and classify those absent in accepted responses hallucinations. How construct the rejected images? The rejected images are built based on the chosen image of the forward diffusion process at T = 500 steps (Sec. 6.2).
5.2 RESULTS AND OBSERVATIONS
Tab. I presents the experimental results of applying CHiP to the LLaVA-1.6 and Muffin on four popular hallucination benchmarks. The main findings are listed below: (1) CHiP significantly re- duces hallucinations of base models Muffin and LLaVA-1.6. Compared to the base model Muffin (LLaVA), CHiP reduced response- (R.) and mention-levels (M.) hallucinations by 71.2% (65.3%) and 66.4% (56.7%) relative point on the ObjHal dataset and human cognitive hallucinations (Cog) by 57.1% % (61.9%) and Hal by 45.2% (49.6%) relative point on the AMBER dataset. Furthermore, the consistent improvements of CHiP in question pairs (qA) and visual understanding types (EA) on the HallucinationBencch, as well as in overall object hallucinations (Overall) on the MMHal dataset. (2) Based on Muffin and LLaVA, CHiP consistently outperforms DPO in reducing hallu- cination on the four benchmarks. This indicates that CHiP, which includes the visual preference optimization module and the hierarchical textual preference optimization module, can effectively improve preference alignment performance. (3) LLaVA and Muffin with CHiP achieve fewer hallu- cinations compared to GPT-4 on the Objlial and AMBER datasets. Compared to GPT-4 LLaVA (Muffin) with CHiP reduced hallucination rates at the response level and mention level by 64.0% (54.4%) and 56.2% (46.6%) relative point respectively dataset, the hallucination rate for the Cog metric decreased by 42.3% (38.5% relative point, with continuous reductions observed across several other categories as well.
the Object HalBench. On the AMBER

---

Published as a conference paper at ICLR 2025
Table 1: The results of hallucination evaluation on the Object HalBench (ObjHal), MMHal-Benchh (MMHal), HallusionBench, and AMBER datasets. Values in bold indicate the best performance under the same setting. "f" indicates that a higher value is better for this metric, while "L" indicates that a lower value is better. The baseline results reported in (Yu al.
2024a) for ObjHal and MMHal, (Guan et al. 2024) for HellucinationBencch, and (Wang et al.
2024) for AMBER.
ObjHal
MMHal
HallusionBench
AMBER
Model
R. M.J. Ova.t R.
qAt IAT aAt
CHAIRI Covert Hall Cogl
Referenced Results (Not Directly Comparable)
LLaVA-1.0 (Liu et al.
2024c
63.0
29.5
70.8
Muffin YYu et al.
2023)
50.5
24.5
68.8
LRV [Lu
20
32.3
22.3
78.1
8.8
13.0
42.8
LLL
et al. 2023
38.1
18.9
2.5
57.0
7.7
52.1
39.0
4.4
InstructBLIP Dai ct al. 2023
25.9
14.3
2.1
58.0
9.5
10.1
45.3
8.8
52.2
38.2
4.4
Qwen-VL- Chat (Bai et al 2023
VL-Chat
43.8
20,0
2.9
43,0
5.9
6.7
39.2
6.6
53,2
31.0
2.9
LLaVA-1 1.5 5 (Liu ct 2023c
46.3
22.6
2.4
52.1
10.6
24.9
46.9
7.8
51.0
36.4
4.2
RLHF-V Yu cl
2024a)
12.2
7.5
2.5
51.0
6.3
46.1
25.1
2.1
HALVA Sarkar et al. (2024b)
13.9
20.1
49.1
GPT-4V (OpenA] 2023)
13.6
7.3
31.3
28.8
39.9
65.3
4.6
67.1
30.7
2.6
Muffin (13B)
21.5
11.6
2.4
60.42
16.0
50.9
8.0
48.3
32.1
3.5
+DPO
13.1
6.6
2.5
52.1
17.4
23.4
52.5
6.2
46.9
26.5
2.5
+CHiP
6.2
3.9
2.6
49.0
19.1
24.9
54.0
4.4
45.3
17.6
1.5
LLaVA-1.6 (7B)
14.1
7.4
2.8
42.7
15.8
20.8
51.6
8.3
61.0
48.6
4.2
+DPO
11.0
6.6
2.7
43.8
22.2
28.3
56.6
5.9
61.0
38.9
3.0
+CHiP
4.9
3.2
2.9
39.6
23.5
26.0
58.5
3.7
57.8
24.5
1.6
5.3 ABLATION STUDY
Effect of Component Combination. To evaluate the contribution of each component in CHiP and the effect of their combinations, we conducted a comprehensive ablation study on CHiP based on LLaVA. The experimental results are shown in Tab. 2 The main observations are follows: (1) Both hierarchical textual preference optimization (HDPO) and visual preference optimiza- tion (CMDPO) are effect. On the ObjHal and MMHal datasets, both HDPO (CHiP-CoPou) and CMDPO (Cdpo,-LpoJ) outperform DPO. This suggests that: (a) more granular preference signals can reduce label ambiguity at the response level (DPO), helping the model learn more effectively; (b) The introduction of visual preference optimization enhances the model's alignment between the image and text. (2) The combination of visual preference optimization and hierarchical preference optimization strategies makes DPO the most powerful. On both evaluation datasets, CMDPO in- troduced segment-level (LDpO.). token-level (Lpot), or both segment- and token-levels (CHiP), resulting in a consistent and significant reduction in hallucination rates across different evaluation perspectives, such as response level and mention level. Specifically, when CMDPO incorporates both segment-level and token-level optimization (CHiP), the response- and mention-levee hallucina- tion rate on ObjHal datasets decreased by 49.6% and 41.3%, respectively.
Human Evaluation. Due incomplete tions on the MMHal, GPT-4 couldn't 't reliably detect hal- lucinations. To make the results more reliable, we invited experts to manually annotate the data to compare CHiP with DPO based on LLaVA. As shown in Figure CHiP and DPO performed equally on 63.5% of samples, with CHiP winning 24%. the 36.5% samples where a dis- tinction possible, CHiP outperformed DPO in 31.6%.
annota-
CHIP Loses
0.00
12.5
CHiP Wins
0000000
24.0
63.5
Tie
02
9.0)
0
10
20
30
40
50
60
Figure 3: Human evaluation results on MMHal-Bench (MMHal).
Strength of Hierarchical Textual Preference Optimization. Hierarchical text preference opti- mization includes preference optimization at the response, segment, and token levels. Here, discuss the impact of their weights. We fully consider response-level since its global textual se- mantics by setting its parameter Fq.. 12). and following Touvron al. (2023), the

---

Published as a conference paper at ICLR 2025
token-level weight 0.1. As for segment-level, given its crucial role in providing fine- ine-grained feedback on the preference of hallucinations, we fully explore the range its weight (as shown in [q. 10). From the results of Fig. 4 we found that the best performance and A = 3 for the Muffin and LL frameworks, respectively, and we adopted these settings in all the experiments presented in this paper.
achieved when À
Table 2: The ablation results of CHiP based on LLaVA. Values in bold denote the best performance.
Table 3: Results of training freezing the visual en- coder (VE) in LLaVA during preference optimization. and v denote the visual encoder states of training and freezing, respectively.
ObjHal
MMHal
Model
MMHal
AMBER
ObjHal
R.4
M.J
Ova.f
R.
Model VE Ova.f R.T CHAIRI Covert Halp
R.J M.
DPO
11.03
6.61
2.73
43.75
LLaVA
2.75
42.7
8.30
61.0
48.6
14.1
7.4
CHiP
4.92
3.21
2.89
39.63
-DPO
9.19
5.77
2.70
42.40
+DPO
x 2.73
43.8
5,94
61.0
38.9
11.0
6.6
-LDP
8.55
5.16
2.69
40.63
+DPO
v 2.71
44.8
5.88
61.6
38.3
10.1
5.7
-PoO
6.08
3.77
2.71
40.75
+CHiP
X 2.89
39.6
3.72
57.8
24.5
4.9
3.2
-LOPO,-Leet
9.76
5.47
2.78
41.71
+CHiP
V 2.68
43.8
3.74
54.9
22.1
5.3
3.3
5,6
ot 8
36
5.4
22
5.5
5.2
Hallucinan natill
CHAIR
32
5.0
20
30
4.8
4.5
4.6
18
4.0
26
24
2
3
2
3
1
3
4
2
3
5
À
(a) Muffin+CHAIR
(b) Muffin+Hal Rate
(c) LLaVA+CHAIR
(d) LLa Rate
Figure 4: Results of Muffin+CHiP and LLaVA+CHiP evaluated on the AMBER dataset with differ- choices of weight À to control the strength of segment-level preference optimization. Findings: When À = 1 (A 3), the best performance of the CHAIR and Hallucination Rate metric is achieved on AMBER based Muffin (LL.VA-1.6).
Impact of Training Paradigm. The misalignment between image and text semantics is signifi- cant cause of hallucinations in MLLMs (Liu al. 2024a). However, most approaches (Wang al., 2024) freeze the visual encoder and train only the connector and LLM during preference optimiza- tion. There raises scientific question: Can full-model training during MLLM preference optimiza- tion reduce hallucinations? To investigate this, we explored the impact of freezing versus training the visual backbone on LLaVA enhanced by CHiP and DPO. The results are shown Tab. Results: DPO achieves lower hallucination rate when the visual encoder is trained, whereas CHiP. which incorporates multi-level textual preference and visual preference optimization, does not achieve the expected reduction in hallucination rate when the visual encoder is trained. A possible reason is that the multiple optimization objectives may dilute the model's focus on image-text semantic alignment during the joint training of the visual encoder.
6 FURTHER ANALYSIS
6.1 GENERAL CAPABILITY ANALYSIS
Preference learning may compromise model's general understanding capabilities. this section, evaluate and analyze the general capability performance of an MLLM enhanced by our CHiP. Specifically, we selected several popular general capability evaluation datasets, including MMMU (val) Yue et al. 2024), MMMU (test), MMB-ENG (Liu al. 2025), MMB-CN ScienceQA (Lu et al. 2022), and LLa Vi-Will (Liu et al., 2024c). We compared the performance of LLaVA and I. CH these datasets, with the results shown in Tab.
8

---

Published as a conference paper at ICLR 2025
Observation: We observe that LLaVA+CHiP outperforms LLaVA five out of the six datasets. This indicates that CHiP slightly improves performance on the MMMU, -LaVA-Wild and MMB- CN while maintaining comparable performance others.
Table The general capability evaluation results. Values in black indicate the best performance, in red show improvement with CHiP, and in green indicate decline. Values with are reproduced as evaluator due to GPT-4-0314 was outdated;
results. For LLaVA-Wild Wild,, we used gpt-4o-2024-055 for MMMU-test, there was lack of official LLaVA-1.6 reports.
MMMU(val)
(MMU((est)
MMB-ENG
MMB-CN ScienceQA
LLaVA Wild
Num Samples
900
10500
6666
6666
4241
90
LLaVA
35.80
31.70*
67,40
60.60
70.10
74.90
LLaVA+CHiP
36.8+1.0
32.1 0.4
66.6 66.6-9.8
60.82 +0.22
70.15 +0.06
76.2 76.2+1.3
6.2 IMPACT OF REJECTION IMAGE CONSTRUCTION STRATEGY
Preference sample quality depends rejection image quality and its gap from the chosen image. In this section, we explore various strategies for constructing rejection images.
Strategies. The construction rejection images are listed below: (I) Diffussio: Following the for-
ward diffusion process in image generation (Ho al. 2020). small amounts of Gaussian noise are gradually added to the chosen image for T=500 steps. (2) Blackness: set all the RGB values of
chosen image to O. (3) Crop: random cropping strategy utilized to the chosen image. (4)
Rotation: randomly rotate the chosen image by 10 to 80 degrees. (5) Randomness: select an image
from the training set randomly.
(a) Chosen
(b) Diffusion
(c) Blackness
(d) Crop
(c) Rotation
(f) Randomness
Figure 5: Examples of rejection images constructed by different strategies. (a) is the chosen image.
The experimental results of CHiP under different construction strategies of rejection images are shown Fig. 5| Observations: The high similarity between the rejection and chosen images can lead to better preference optimization with CHiP. The dí ffusion in and croppi ng strate- gies represent blurred and sub-images of the chosen image, respectively, both retaining significant amount of the chosen image's visual information, resulting in better performance. However, the blackness and randomness strategies by completely masking and replacing the chosen image
Results.
Table 5: Results of CHiP under different re-
jection image construction strategies. The bold values indicate the best performance. Observation: CHiP achieves the best per- formance with the diffusion strategy con- structed rejection images.
Response Hal
10
5.5
Mention Hall
8
4.5
7
4.0
ObjHal
MMHal
200
400
600
800
1000
200
400
600
800
1000
Strategy
R.
M.
Ova.
R.
Noise Steps
Noise Steps
Diffusion
4.9
3.2
2.9
39.6
Figure 6: Results of LLaVA+CHiP evaluated on the ObjHal dataset with different values of noise step T. "Response" represents the response-level hallucina- tion rate, while "Mention" represents the mention- level hallucination rate.
Black
9.4
5.0
2,4
43.8
Cropping
5.8
3.6
2.8
40.6
Random
10.9
5.9
2.9
41.7
Rotate
7.8
4.4
43.8
9

---

Published as a conference paper at ICLR 2025
World
Co
sraund
PC321
PC200
PC229
PC2
FC2
PC1
PCE
LLaVA
(b) LLaVA+DPO
LLaVA+CMDPO
(d) CLIV
Figure 7: The visualization of representation distributions of various preference optimization strate- gies. Findings: our CHiP makes more aligmment between images and non non-hallucination descriptions and improves the model's ability to distinguish between hallucinatory and non-hallucinatory text.
almost do not retain the information of the chosen image, leading to poorer performance. Although rotation ion preserves much of the chosen image's 's information, it creates significant differences in visual tokens after tokenization, resulting poor performance.
Impact of Noise Step T. We further explored the impact of noise steps T on the performance of CHiP, as shown Fig. for the ObjHal dataset. CHiP performs best T=500. Possible reasons: (1) Fewer noise steps make the rejected image too similar the chosen one, causing label ambiguity and weakening optimization. (2) Too many noise steps erase image details, making distinctions too easy and limiting the module's 's use of visual information.
6.3 REPRESENTATION VISUALIZATION
Ideally, a MLLM with a low hallucination rate should ensure that the representations of the textual description for the image keeping the representations of hallucinated and non-hallucinated texts as far apart as possible. We analyze the effectiveness of CHiP compared to DPO and several ablation preference optimization strategies from a representational perspective. Specifically, we sampled 150 image-ground truth description pairs from the COCO-201 (Lin al., 2014) validation set, and GPT-4 was used to generate more detailed non-hallucinated descriptions (manually verified), well as hallucinated descriptions. We then input the images, non-hallucinated texts, and hallucinated texts into LLaVA separately, and took the representation of the last token of the LLM (in this case, LLaMA) as the representation of the text or the image. We applied Principal Component Analysis (PCA) to reduce the dimensionality of the high-dimensional representations, and the results are shown in 1g..
close as possible the representations of the image itself, while
Compared to LLaVA, DPO struggles with aligning image and text representations but distinguish hallucinated from non-ballucinated text. After introducing image preference op- timization based on DPO, namely CMDPO, the model not only distinguishes between hallucinated and non-hallucinated texts but also brings the representations of the image and the ground-truth de- scription closer. With the introduction more fine-grained text and image preference optimization, namely CHiP, the alignment between the image and ground-truth descriptions becomes even closer, while maintaining the ability to distinguish between hallucinated and non-hallucinated texts.
O Observations:
7 CONCLUSION
In this paper, we tackle the issue of hallucinations in multimodal large language models (MLLMs) by proposing Cross-Modal Hierarchical DPO (CHiP). CHiP integrates visual and hierarchical textual preference optimization, facilitating cross-modas preference capture and finer-grained distinctions. Experiments on four widely-used datasets demonstrate that CHiP effectively reduces hallucciations. We visualized the representations of images, non-hallucinatory descriptions, and hallucinatory de- scriptions. The results show that CHiP, compared standard multimodal DPO, more effectively bridges the semantic gap between images and non-hallucinatory descriptions while enhancing the distinction between hallucinatory and non-hallucinatory descriptions.
httpps:/ wik httpes///een.iiiieeaaa..oooo htw.a///.o...oooooooooooooooooon wwwp.ooooee__aaaaaa.oa
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research supported by A*SRAR, CISCO Systems (USA) Pte. Ltd and National Univer- sity of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 12100100002. This work was also supported by the National Key Research and Development Pro- gram of China (No. U24B20181).
REFERENCES
AI@Meta Llama 3 model card. 2024. URL https://gittuub.coommtttt 1lama/
[lama3/Ba wwwoooomaainin n/MDELL CARD md
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. owen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308. 12966, 2023. doi: 10.48550/RXXIV.30000 12966. URL/ttpss: //oot org/ 10. 48548855/aa20 2308. 12966
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https:/viicua. Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. En- hancing large vision language models with self-training on image comprehension. arXiv preprint arXiv:2405. 291.11965....66.... 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen. Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18135-18141 2024.
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402. /530, 2024.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpu///sss///pooeeeee.....eeee htt.:erccpss.cc/ppar////////aaa 4c5bo htttaaaol...ccaab-taaat........

---

Published as a conference paper at ICLR 2025
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection alllocatioooo In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
Chaoya Jiang, Haiyang Xu, Mengfan Dong. Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27036-27044 2024.
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Preventing reward hacking with occupancy measure regularization. arXiv preprint 2tt.:240330318855 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pp- 19742, 2023a.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint WiXi::2312.10663 2023b.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10965-10975, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hal- lucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representutions, 2023b.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint 440.202222,00 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744, 2023c. doi: 10.4850/ARXIV2231000777 URL https://
doi org/10........o.....oo..o....... 48550/ 2310. 03744
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL /htps::/
llava-vi. http:uu....ooooo//////2//// -110//2224--113000 01- 30- llava-next/
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference Computer Vision, pp. 216-233. Springer, 2025.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
12

---

Published as a conference paper at ICLR 2025
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. CoRR, abs/2403.08730, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Marina
Sutskever. Learning transferable visual models from natural language supervision. Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021a. URL httt:://proceedingg mir. (ress/v139rraffor2222
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint FrXi::88090025550 2018.
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö. Arik, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning. CoRR, abs/2405. 18654, 2024a.
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö Ank, and Tomas Pfis- ter. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprint 1rX5: 2024b.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:23007.022 200700988, 2023.
Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint ar arXiv:2406. / 1839, 2024.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimmesional benchmark for milms hallucination evaluation. arXiv preprint 1 /. /07397,, 2023a.
Wenhui Wang, Hangbo Bao, Li Dong. Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEIT pretraining for vision and vision-language tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 19175-1911111 IEEE, 2023b. doi: 10. 1 www.////222299222....00 7/doi maa.i.org/100 100
URL https:
2023. 01 838
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal. Subhojit Som, al. Image as foreign language: Beit pretraining for vision and languuel tasks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 19175-19186, 2023c.
13

---

Published as a conference paper at ICLR 2025
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint 2408.01 800, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. arXiv preprint arXi::2310.1644 2023.
Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al. Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:2310.00 231.00055,, 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang. Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. RLAIF-V; aligning milms through open-source Al feedback for super GPT-4V trustworthiness. CoRR, abs/2405 17220. 2024b. doi: 10.48550/ ARXIV.2405.17200 URL /tt.:ttte//.....o////4.4445////. (rt.8550//axi.... 2405.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9561 2024.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. In Forty-first International Conference on Machine Learn- ing, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:
//operreview. wett/formmmddd1222kaa
Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. Reflective instruction tuning: Mitigating ballucinations in large vision-language models. arXiv preprint arXiv:2407. 11422, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46622 2023.
14

---

Published as a conference paper at ICLR 2025
A DIFFERENCES BETWEEN TOKEN- AND RESPONSE-LEVEL OPTIMIZATION
Here, we first summarize the differences between token-level optimization and response-level op- timization (in Sec. A.I. For clarity, the following subsections provide detailed derivations of response-level (in Sec. A.2) and token-level (in Sec. A.3) optimization for reference.
A.1 OVERVIEW
The preference optimization function can be divided into two components: (1) the reward function, which quantifies user preferences and drives the optimization direction; and (2) KI. divergence, which controls the difference between the output distributions of the policy model and the preference model.
Given the input instruction x, image m, and response y, the Response-level Preference Optimiza- tion can be formulated as below:
max mn.. yromm (a, m, v)] 30k [T æ, m)) || merly a, m)]
(13)
Ti
where the reward function can be denoted as:
r(r, m, y) log To (y) x m) T m) + 3 log ((x, m),
(14)
And, the Token-level Preference Optimization can be defined as below:
http:///tt...////oollll.l/..ppp [r(z, m, y)] DsegKL (2, m, yi Tre)))
(15)
where the reward function can be denoted as:
rxx, m.
R m, (1)
(16)
where DšeqKL (I, m, y: To Free) denotes the sequential KL divergence, and it can be defined as: DieeKi (r, m. U3 Kref) (KI(TO(-ll,, m, (i
(17)
Comparison:
(1) Reward Function For response-level preference optimization, the reward function is calcu- lated based on the generation probabilities a response by the policy model and the reference model (as shown in Eq. 14). The probability of the entire response is first obtained, and then the reward for the entire response calculated.
For token-level preference optimization, the reward function calculates the reward for each token individually (for example, y". based on the x and m, and S and then sums up the rewards of all tokens obtain the reward for the entire response, shown in Bq. 16
(2) KL Divergence For response-level optimization (as shown in Eq. 13p, KL divergence is calcu- lated as the distance between the distributions of the response y given x and m, as modeled by the policy model and the reference model (note: this is based on the response distribution).
For token-level optimization, KL divergence is computed as the distance between the distributions of yd given and m, and S as modeled by the policy model and the reference model. The overall distance between the policy and reference models is obtained by summing these distances across all tokens. Eq. 17 formulate this process.
A.2 RESPONSE-LEVEL PREFERENCE OPTIMIZATION
Direct Preference Optimization (DPO) Rafailov et al., 2024) is primarily a preference optimization method that focuses aligning language models with human preferences without the need for
15

---

Published as a conference paper at ICLR 2025
explicit reward modeling reinforcement learning. Given model to be optimized ma. and the reference policy Tits which is a supervised fine-tuning model, the RL optimization of RLHF can be formulated as:
max m. yoae [r(a,, m, v)] - 3LL [TO(y m) || Mer(y Ir m)]
(18)
By maximizing the KL-constrained reward objective to obtain the optimal solution and establishing mapping between the reward model and the optimal policy, the representation of the reward function is derived as follows:
To(ulæ, x, m)
(x. m. y) = log
(r, + 3 log (zx, m)
(19)
where x is the input instruction, m is the image, y is the response, / is a constant, and ((x, m) is the partition function.
Given the chosen response ywu where the evaluator preferred it over the rejected response yı. DPO expected to learn maximize the reward difference between chosen (yw) and rejected responses (yr). The preference optimization objective becomes:
LDPO
E (a.m. [log (r(x,, m. Vie r,,,,,,1)) 31)]]
E (a.mwwa))) [log (3 log T0(yul2 m)
(ice x,, m)) 3 log To [z, m) Tee x m
(20)
}]
where DPO learns preferences based on the ranking of the entire response, and the action score can be formulated as:
log (y|z, m)
Σ Im..e))
(21)
WIEN
where Y/ denotes the i-th token of the response y. During DPO training, the reference model [ x. m) is usually kept fixed while the policy model To x m) is updated.
A.3 TOKEN-LEVEL. PREFERENCE OPTIMIZATION
The objective function of DPO operates the sentence level, as shown in Eq. 18 The principle of token-level preference optimization is similar to sentence-level preference optimization. The difference between them lies in the reward function. In token-level preference optimization, the reward function is token-wise wise, which can be viewed as the cumulative reward for generating the text.
where y² E V and y' represents the vocabulary. Additionally, we define y []. Given prompt z, image m, and model-generated response y's first 1 tokens, the LM predicts the probability distribution of the next token can be formulated as Ta (-1[z, m, "1) Therefore, the objective of token-level preference optimization can be denoted below:
Given response composed of T tokens y
T
max E
R(K,, m,
",,'') - Dseegt (x, m, yr गe T Tree ())
(22)
where DSegKL (I,, m, yi To Kref) denotes the sequential KI. divergence, and can be defined as:
DsegKL (I, m, yr no [ye() (Ra(TTO-l(a, m, " (-Ila, m, m,y (I)).
(23)
where DE R[2 m.y "1 is the accumulate reward, and represents weight and is constant.
We follow similar approach response-level preference optimization by maximizing the KL- constrained reward objective to obtain the optimal solution, thereby deriving reward function simi- lar to Eq. 19 And further derive the token-level preference optimization function like sentence-level
16

---

Published as a conference paper at ICLR 2025
shown in Eq. 20 However, Z( m, v='):) * Z(2x, m, B),, which means that optimization at the sentence- level preference pairs can result in the cancellation of policies, while the cancellation does not occur in token-level preference optimization.
Here, we directly employ the Bradley-Terry model to represent the probability of human preferences based the optimal policy. In the KL-constrained advantage maximization problem associated with Eq. 22] the Bradley-Tere model takes the optimal policy wy and the reference policy Tret to expresses human preference probabilities:
(y > yila,, m) (A(2,, m, Mnr, uc) 6(2,, m, Mmr ur),,
(24)
where, A(x, m, ywu Vr) refers to the difference in rewards implicitly defined by the language model Ty and the reference model Theef (Rafailov al. 2024), represented as
To
x. m)
3 log (o xr m)
Max m. you 31) = = fog
(25)
Taat (yw
x, m)
Toot (y) 22, m)
and 8(x, m, yw 31) is the difference in sequential forward KL divergence between the preference pairs (I, m, Mho and (ar, m, st).
(0, m, ywus 31) BDDggLLL (z, m, Ywi Aeet [e) BDseqKL (z. m, yı: Tret [Ta)
(26)
where 3 is the weight. Put them together, we obtain the loss function for token-level preference optimization:
LTDO = - -. log A(x, m. Mw: vi) - 8(r,, m, Ue: 21) ]
To (ow
x. m)
BBoog To( xr, m)
-
log a BBog
(27)
(ee x, m.)
Tree 22, m)
-a( B DisggI. (I, (2,,, y23 Teae| - sg (3 Disggt. (1,, m. y1; Teet (To
Token-level Preference Optimization (Cpox). The difference in reward X m, Dan 91) appears in both the response-lewe and segment-level preference optimizations this work. Therefore, we consider solely sequential KL divergence, which serves as the optimization term for token-level preference optimization:
LPok =sg (3 (BDgeL (x, m. yws Tret (oT)) DD (I, m, [ree
(28)
where sg represents the stop-gradient operator, and
DieegLL (I. m. y: Teerl =
DEL
(29)
t=1
B FURTHER ANALYSIS
B. DOES CHIP MAKE THE MODEL LESS TALKATIVE?
Why does CHiP lower performance on the metric of Cover (Object coverage of responses)? The results in Tab. show that while CHiP reduces the hallucination rate, it also decreases object coverage on the AMBER dataset. This raises a question: does CHiP reduce hallucinations by limit- ing the amount of text generated? To answer this question, we calculated the average output length of LLaVA, LLaVA+DPO, and LLaVA+CHiP across three generative datasets, namely AMBER, multiple-choice task, calculating response length
MMHal, and ObjHal. Since HallusionBench is not meaningful, so we have omitted it here. The results are presented in Tab. 6 We can observe that the output lengths of the three models are comparable. Therefore, CHiP lowers the hallucination rate without making the model less talkative. Furthermore, manual analysis of LLaVA+CHiP's sponses reveals that when an image contains ambiguous or uncertain objects or attributes, CHiP tends to omit mention them, effectively reducing hallucinations.
17

---

Published as a conference paper at ICLR 2025
Why does CHiP lower performance in fA (Figure Accuracy)?
The fA metric is designed to evaluate the model's ' logical consistency, specifically ensuring that responses to questions not based on random guesses. From Tab. 1 we observe that on LLaVA- 7B, CHiP achieves a lower fA compared to DPO, but higher than the original LLaVA VA-7B. A possible reason is that DPO directly targets response-leved preference optimization, focusing aligning the model's s outputs with human preferences at the response level, which makes it better at maintaining logical consistency. In contrast, CHiP's optimization objectives include segment-level, token-level, and image preference optimization. This additional complexity may dilute the focus on logical consistency during the optimization process, resulting CHiP's slightly lower FA compared to DPO.
Table Average output length statistics different models.
Model
AMBER
MMHal
ObjHal
LLaVA
131
164
LLaVA+DPO
127
40
160
LLaVA+CHiP
124
43
151
B.2 ANNOTATOR BACKGROUND IN HUMAN EVALUATION
To mitigate the errors in GPT evaluations caused by incomplete annotations in the MMHal dataset, we introduced human experts for manual evaluation to ensure more reliable conclusions: CHiP has a lower hallucination rate compared to traditional DPO. Details about the annotators: We invited three human experts who specialize in multimodal hallucination tasks. These experts are well- versed in various types of hallucinations, such as object hallucinations, attribute hallucinations, and environment hallucinations. We consider object hallucinations to be more severe than attribute and environment hallucinations. This prioritization allows the expert to efficiently classify and label hallucination types during the annotation process, enabling them to assign reasonable and accurate scores.
B.3 EFFECT OF TOKEN-LEVEL PREFERENCE OPTIMIZATION
In this section, explored whether using token-level preference optimization independently optimization objective would still allow the model to work effectively. The evaluation results on four datasets presented in the Tab. can be observed that solely using token-level preference opti- mization results in lower hallucination rates compared to traditional DPO and the original LLaVA, which holds for the four datasets.
Table 7:
Performance comparison of LLaVA after token-level preference optimization (TDPO) and direct preference optimization (DPO). Values in bold indicate the best performance.
ObjHal
MMHal
HallusionBench
AMBER
Model
Resp.l
Mention]
Overall
Resp-l (qAce)
(fAce)) (aAce)
CHAIR]
Covert
Hall
Cogl
TDPO
9.56
5.50
2.73
42.71
22.64
28.32
57.22
5.90
61.30
37.60
3.10
DPO
11.03
6.61
2.73
43.75
22.20
28.32
56.60
5.90
61.00
38.90
3.00
B.4 FINE-GRAINED ANALYSIS
Here, we conducted a fine- fine-grained grained evaluation of the base model (i.e., Muffin and LLaVA), DPO, and CHiP on the AMBER, HMMallBench (MMHal), and Object HalBench (ObjHal) datasets. The results are shown in Tab. 8 Tab. 9 and Tab. 10 The main finding that CHiP performed well on many fine-grained evaluation metrics across these three datasets.
18

---

Published as a conference paper at ICLR 2025
Table 8: Fine-gree Fine-grained results AMBER. Bold values indicates the best performance. indicates that a higher value represents better performance, while & indicates that lower value is better. Findings: Our CHiP achieves the best AMBER Score under both the base model Muflin and the LLaVA model.
Generative
Discriminative
AMBER
Model
CHAIRI
Covert
Hall,
Cogl
FIf
FIET
FIAt
FIRT
Scoref
Muflin (13B)
8.0
48.3
32.1
3.5
86.4
95.0
79.3
71.4
89.20
+DPO
6.2
46.9
26.5
2.5
79.9
70.4
90.35
+CHIiP
4.4
45.3
17.6
1.5
87.6
96.1
80.5
73.3
91.60
LLaVA-1. (7B)
8.3
61.0
48.6
4.2
87.0
69.6
89.35
+DPO
5.9
61.0
38.9
3.0
87.4
97.8
81.3
63.4
90.75
+CHiP
3.7
57.8
24.5
1.6
86.9
98.3
80.3
62.0
91.60
Table Fine-gree graengrained results on MMHallBench (MMHal). Bold values indicates the best perfor- mance. indicates that a higher value represents better performance, while J indicates that lower value better. Findings: Our CHiP achieves best Overall score and Hallucination rate under both the base model Muffin and the LLaVA model.
Score in Each Question Type f
Model
Overallf Hallu
Attribute Adversarial Comparison Counting Relation Environment Holistic Other
Muffin
2.41
60.42
2.67
3.17
2.83
2.83
2.42
3.00
2.08
0.25
+DPO
2.49
52.08
3,50
2.33
2.92
2.08
2.50
2.67
2.33
1.58
+CHiP
2.58
48.96
3.58
2.58
2.08
3,42
2.83
2.58
1.50
2.08
LLaVA
2.78
42.71
3.75
3,50
3,50
1.50
1.92
4.08
1.75
2.83
+DPO
2.73
43.75
4.17
2.92
3.00
2.67
2.67
4.25
1.17
1.00
+CHIiP
2.84
39.58
4.17
3.33
2.67
2.67
2.25
4.08
1.67
1.92
Table 10: Fine- line-grained results on Object HalBench (ObjHal). Bold values indicates the best per- formance. t indicates that higher value represents better performance, while indicates that lower value better. Findings: Our CHiP performs best on all the fine-grained evaluation metrics (except for the Object Recall) under the base model Muffin and the LLaVA model.
Model
Response Hall
Object Hallf
Response Correctt Object Correct Object Recall
Muffin
21.53
11.61
78.47
88.39
56.29
+DPO
10.65
5.18
89.35
94.82
51.78
+CHiP
6.17
3.91
93.83
96.09
40.99
LLaVA
14.08
7.37
85.92
92.63
55.03
+DPO
11.03
6.61
88.97
93.39
52.83
+CHiP
4.92
3.21
95.08
96.79
48.95
19

---

Published as a conference paper at ICLR 2025
Your task consists of two parts: one is to provide a more detailed and accurate description of no less than 100 wordsbased the ground truth caption of the image. Secondly, modify the description you generated in the first step create similar but incorporate many elements of hallucination.
Here are three examples for your reference:
- Sttrtt of examples-
Example-1:
Ground Truth Caption:
white plate with sandwich and sides on
Non-Hallucinative Caption:
A white plate with sandwich and sides it. The sandwich is made of two slices of bread with filling inside. The sides
small bowl of white creamy dip and some fried onions. There is also a side dish of sliced oranges in a clear glass bowl.
Hallucinative Caption:
A white plate with a sandwich and sides on it. The sandwich made of two slices of bread with filling of ham and cheese inside. The sides small bowl of creamy dill dip and some crunchy onions. There is also side dish of sliced oranges clear glass bowl, served with glass of wine. The plate placed table with a placemat made of woven reeds.
Example-2:
Ground Truth Caption:
An orange truck driving down street full of men in the back.
Non-Hallucinative Caption:
An orange dump truck is driving down the street with several men standing in the back of the truck bed. All the men are wearing red caps and white shirts. There is smaller yellow bus to the left of the truck, and there is light pole in the middle of the frame. Hallucinative Caption:
An orange dump truck driving down a street full of people the back, of them wearing red caps and white shirts. The truck is driving past a yellow bus that is full of people and is about to pass a fire hydrant on the side of the road. The truck is driving very fast and there are police cars chasing it. The men in the back of the truck are holding their hands in the air and yelling, "We're not gonna stop! We' re gonna take back the streets!". Example-3:
Ground Truth Caption:
pizza sitting on top of a pizza box covered in cheese.
Non-Hallucciaati Caption:
and toppings sits inside cardboard pizza box. The pizza appears be into slices and the box has some grease stains. Hallucinative Caption:
pizza with cheese, tomato
A perfectly cooked pizza sits inside a cardboard pizza box. The pizza is covered in melted cheese and pepper- onis and there are traces of red pepper flakes around the edges. A small, green pepper rests top of the pizza. The box is stained with sauce that most likely marinara. -End examples-
Ground Truth Caption: [caption]
Please output this format:
Hon-Hallucinative Caption:"
Hallucinative Caption:]
Table 11: The prompt used generate hallucinated and non-hallucinated descriptions for given input image.
B.5 PROMPT
To reveal the limitations of existing multimodal DPO from the representation perspective and demonstrate how our method, CHiP, effectively mitigates these limitations, we extracted 150 images and their ground-truth descriptions from the COCO-2017 (Lin et al.) 2014) validation set. Gemini (The version we choose to use detailed non-hallucinated and hallucinated descriptions, with the non-hallucinater descriptions man- ually verified. TTb. IT presents the prompts we used to guide Gemini in generating these detailed non-hallucinated and ballucinated descriptions.
soaa-..s-llshh-lateee was then used to generate more
20

---

Published as a conference paper at ICLR 2025
LLaVA
There are six oysters visible photo.
LLaVA +PO
There are six oysters in the photo.
LLaVA +CHiP
How many oysters can you see in the photo?
Question:
There 5 oysters in the photo.
LLaVA
In the reflection from the sunglasses, you can see hlurred image of what appears to be a person wearing top hat and sunglasses. The person in the reflection seems to be wearing similar outfit to the person in the foreground, which includes a top het and sunglasses, but the details are not clear due to the bluc
LLaVA DPO
In the reflection from the sunglassos, wearing the sunglasses and the person in the beckground. The person in the beckgroun appears be weering a white shirt.
see the person
LLaVA CHIP
Question: What can you see in the reflection from the sunglasses?
In the reflection from the sunglassos, be the upper part of another person's bead and shoulders, sugpesting that the person wearing the sunglasses not alone in the image. The have
see what appears to
reflection is somewbat blurre,, indicating that the sunglasses reflective surface that distorts the image being reflected. The details of the other person's appearance are clear due to the reflection's distortion.
Figure 8: Qualitative results of LLaVA+ compared with LLaVA+DPO and LLaVA MMHal- Bench dataset. Correct answers and hallucinations are highlighted green and red, respectively.
C QUALITATIVE RESULT
To demonstrate the effectiveness of our approach, we present qualitative results on the MMHal- Bench dataset in this section, shown in Fig. 8 LLaVA+CHiP demonstrates significant reduction hallucinations text generation tasks. This improvement can attributed to the multi-level preference optimization, which enables the model to capture image-text relationships across varying granularities. Additionally, the visual preference optimization module enhances semantic alignment between the two modalities, further contributing the reduction in hallucinations.
21


Reviews:
[{"review_id": "ZLrNRUwgRf", "reviewer": "Reviewer_NXL3", "summary": "Introduces a new optimisation recipe to reduce hallucinations for multimodal models.\nHierarchical Textual Preference Optimisation tries to solve the problem that response-level preference optimizations have: they can’t clearly identify what segments/tokens contain hallucinations. They consider three levels of preference optimisation for text: response-level, segment-level, and token-level. \nVisual Preference Optimisation tries to reduce reliance on large language models, which start hallucinating (ignoring the image) once the image is not well aligned. They take the ground-truth image and create a synthetic rejection image (using e.g. rotating, cropping, adding noise etc).\nCombining both the hierarchical textual preference optimisation, on all three levels, and the visual preference optimisation, they introduce CHiP. They ablate each component individually, but show that combining all significantly improves performance on hallucination benchmarks.\n\n(update: and they show equality in non-hallucination benchmarks.)", "questions": "1. “Empirically, we set the weight of response-level preference optimisation to 1” (page 8) — can you share those empirical results?\n2. “We found that the best performance was achieved when λ = 1 and λ = 3 for the Muffin and LLaVA-1.6 frameworks” (page 8) — are you not afraid of overfitting on the dataset with these hyperparamters? You’re not using a validation set vs test set here, right?\n3. On page 6, \"Implementation Details\", how were 0.5 and 0.1 chosen?\n3. What is the impact of this preference optimisation stage for the performance on existing benchmarks? You show it improves on hallucination benchmarks, does it keep its performance on previously measured benchmarks?\n4. \"To make the results more reliable, we invited experts to manually annotate the data to compare\" (page 7) -- Do you have details about these annotators? How many/anything special that can be shared?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* All code and datasets are public.\n* Well written.\n* Is very clear about how baselines should be compared (not everything can be directly compared, but they still include them).\n* Introduces all benchmarks and evaluation tasks/metrics in the paper with a quick overview. \n* Results seem strong for the benchmarks this paper focuses on. Good analysis of the components that were used to achieve this.", "weaknesses": "1. While showing improvements on the benchmarks that this paper focused on, it does not show the impact on non-hallucination benchmarks. This makes it harder to estimate whether this is useful in training recipes in ‘the wild’.\n2. All of the ablations/experiments to hill-climb results were done using the final evaluation sets (?). HallusionBench is used in the final overview, and it still outperforms DPO, but the results don’t seem to be as clear as the benchmarks that were mainly used during hyper parameter estimations.\n3. Nit: \"Reference results (these results cannot be directly comparable)\" (page 7) -> \"cannot be directly compared\" or \"are not directly comparable\"\n4. Nit: not sure if we already need to mention \"We select the strategy with the best performance, which generates the rejection image by adding noise ... \" (page 5) there, since it seems to be part of results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B5dh9Wpexw", "reviewer": "Reviewer_Gv4h", "summary": "This paper introduces Cross-modal Hierarchical Direct Preference Optimization (CHiP). This approach incorporates a visual preference optimization module alongside a hierarchical textual preference optimization module (preference learning in response, segment, and token-level), allowing MLLMs to learn from both visual and textual preferences across various levels of granularity.", "questions": "1. The explanation of Segment-level Preference Optimization is not clear. Author claims that \"we assign higher rewards to the segments that differ between the chosen response and the rejection response.\". How the non-hallucinated and hallucinated segment pair is determined? If not human labeled, will the wrongly labeled segment affect optimization effect? Meanwhile, the Token-level Preference Optimization seems strange. From the equation9 we can see the whole sequence is calculated during the reward calculating, but why use the whole sentence when non-hallucinated and hallucinated segment pair is labeled? Will the token-level strategy work if only consider the token in labled segment?\n\n2. Lacks general capability evaluation, it is known that preference learning may harm general understanding capability. Can arthors provide results on general capability benchmarks, such as MMMU and MMBench? Meanwhile, more human evaluation is required, does proposed method eleiminate hallucination by making model less talkative, for example, the cover rate in AMBER decreases after optimization.\n\n3. The proposed visual preference optimization and the result is interesting. However the IMAGE CONSTRUCTION STRATEGY remain exploration. From results, we can see that rejection image that differs a lot from original image can lead to sub-optimal results. A strategy may outperform diffusion is finding rejection images as close as possible. For example, in figure5, a rejection image maybe also include a bear, ocean, mountain, but inconsistent with the correct response.\n\n4. Formatting errors. Name unclear and inconsistence (CMDPO in Table3 but not mentioned, and HCMDPO in figure9), table 6 written as figure.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes to eliminate hallucinations from multiple levels, including response-level, segment-level, and token-level.\n\n2. The proposed Visual Preference Optimization is interesting, make model less over-relied on language counter part.\n\n3. The proposed method achieves solid results on multiple hallucination benchmarks.", "weaknesses": "1. The technical details of some methods are not clear enough.\n\n2. lacks experiments on general capability benchmarks.\n\n3. The selection of rejection image in proposed Visual Preference Optimization remain improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PJZN70e4Uc", "reviewer": "Reviewer_xtwk", "summary": "Direct Preference Optimization (DPO) is frequently used to address the hallucination problem in Vision-Language Models (VLMs). The paper identifies a limitation with DPO, noting that it does not effectively align image and text representations, and proposes Cross-modal Hierarchical Direct Preference Optimization (CHiP) as a solution to this issue.", "questions": "- How are segments determined in Segment-level Preference Optimization? Are partially pre-modified sentences prepared for this purpose?\n- I initially thought Response-level Preference Optimization also involved optimization at the token level. Could you clarify the difference between Token-level and Response-level Preference Optimization?\n- In Equation (8), does \"sg\" refer to stop gradient? I’m curious about the reasoning behind this.\n- What could be the reason for the lower performance in fA and Cover in Table 1?\n- In Table 3, what insight is expected from the experiment comparing training vs. freezing the Visual Encoder, and what is the purpose of this comparison?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is highly readable.\n- It presents a novel perspective on the limitations of traditional DPO, highlighting that it does not apply weight for important segments, resulting in the possibility of applying DPO even to non-hallucinatory segments of a rejected response.\n- The idea of assigning higher rewards to important segments is innovative and convincing.\n- The approach of learning both Textual and Visual Preferences is new and valuable.\n- The experiments effectively demonstrate the hallucination mitigation, and the overall results are strong.", "weaknesses": "- While the paper mentions that Visual Preference Optimization trains the model to understand images that better match the chosen response, the concept of \"Visual Preference\" itself is unclear.\n- The use of augmented images as rejection images is confusing, as it is unclear why semantically similar images are used as rejection images. In Figure 5, except for (c) Blackness and (f) Random, the other images share the same semantics. This approach differs from typical augmentation techniques that increase model robustness by training with transformed images. If (c) and (f) are used as rejection images, it can be understood as a way to prevent generating responses without observing the image. However, other images retain the same semantics, making it difficult to consider them as proper rejection images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)