Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization
Authors: Xintong Li
Chuhan Wang
Junda Wu
Rohan Surana
Tong Yu
Julian McAuley
Jingbo Shang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal
Importance Sampling
Direct Preference Optimization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: MISP-DPO improves multimodal alignment in DPO by selecting semantically meaningful, diverse image negatives through importance sampling.

Abstract: Direct Preference Optimization (DPO) has recently been extended from text-only models to vision-language models. However, existing methods rely on oversimplified pairwise comparisons, generating a single negative image via basic perturbations or similarity-based retrieval, which fail to capture the complex nature of multimodal preferences, inducing optimization bias and hallucinations. To address this issue, we propose MISP-DPO, the first framework to incorporate \emph{multiple}, semantically \emph{diverse} negative images in multimodal DPO via the Plackett-Luce model. Our method embeds prompts and candidate images in CLIP (Contrastive Language–Image Pre-training) space and applies a sparse autoencoder to uncover semantic deviations into interpretable factors. Negative samples are selected based on reconstruction difficulty, semantic deviation from the positive, and mutual diversity, yielding broader and more informative supervision. To handle multi-negative comparisons, we adopt a Plackett–Luce objective and introduce an importance sampling strategy that improves training efficiency. Experiments across five diverse benchmarks demonstrate that MISP-DPO consistently improves multimodal alignment over prior methods, validating the effectiveness of semantic-aware, multi-negative sampling in preference-based learning.

Published as a conference paper at ICLR 2026
IMPORTANCE SAMPLING FOR MULTI-NEGATIVE MUL- TIMODAL DIRECT PREFERENCE OPTIMIZATION
Xintong Li', Chuhan Wang'* Junda Wu', Rohan Surana', Tong Yu?, Julian McAuley', Jingho Shang' 'University of California, San Diego Adobe Research
(xi1240, chw1 36, juw069, rsurana, jmcauley, ishang jshang)@ucsd.ec. @uccsd tyu@adobe.. com
ABSTRACT
Direct Preference Optimization (DPO) has recently been extended from text-only models vision-language models. However, existing methods rely on oversim- plified pairwise comparisons, generating a single negative image via basic per- turbations or similarity-based retrieval, which fail to capture the complex nature of multimodal preferences, inducing optimization bias and hallucinations. To ad- dress this issue, we propose MISP-DPO, the first framework to incorporate mud- tiple, semantically diverse negative images in multimodal DPO via the Plackett- Luce model. Our method embeds prompts and candidate images in CLIP (Con- trastive Language-Imag Pre-training) space and applies sparse autoencoder to uncover semantic deviations into interpretable factors. Negative samples are se- lected based on reconstruction difficulty, semantic deviation from the positive, and mutual diversity, yielding broader and more informative supervision. To han- dle multi-negative comparisons, adopt Plackett Luce objective and intro- duce an importance sampling strategy that improves training efficiency. Experi- ments across five diverse benchmarks demonstrate that MISP-DPO consistently improves multimodal alignment over prior methods, validating the effectiveness of semantic- aware, multi neggtive sampling preference- learning.
INTRODUCTION
Direct Preference Optimization (DPO) [Rafailov et al.) 2023 Amini et al. 2024) has shown great promise for aligning language models by learning from pairwise comparisons, bypassing the need for explicit reward modeling. Recent efforts have extended DPO to multimodal contexts, enhancing dision-language model (VLM) alignment through image-text feedback Boodback(Wangggg et al. 2024a] Tiang
et al. 2024 Deng et al.) 2024 Fu et al.] [0225] Liu et al.) [2025a] Wu et al.] 2025 Xing et al.) 2025). While these approaches improve multimodal alignment, they also expose new challenges, most notably persistent hallucinations and spurious correlations (Wang et al. 2024a Fu al. 2025
Wu et al. 2025).
A key limitation of existing multimodal DPO methods lies in how negative examples are constructed.
Most approaches generate only a single negative image per comparison, typically via adversarial
cropping, random perturbations, similarity- -based retrieval (Liu et al. 2025a Fu et al. 2025 Wu ct al. 2025 Xing et al.) 2025). This reduces supervision to a oie-dimensoonal deviation from the positive image, severely underspecifying the space of visual errors. For instance, illustrated in Figure avoiding single negative depicting a "green apple" might teach the model to reject green hues but ignore mismatched contexts like "kitchen counter" incorrect objects like "pear." As result, optimization occurs along a narrow deviation axis, encouraging spurious shortcuts and lim- iting generalization across orthogonal failure modes. Even recent listwise extensions suffer from similar issue. Although multiple negatives are introduced, they are typically generated by perturbing a single positive image, resulting in homologous variants with substantial semantic overlap (Zadeh) a (2255] Liu et al. 2025d). The learning signal remains concentrated around closely related devi- ations, rather than spanning diverse semantic discrepancies (Liu et al. 2025b c). This oversimplifies
"These authors contributed equally this work.

---

Published as a conference paper at ICLR 2026
Question
CLIP
What shown
Chosen concat
Text Encoder
polided worden
xuady rouen.
Pos Img Vector
PosVecd Texttvae TextVec- NegVec
Sparse Autoencoder
Positive Img
Encoderr
Vision
Candidate Neg lmg Vectors
SAE Activation Matrix
Candidate
Selected Neg Imgs
unit
Figure 1: Overview of the MISP-DPO framework, which integrates CLIP encoding and sparse au- (oencoder-guides selection to identify diverse negatives for multi-negative preference optimization.
the rich space visual inconsistencies and constrains the model's ability disentangle distinct sources of hallucination.
The core challenge is that images lack explicit, compositional units like text tokens, making diffi- cult to isolate meaningful visual deviations (Sahin et al. [024] Zeng et al. 2 Zheng et al.) [2024] Hsieh et al. 2023 Kamath et al., 2024 2023). Naive perturbations often destroy overall coherence without isolating meaningful deviations, making it difficult to systematically explore the negative factors model weaknesses. Effective learning requires disentangling multiple latent error factors, such object identity, color, spatial layout, contextual mismatch, while preserving prompt rele- vance. Existing methods implicitly entangle these factors within a single negative, leaving models blind to orthogonal dimensions of failure.
To address this, we propose MISP-DPO, a framework that introduces multi- multi-negative, semantically diverse supervision into multimodal DPO. We tackle the underexplored problem of constructing multifaceted visual negatives that capture distinct semantic deviations, enabling richer and more structured preference signals. Our approach consists of two stages,
the first stage, we select diverse image-side negatives from large open-domain pool. Prompts and candidate images are embedded in CLIP (Radford al. 2021) space, and sparse autoencoder (SAE) decomposes their semantic differences into disentangled latent factors (e.g., object, color, layout). We prioritize negatives based deviation from the positive sample, and (3) mutual diversity, ensuring broad coverage of negative types.
reconstruction (informativeness). (2) semantic
In the second stage, we integrate these multiple negatives into generalized DPO objective us- ing the Plackett-Luce model. Rather than relying on binary comparisons, our approach ranks positive image above diverse set of negatives, forcing the model resolve multiple constraints simultaneously. further introduce an importance sampling scheme guided by SAE-derived scores, improving training efficiency.
We evaluate MISP-DPO on five multimodal benchmarks (Sun et al. 2023 Guan et al. 2024
et al. [024 Tong al. 2024 Li al. (202aa) focused on hallucination reduction and visual ground- ing Our method consistently outperforms strong basclines, achieving notable hallucination reduc- tion and improved alignment, including 30.09% average improvement over LLa
(Liu
et al. 2023).
Our contributions as follows:
We introduce a multi-negative multimodal DPO framework that provides semantically diverse supervision orthogonal visual deviations, moving beyond single-axis negative optimization.
2

---

Published as a conference paper at ICLR 2026
We propose efficient negative sampling strategy based CLIP embeddings and SAE-guided importance sampling, providing informative and diverse visual discrepancies from open-domain image pools.
Extensive evaluations demonstrate that our method substantially reduces hallucinations and achieves robust multimodal alignment across multiple benchmarks.
2 RELATED WORKS
Multimodal Direct Preference Optimization. DPO (Rafailov et al., 2023) has become a widely adopted method for aligning LLMs with human preferences due its simplicity and stability. How- ever, when extended to multimodal scenarios, especially for hallucination-prov DPO often fails effectively incorporate visual signals, leading models to overfit textual biases and ignore image-grounded constraints. To mitigate this, recent works have adapted DPO for multimodal hallucination reduction by incorporating visual preference supervision. mDPO (Wang et al. 2024a) introduces conditional preference learning and reward anchoring. using lightweight perturbations (e.g., cropping, diffusion) to construct visual negatives. CHiP (Fu et al. 2025) further complements this with hierarchical textual supervision and a visual contrastive loss better align fine-grained text and image semantics. While both methods demonstrate notable gains on hallucination benchmarks, they rely on limited forms of visual augmentation, often constrained to local perturbations with nar- row semantic variation. Other approaches, such as S- V-V (Wu et al. (025) and Re-Align (Xing al., 2025), explore counterfactual or retrieval-basee visual negative generation, but the cost of high computation and limited scalability. In this work, we follow the Inlluciinationcennti prefer- ence optimization paradigm initiated by mDPO, and propose a scalable framework for generating informative visual negatives tailored for multimodal preference learning.
tasks, standard
Multi-negative Preference Optimization. Recent works in textual and recommendation do- mains (Amini et al.) 2024 Shi et al.) 2024 Baruch al.) have extended DPO to multi-negative settings, ranking positives above multiple negatives to enhance robustness. For example, Softmax- DPO (Chen et al. 2024a) and DMPO (Sh et al. 2024) adopt soft ranking Plackett-Luce objec- tives to reduce noise sensitivity. However, such techniques remain underexplored in vision-language models, where negatives must capture subtle cross-modal semantic shifts. Inspired by findings in attribute-based recognition Yann et al.) 2023 Wang et al., (024b) showing that compact, curated subsets can match large noisy sets, adapt this insight multimodal preference learning. Our framework uses a sparse autoencoder in CLIP space to select semantically diverse negatives, en- abling soprraance-eiihhte ranking over multiple contrastive examples and capturing fine- fine-grained failure modes more effectively.
3
PRELIMINARIES
3.1
MULTIMODAL DIRECT PREFERENCE OPTIMIZATION
DPO (Rafailov et al. 2023) provides principled way to align learned policy with human pref- erence judgments without explicitly modeling rewards. In the RLHF framework, solving for the optimal policy π under a fixed reference policy Feee yields latent reward function
x <² x) + Z(x),
r(x, v) B log
Teer x))
where 3 scales the strength of alignment and (xx) a prompt-depended normalizer. Substituting this into the -rarley-Terry-Luuee model and dropping 7(x) gives simple training objective for parametric policy ne.
(D) = -E M. log a (Blog
ma(yala) (oa((pp)))
B log
malye Teat(We
Recent work extends DPO to vasion-language models (VLMs) by incorporating visual preferences. Let denote the multimodal prompt, m, preferred image aligned with textual response yp, and m,, a rejected image. The multimodal reward r(m, x, y) now depends on visual grounding, with preferences modeled as,
p" (up yn m) (r(m, Up) rm,,
3

---

Published as a conference paper at ICLR 2026
To ensure fair comparison across images. we hold V/ fixed and vary only the image imput, the multimodal DPO loss (Wang et al. 2024a Fu et al. 2025 Wu et al. 2025) focuses on visual dis- crimination,
LpPO (0) = -IE a J log (33 log mo(y (ny.))
B log se(yulmn.a)
]]
(3)
Meet (m)))))
This formulation supports joint optimization visual and textual inputs, enabling the policy to associate preferred images with relevant multimodal features.
3.2 MULTI-NEGATIVE PREFERENCE OPTIMIZATION
Multi- Multi-negative preference optimization (Chen et al. 2024a) extends the Direct Preference Optimiza- tion approach (Rafailov et al. 2023), enabling language models to be trained against several negative preferences rather than just one. Instead of using the Bradley-Terry formulation for single pairwise comparisons, this method adopts the Plackett- model (Plackett 1975 Luce et al. 1959) to score a target choice in relation to an entire set of inferior alternatives.
Given a prompt æ, a preferred response Up, and a set of N non-preferred responses Vn { 1. the Plackett-Luce probability that Up is ranked above all y is
1i=1
exp exp(r(x,M)) Mp)
p" (yp Vn (x)
(4)
exp(r(z,U)) Up)) + Σ exp(r(z,
where r(x, y) is the latent reward function. Substituting
x)) + ((x)
r(z, v) 3 log
Teef (y x))
and noting that (1r) cancels in the ratio gives
To(y x) neal (1h x))
(B x))
yd (Up 3n x) xi-1 exp A.)
A, log
log
Tref (Upp x))
Hence the multi-negative DPO training objective becomes
N Σ exp A.)]
CMN-DPO (0) = E... logo log
(5)
f=1
Notably, when N = 1, LMN-DP in equation 5 reducess exactly to the single-negative DPO loss.
4 FRAMEWORK
We propose MISP-DPO, a framework that address the limitations of single-negative supervision by introducing multi-negative learning through two core components: (1) diverse negative sampling strategy using sparse autoencoders to identify semantically meaningful deviations, and (2) general- ized TaaketttLLuce ranking objective that integrates multiple negatives promote robust alignment. An overview of the framework is shown in Figure
4.1 MULTI-NEGATIVE OBJECTIVES
Due the limitations of errors, we extend multimodal DPO to multi-negative preference optimization setting. Let denote the VLM policy to be optimized. Each training instance consists of multimodal prompt preferred image mp paired with an aligned textual response μp. and set of N negative images S, (m; [m, I from open- domain sources. Following the Plackett-Luce formulation from Eq. equation we adapt equation to visual preferences.
supervision and the inherently multi-faceted nature of visual
Blog Tolyn Ir m²,) Tref (yp Ix, m(,)
T 2. mp)
Cimg(:: Sp) log (-lg log
Σ exp
B log
Tref Up xr, m.p)
SE
This extends Eq. equation 3 to multiple negatives through the softmax aggregation and encourages the model to assign higher preference scores to the correct image mp compared to all negative images in S,, thereby promoting more robust visual grounding.
4

---

Published as a conference paper at ICLR 2026
Lemma 4.1 (Gradient Decomposition) Defining the preference advantage of each negative image and the preference distribution as
TTe (yp x, mmam) a; = =110
log (bp mp)))) Trof (Upp I. mp))
exp(a;)
po po(m)) 2, m, 1/p)
Trat (ro / L. m
2==1 Σ exp(ay)
Then the gradient of equation 6 decomposes as,
N
VoLing (0C S.)) = 3o (1gg Σ exp(a:) Σ (m m. Up Ao mp 1x,9p,,
(7)
where As (m mp v, (p) = V log (Up az, m - V log Ne (Up 2, mp).
This result shows that the gradient is weighted combination of correction signals across the image space, offering interpretability in terms of how the model adjusts its predictions in response to each visual discrepancy.
Although Eq. (6) and its gradient give an unbiased update, they require drawing a large set of nega- tives from the true pa (m,, x, pp: Mp) and computing S((m) }). realistic image domains, neither step is tractable. To alleviate this, we introduce a learnable distribution (m x, my: Mp) to sam- ple a small candidate pool Sn, Rewriting the gradient illustrated in Lemma 4.1 as an expectation under d gives a importance-samplinn estimator,
Vo VoLing (0; S..) 33 (log Σ exp(as)
Σ iES,
(o(m) xr, mps Up) Am(m) mp Up)-
exp(u:)
(8)
ieS,
To ensure that this approximation remains theoretically sound, we prove in Appendix A.3 that the resulting importance-samplinn gradient estimator is strictly unbiased and admits finite mame- variance bound. In particular, its mean-square error scales as O(1/K), with the constant governed by the maximum importance weight. These guarantees justify the of small candidate pool while preserving stable optimization.
We further promote cross modal alignment by incorporating textual preference supervision. Follow- ing recent multimodal DPO approaches (Xie et al. [2024] Wang et al.) 2024a Ouali et al. (2024) we replace traditional text-only preferences with taaage-grounder negative responses yn conditioned the same prompt and image mp. The corresponding DPO loss is,
Liext (0; S,.) = log (Blog To( I, mp)
log no ( Trof x. m.
I, (2,1pp)
Teee x., mp)
Our final loss combines both visual and textual preference signals,
C(;; S..) = Limg (0; + X (0;
(10)
where balances the contributions of image-based and text-based supervision. This unified for- mulation supports joint alignment across modalities, improving robustness and alignment quality in VLMs.
4.2 IMPORTANCE SAMPLING VIA SPARSE AUTOENCODER
To address the limitations of existing methods that rely simplistic, one-dimensional negatives, employ SAEs to disentangle and surface semantically meaningful variations the visual space. By providing structured and interpretable latent representation, SAEs enable principled importance sampling over diverse negative sxamples-prioritiiingg those that capture distinct failure modes and most informative for effective preference learning.
Embedding and Difference Vectors. Let T = {(pp, x}} be the training set of positive im- age age-prompt pairs. We use CLIP's image and text encoders, and fi to obtain d-dimensional embeddings he = fo (mp) and h = fx),, then fuse them via outer product and vectorization
vec (hv h E gdd For each negative candidate m form the difference vector
d; e(mp. x) e(m,, 2))
5

---

Published as a conference paper at ICLR 2026
Algorithm Greedy Diversity-Promooie Selection
Input: Difference vectors {di {di scores {si}, encoder &, selection size
2: Sp +
3: while [S,] < K do
4: i* - arg igs. (si + 3 min (os(((()))
5: Sn S, {
6: end while
7: Output: selected negatives set Sn,
Sparse Autoencoder Training. We train an SAE with encoder & and decoder D to decompose d: into sparse latent factors. The loss combines reconstruction fidelity and activation sparsity,
N Σ1 (mp.))ET 1=1
LSAE = ITN
Σ
-(E
Σ (T(()))))
(11)
wherepy is the average activation of hidden unit j. E (0, 1) is the target average activation, and balances reconstruction against sparsity.
Diverse Negative Selection. We score each candidate m., by,
||d; -((E()))
||E(4)] max, M/
(12)
max,
where ε and fy are the reconstruction error and activation magnitude across all candidates. To choose the final top- negatives Sn, we run a greedy selection that maximizes coverage of distinct error types while emphasizing hard negatives. We illustrate this algorithm in detail in Algorithm
The selected set Sn is then used in the importance-sampling gradient estimator of Eq. equation g
EXPERIMENT
5.1 EXPERIMENTAL SETTINGS
Models. We apply MISP-DPO to three widely-used multimodal LLMs: LLaVA-1.5-7B-HE Qwen2.5-VL-7B, and Qwen2.5-VL-3B These models are chosen due to their open availability, competitive performance, and diverse architectural designs (Chen et al. 2024b Zhang et al. 2024). LLaVA-1.5-7B. HF (Liu et al. 2023) integrates CLIP as the vision encoder with Vicuna-1.55 1.5- as the language backbone. Qwen2.5-VL-77 (Bai et al. 2025) uses a proprietary vision module and the strong Qwen2.5-7B language model. Qwen2.5-VI. VL-33B is lightweight 3B variant of the same architecture, providing a better balance between efficiency and capability.
Training data. We choose RLHF-V-Datad (Yu et al. 2024) as our training dataset. contains more than 5K samples, each with an image and pair of text responses indicating preference. RLHF- provides fine-grained, segment-level human feedback
diverse vision-language instructions, which has been shown to largely reduce hallucination while preserving informativeness. We treat the paired image the positive sample and select negative images per sample from COCO training dataset (Lin et al., (014)) using our importance sampling method, enabling effective training in our MISP-DPO framework.
Baselines. We compare MISP-DPO against five baselines: (1) the pretrained model without pref- erence tuning, (2) standard DPO (Rafailov et al., 2023), which uses only single text preference without any image-based supervision, (3) mDPO (Wang et al.) 2024a) and (4) CHiP (Fu et al.) 2025), both of which incorporate image preferences but rely on single negative image per com- parison, and (5) variant of our framework that uses multi multi-negative image preference optimization with negatives randomly sampled from the COCO dataset. All methods are trained under the same settings for fair comparison.

---

Published as a conference paper at ICLR 2026
Hallucination
Vision-Centric
Total
MMHHlBench
HallusionBench
POPE
WildVision
MMVP
avg impr.
Benchmarks
Score
HalRate (U)
(?)
(1)
qA (1)
(1)
Reward (1)
WinRate (1)
(*)
over BASE
Ilarn- 1.5-7b-hf
Base
2.78
51.04
47.73
17.63
12.30
84.37
-5.7
17.0
60.67
0%
DPO
3.29
37.50
55.62
22.83
22.63
83.02
-52.7
18.4
62.66
+21.13%
mDPO
2.99
49.81
47.32
20.52
13.19
83.25
62.1
14.6
58.33
+0,22%
CHiP
3.13
34.04
51.95
17.92
19.78
82.56
-68.4
12.2
52.33
+5.59%
Random
3.42
36.46
55.94
23.69
22.63
82.61
-51.3
18.3
60.33
+22.23%
MISP-DPO
3.51
32.29
57.52
25.43
24.83
83,94
-46.4
20.6
63,00
+30.09% %
Base
4.61
70.45
43.06
45.27
87.65
69.2
77.67
Qwen2.5. Qwen2.5.VV
DPO
4.92
11.46
69.92
42.48
43.51
.64
32.8
68.6
78.00
+3.85%
mDPO
5.01
14.89
67.40
41.33
42.20
87.02
28.5
66.2
76.33
-1.16%
CHiP
5.02
13.83
66.14
39.02
40.88
88.
28.5
66.4
77.00
-1.33%
Random
4.75
16.67
70.24
42.48
43.95
87 (0.0
30.7
67.8
78.33
-0.36%
MISP-DPO
5.05
11.46
71.24
43.77
45.61
88.66
32.4
68.4
79.00
+5.35 %
Base
4,20
22.34
64.67
37.57
36.70
87.48
-0.1
46.6
70.60
0SW
DPO
4,50
18:75
65.19
36.41
37.14
87.42
7.5
51.2
71.33
+1..12%
mDPO
4.47
21.28
62.88
35.84
37.14
87.65
50.8
69.33
+10.51%
CHiP
4.51
62.14
36.13
34.29
$7.30
51.0
70.33
+11.81%
Random
16.67
64.98
38.44
37.36
87.52
48.6
72.00
+8.57%
MISP-DPO
4.61
13.54
65.51
38.44
38.02
87.77
8.6
52.4
74.25
+19.89%
Table 1: Comparison of MISP-DPO against baseline methods across five vision Llnguaee bench- marks and three model backbones. The benchmarks cover hallucination detection and vision-centric reasoning. Average improvement over BASE is reported. 1: higher is better; L: lower is better.
Evaluation Benchmarks. We evaluate MISP-DPO and all baselines across five benchmarks span- ning hallucination detection and vision-centric reasoning. HAMHal-Benchee Sun et al. 2023) a mallucination-foousse VQA benchmark covering 8 question types and 12 object categories. Hal- AusionBench(Guaae et al. 2024) measures visual and factual hallucinations: we report all-answer accuracy (aA), figure-basse based accuracy (fA), and question-type accuracy (qA). POPE(Li et al.) (023b) evaluates object hallucination in VLMs via Yes/No probing under random, popular, and adversarial object settings. WildVVision[La et al. 2024) evaluates real-world user preference alignment with 500 curated human-model interaction samples; we report reward score and win rate. MMVP [Tong et al., 2024) assesses gre-ggrined visual reasoning using CLIP-blind image pairs, with accuracy reported over 135 zero-shot questions across 9 pattern types. Except for OMMHal-Bench all evaluations are conducted using Duaaa al., 2024). an open-source evaluation toolkit for vision- language models. For MM BBechh, we use GPT-4.1-mini Achiam al. 2023) the evaluator and report overall response quality and hallucination rate.
Implementation Details. For training the Sparse Autoencoder, we set the latent dimension to 128 and the sparsity weight 7 to 1, balancing reconstruction fidelity with latent sparsity. Following select three negative
prior work multi-negative preference optimization (Chen et al. 2024a), images per instance. For multimodal DPO training, we set the supervision balance parameter to 1 to equally weight image-basee based and text- bext-based preferences. All models fine-tuned using NVIDIA A100 GPUs, with a per-device batch size 2, gradient accumulation steps of 8 (yielding an effective batch size of 32), and learning rate of 10-5, The preference optimization coefficient is set to 0.5. Following prior work on mDPO, we adopt LoRA for parameter-efficient tuning, with rank of 64 and scaling factor a 128. For baseline methods, we strictly follow their original settings: mDPO is trained for 3 epochs with the same learning rate (lc-5), 8 = 0.1; CHiP is trained for 3 epochs with batch size of 32, learning rate of 5c-7, 8 = 0.5, and full- parameereetter finetuning.
5.2 OVERALL PERFORMANCE IMPROVEMENT
Table I shows the performance of MISP-DPO and baselines across five representative benchmarks. grouped into two categories: hallucination detection (MMHalBench,, HallusionBench, POPE) and vision-centric reasoning (WildVisson, MMVP). Our proposed MISP-DPO consistently achieves su- perior results over all evaluation domains and model backbones.
The largest gains appear on hallucination benchmarks. MISP-DPO substantially reduces hallucina- tion rates on MMHalBench (e.g., 32.29%, 11.46% and 13.54% across different backhones) while
7

---

Published as a conference paper at ICLR 2026
100
Randem Handin
reffrmmnce (14
94-
92-
Hidbesionlent (sA)
2 1
FONE
NRVP (ACC.)
Nettse (Score)
-10
30
Number of heguitee Samples
Figure 2: t-SNE visualizations and benchmark results for negative sampling, Left: importance- sampled negatives selected using SAE scoring exhibit broad semantic dispersion across three selec- tions. Middle:
randomly sampled negatives form tight, low- cow-diversity clusters. Right: performance across benchmarks with different numbers of negatives selected by MISP-DPO.
image
negative images
Instruction: Detai the image's components with particular focus on each ently.
Response: foreground this fruit-lpppd strawberries, bluvbberris, might used colebrate Besides, another dessert can seen background. Furthemmre, there cups nearby The fruit cake the main focus.
chosen image
retrieved negative images
Instruction: What being used mouse pad?
Chesen Response: the Image, photo being used as mouse pad. makeshif mouse present on the desk are other itmm such as cell phone, a notebook, and cup, indicating that the workspace actively being used various tasks.
control of laptop.
Figure 3: Negative image retrieval using MISP-DPO method. Each shows chosen image and three negatives; highlighted phrases in red, blue and green mark mismatches with negatives.
also achieving the highest accuracy across all HallusionBench metries. POPE further confirms these advantages. These improvements stem from the combination of diverse negative sampling, which exposes the model varied error types such object mismatches and attribute distortions, and importance sampling, which prioritizes hard negatives with high reconstruction errors from SAE, leading to stronger visual grounding. On vision-centric reasoning tasks, MISP-DPO also provides consistent gains. For example, it achieves the best reward (+8.6) and win rate (52.4) on WildVi- sion for Qwen2.5-VLL-3., while also outperforming baselines on MMVP across different model sizes. These results suggest that our method not only suppresses hallucinations but also enhances the model's ability to generate fine-grained, visually aligned responses.
We also conduct experiments under different 3 values on MMHalBench to balance reward learning and regularization. Figure reveals performance peaks at 3 ranging from 0.45 to 0.75, with degra- dation at extremes 3 0.T/I.0. We choose 3 0.5 for optimal trade-off between hallucination control and response quality, as it maximizes accuracy while minimizing hallucination rates across all backbones.
5.3 EFFECTIVENESS OF IMPORTANCE SAMPLING
We analyze the impact of our importance sampling strategy using t-SNE visualizations of high- quality negative images, shown Figure The left plot displays negatives selected by our SAE- guided strategy, while the middle shows randomly sampled ones. Importance-sampled negatives
8

---

Published as a conference paper at ICLR 2026
Model
Benchmark
MMHlalBench
Helc
POPE
WildVision
MMVP
Score (1)
HalRate (L)
aA (1)
(1)
qA (f)
Acc. (f)
Reward (1)
WinRate (1)
Acc. (1)
mDPO
5.01
14.89
67.40
41.33
42.20
87.02
28.5
66.2
76.33
diffusion
5.12
12.50
69.50
43.64
43,95
87.52
30.7
66.4
78.00
Qwen2.5-7B
crop+diffusin
13.54
70.35
43.35
44.61
87.35
30.7
78.33
similarity
12.50
69.82
42.77
87.24
30.1
MISP-DPO
5.05
11,46
71.24
43.77
45.61
88.66
32.4
68.4
79.00
mDPO
4.47
21.28
62.88
35.84
37.14
87.65
7.2
50.8
69.33
diffusion
14.58
65.19
39.31
37.14
87.35
52.4
71.33
Qwen2.5-3B
crop+diffusion
17.70
64.03
71.33
similarity
4,20
19.79
65.08
39.01
36.92
87.62
4.7
49.4
70.30
MISP-DPO 4.61
13.54
65.51
38.44
38.02
87.77
8.6
52.4
72.00
Table 2:
Comparison of different negative sampling strategies across five benchmarks. All variants share the same model and loss design, differing only in how negative images are constructed.
are well-dispersed across the embedding space, indicating high semantic diversity, whereas random negatives cluster tightly, indicating redundancy and limited coverage. This distribution reveals that our method captures heterogeneous error modes rather than collapsing onto narrow, repetitive subset of examples, thereby providing more informative supervision.
As illustrated in Figure deviations: one introduces different types of cakes with distinct decorations, while others include unrelated food items such sandwiches, scenes where the cake is present but not the main visual focus, thereby weakening semantic alignment. Similarly, the workspace scene, negative examples capture meaningful variations such as differences in desk arrangement and surrounding objects-incllddng notebooks and cups-each impacting multimodal alignment differently. These examples highlight how our sampling method uncovers diverse error modes, encouraging the model to learn more robust visual distinctions, strengthening training signals, and improving generalization beyond narrow, dimensniiio deviations.
selected negatives for the fruit cake image exhibit clear semantic
5.4 COMPARISON OF NEGATIVE SAMPLING STRATEGIES
We evaluate five negative sampling strategies that share the same model architecture and loss for- mulation but differ how negative images are constructed: (1) mDPO, which relies on a single crop-generated negative; (2) diffusion, which combines one diffusion negative with two negatives selected by our method; (3) crop+diffusion, which mixes one cropped, one diffusion, and one MISP- DPO Pel-selected negative; (4) similarity, where three negatives retrieved based similarity to the positive image; and (5) our model. As shown Table 2
mDPO yields the weakest performance, highlighting the limitations of single-negative supervision. Among multi multi-negative variants, the dif- fusion strategy outperforms crop+diffusion, suggesting that a higher proportion semantically di- verse negatives from method improves supervision quality. The similarity variant performs worse than diffusion, underscoring that naive retrieval of visually similar negatives does not pro- vide the challenging guidance needed. In contrast, MISP-DPO consistently achieves the best scores across all benchmarks, validating the effectiveness of structured multi-negative selection via SAE- guided
ineodddmooootn importance sampling. Additional results are reported in Table in the appendix.
Beyond the choice of sampling strategy, we further explore the number of negatives required. Fig- ure 2
(right) shows performance across HallusionBench. MMVP, and MMHalBench with varying numbers of negatives. One or two negatives are insufficient to provide high-quality supervision. Performance peaks three negatives, while increasing beyond this offers no further benefit. For HallusionBench, adding five negatives even reduces performance. likely due to noise introduced by redundant or low-quality samples. Together, these results demonstrate that three carefully chosen negatives strike the best balance between informativeness and robustness.
5.5 HALLUCINATION REDUCTION WITH MISP-DPO.
Figure in the Appendix presents qualitative comparisons highlighting the impact of multi negaiie supervision. Baseline methods such as DPO and CHiP frequently introduce hallucinated details

---

Published as a conference paper at ICLR 2026
Method
HallusionBench
POPE
MMVP
Method
LLaVA
Qwen-7B
Qwen-3B
aA(f)
(A(1)
gA(t)
Acc.(f)
Acc.(t)
Base
64.05
77.13
73.99
w/o PL loss
64.34
36.64
37.05
87.37
72.33
DPO
65.07
77.24
74.02
w/o SAE neg
64.98
38.44
37.36
87.52
72.00
mDPO
65.35
75.53
73.20
Isolate Text
63.45
34.08
35.27
85.43
71.66
CHiP
60.98
76.78
74.10
Isolate Image
61.59
32.11
35.44
85.01
71.30
Random
65.91
77.16
74.25
MISP-DPO
65.51
38.44
38.02
87.77
74.25
MISP-DPO
66.26
77.18
74.32
(a) Component Ablation
SEED- Generalization
Figure 4: Left: Component ablation Qwen2.5-VL-3B. Right: SEED-Bench evaluation across
three backbones. MISP-DPO improves both hallucination-orientedd and general multimodal reason- ing benchmarks.
(e.g., incorrect objects, colors, spatial relations), while MISP-DPO generates more faithful and grounded descriptions. For instance, in the first example, only MISP-DPO correctly identifies that the image lacks sand and accurately describes the bird. These results illustrate that incorporating diverse negatives enables the model to better distinguish relevant from spurious cues, improving factual accuracy in vision-language alignment.
5.6 CONTRIBUTION EACH COMPONENT IN MISP-DPO
We perform controlled ablations on Swen2.5-VL-33 to isolate the contribution of each core com- ponent Figure + (a). Replacing the Plackett-Luce multi-negative objective with simple aver- aged pairwise loss consistently degrades performance, indicating that jointly comparing positive against multiple negatives provides stronger and more structured supervision than independent pair- wise comparisons. Substituting SAE-guided negatives with randomly sampled images also reduces accuracy, confirming that semantically informative and diverse negatives are crucial for effective preference learning. Finally, training with image-only or text-only loss results in the largest per- formance drops, highlighting the necessity of complementary multimodal signals for robust visual grounding. Overall, MISP-DPO achieves the best results, indicating that its gains are derived from the synergy between structured multi-negatee construction and principled ranking optimization.
5.7 GENERALIZATION BEYOND HALLUCINATION BENCHMARKS
To assess whether the improvements of MISP-DPO extend beyond callucination-foousse tasks, we further evaluate cring reasoning. visual understanding, and instruction following. As shown Figure (b), MISP- DPO consistently matches or outperforms all baselines across three backbones. Notably, the gains over random multi-negative sampling indicate that improvements stem from semantically structured negative construction and principled multi-negative preference optimization instead of increased supervision alone. These results demonstrate that our method improves the overall multimodal rea- soning capability rather than overfitting hallucination-sseccfic metrics.
(Li et al. 2023a), general-purpose multimodal benchmark cov-
CONCLUSION
We present MISP-DPO, a novel framework that introduces multi-negative, semantically diverse pervision into multimodal Direct Preference Optimization. By leveraging CLIP-based embeddings and sparse autoencoder, our method efficiently selects image-side negatives that vary across multi- ple semantic facets and reflect diverse failure modes. These negatives are integrated into Plackett- Luce-style ranking objective with importance sampling, enabling the model learn from richer and more structured supervision. The method remains efficient and scalable for real-world multimodal applications. Extensive experiments across five benchmarks demonstrate that MISP-DPO consis- tently outperforms strong baselines in hallucination reduction and visual grounding. While our evaluations rely on GPT-based scoring-whiie may introduce bias or inconsistency when assessing fine-graine alignment-ouu findings validate the effectiveness of semantic- multi negative sampling for robust multimodal alignment and open up promising directions for scalable and inter- pretable preference-based learning.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 20X1v22303.08774, 2023.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402.105777 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Lior Baruch, Moshe Butman, Kfir Bar, and Doron Friedman. Preference tree optimization: Enhanc- ing goal-oriented dialogue with look-ahead simulations. Scaling Self-Improving Foundation Models without Human Supervision.
Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. arXiv preprint arXiv:2406.09215, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 1rXiv:2412.05271, 2024b.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Y Zou, Kai-Wei Chang. and Wei Wang. Enhancing large vision language models with self-training on image comprehen- sion. Advances in Neural Information Processing Systems, 111:33369-13199, 2024.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang. Jiaqi Wang, al. Vlmevalkit: An open-source toolkit for evaluat- ing large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pp. 11198-11201, 2024.
Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, and See-Kiong Ng. Chip: Cross-modal hierarchical direct preference optimization for multimodal llms. arXiv preprint 250i::2000.666299 2025.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Cheng-Yo Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural informa- fion processing systems, 36:31096-31 31116,, 2023.
Songtao Jiang, Yan Zhang, Ruizhe Chen, Yeying Jin, and Zuozhu Liu. Modality-fair preference optimization for trustworthy milm alignment. arXiv preprint arXiv:2410.15334, 2024.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. Text encoders bottleneck compositionality in contrastive lasion-language models. arXiv preprint 20Xx:23055.1887, 2023.
Amita Kamath, Cheng- Yu Hsieh, Kai-Wei Chang, and Ranjay Krishna. The hard positive truth about vision-language compositionality In European Conference on Computer Vision, pp. 37- 54. Springer, 2024.
Bohao Li, Rui Wang, Guangzhi Wang. Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension. arXiv preprint arXiv:2007.1613, 2023a.

---

Published as a conference paper at ICLR 2026
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint 3xXi:23305.1035 2023b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland. September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Chaohu Liu, Tianyi Gui, Yu Liu, and Linli Xu. Adpo: Enhancing the adversarial robustness of large vision-language models with preference optimization. arXiv preprint arXiv:2504.01735, 2025a.
Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Ruibo Chen, Yihan Wu, Junfeng Guo, Tianyi Zhou, and Heng Huang. Modality-balanein preference optimization of large multimodal models by adversarial negative mining. arXiv preprint arXiv:2506. 08022, 2025b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Shujun Liu, Siyuan Wang, Zejun Li, Jianxiang Wang, Cheng Zeng, and Zhongyu Wei. Ovip: On- line vision-language preference learning for vlm hallucination. arXiv preprint arXiv:2505.15960 2025c.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Moham- mad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-to-rant to-kkkk..
Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 2404-2420, 2025d.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wild- vision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv:2406. I 1069, 2024.
R Duncan Luce et al. Individual choice behavior, volume 4. Wiley New York, 1959.
Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Clip-dpo: Vision- language models as source of preference for fixing hallucinations in Ivlms. In European Con- ference on Computer Vision, pp. 395-413. Springer, 2024.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C:
Applied Statistics, 14(2):193-202. 1975.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-523 53741, 2023.
Ugur Sahin, Hang Li, Qadeer Khan, Daniel Cremers, and Volker Tresp. Enhancing multimodal compositional reasoning of visual language models with generative negative mining. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5563-5573, 2024.
Wentao Shi, Menggi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng. Direct multi-tumn preference optimization for language agents. arXiv preprint arXiv:2406. 14868, 2024.
Zhiqing Sun, Sheng Shen. Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
12

---

Published as a conference paper at ICLR 2026
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406. 11839, 2024a.
Xin Wang. Hong Chen, Si'ao Tang. Zihao Wu, and Wenwu Zhu. Disentangled representation learn- ing. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024b.
Shengguang Wu, Fan- Yun Sun, Kaiyue Wen, and Nick Haber. Symmetrical visual contrastive op- timization: Aligning vision-language models with minimal contrastive images. arXiv preprint 12Xi::2502.13928,, 2025.
Yuxi Xie, Guanzhen Li, Xiao Xu, and Min- Yen Kan. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. arXiv preprint arXiv:2411.02712. 2024.
Shuo Xing, Yuping Wang. Peiran Li, Ruizheng Bai, Yueqi Wang, Chengxuan Qian, Huaxiu Yao, and Zhengzhong Tu. Re-align: Aligning vision language models via vetrieval-augmentte direct preference optimization. arXiv preprint arXiv:2502. 1314500..114.... 2025.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingho Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3090-3100, 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024.
Fatemeh Pesaran Zadeh, Yoojin Oh, and Gunhee Kim. Lpoi: Listwise preference optimization for vision language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 26830-26844, 2025.
Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jic, Zhenhua Chai, and Liang Wang. Investigating compositional challenges in vision-language models for visual grounding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14141-14151, 2024.
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm 1lms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.1360) 2024.
Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning im- proves compositionality large vision-language models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 13785-13799 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EFFECT OF ON HALLUCINATION AND QUALITY
Figure illustrates the impact of different 3 values both response quality (left) and hallucination rate (right) on MMHalBench. We observe that the model performs best when / lies in the range of 0.45 to 0.75, achieving good balance between response quality and hallucination suppression. To ensure both accuracy and stability, we set B = 0.5 as the default value in all main experiments.
A.2 EXAMPLES OF HALLUCINATION REDUCTION WITH MISP-DPO
To illustrate the impact of our approach, Figure presents qualitative comparisons across three representative prompts, highlighting the improvements brought by MISP-DPO over baselines in- cluding LLaVA, DPO, and CHiP. MISP-DPO demonstrates stronger ability avoid hallucinations and produce factually accurate descriptions grounded in visual evidence. In the first example, it cor- rectly identifies the absence of sand and avoids misidentifying the bird species. In the second case, faithfully describes the structure and positioning of the gloves despite occlusion. In the final exam- ple, it provides a precise spatial interpretation of the two watches without fabricating brand- spec-sp details. These results suggest that multi- nulti-negative supervision strategy improves the model's sen- sitivity to fine-grained semantic cues and its ability to reject spurious correlations and hallucinated attributes, leading more factually consistent vision-language generation.
53
40
LLaVAL.5-21
5.18
Owen2.5573
3.0
Score (1)
4.5
23
HalRate
20
4.0
3.51
3.5
LLAVAL.5-78
Owen2. 5-73
3.0
0.2
0.3
0.6
0.7
0.9
1.0
0.1
0.2
0.4
0.5
0.6
0.7
0.8
1.0
Figure 5: Performance comparison of Score and Hallucination Rate across different 3 values for on MMHalBench.
A.3 UNBIASEDNESS FINITE-SAMPLE VARIANCE BOUNDS FOR IMPORTANCE SAMPLING GRADIENT
For convenience, we define the inner gradient term in Lemma
9(0) =
Pa (m, x, mpp Up) A (m mp xr, Up)
(13)
Our importance-sampling scheme (Sec. 4.1 introduces proposal distribution 4s (m, x, mp Up)
induced by the CLIP+SAE pipeline, and samples a small candidate pool S, fm Ik= k=1 from Jo- We then approximate the exact gradient using an importance-weighted estimator.
A.3.1 UNBIASEDNESS OF THE GRADIENT
Let the importance weight
po(Min Ir, mp. 4p)
wmm..)
(m xr, mp Up)
14

---

Published as a conference paper at ICLR 2026
Lemma A.1 (Unbiased IS gradient) Assume that 9o(mn I, mp, Up) > O. Define the Monte Carlo estimator of g(0) as.
0 whenever (m x, pp.p Up)
K
(+)
Σ (m(mm) K
1
Do (m mp x. vp),
m go(- 2. mps Up).
Then Gb. (0) is an unbiased estimator of g(0),
Em brog [+)( g(0).
(14)
Proof A.1 Using linearity of expectation and the definition of importance weights,
E [mm [0){ = En [w(m,,) A0 mp
2. Up)]
po mpp Up)
Σ (m 2, mp, Up)
Do (mi, mp a. Up)
(m',
x, m. Fp)
N
po(m" j. mp, Up) Ao(m" mp x, Up)
(
which proves the claim.
Thus, the IS estimator recovers the exact gradient term in expectation, and hence the overall gradient Eq. 7
V o (o: (0: o log Σ exp(a,) g()),
9(0),
(15)
admits an unbiased importance-sampling estimator by replacing g(0) with Tx (0).
A.3.2 VARIANCE BOUND UNDER FINITE NEGATIVE SAMPLING
We next calculate the variance of gv. (0) when only finite number K of negatives sampled.
Define the maximum importance weight,
po(m I, pp Up)
|| = max m. ES.. go (m, x. mp: Up)
Assume that the gradient differences are bounded, which in practice is achieved by gradient clipping [4-6]. There exists do such that,
Ao(mn mp x. Np)ll22 &
Vm, E Sn.
(16)
Lemma A.2 (Variance bound) Under the assumptions above, the mean-square error of the IS gra- dient estimator isbounded as,
L
T (0) 9(0)
S
||
K
Consequently, the expected absolute deviation satisfies,
E (1) -(0))l2) S L
(17)
K
Proof A.2 Because the K samples are i..d..
(ar( K
Var(w(ma Ao (m mp x, 4p))
15

---

Published as a conference paper at ICLR 2026
Using the bound [m mp
1, 8p) 1|
L, obtain
B(lØx c00)l
Emanago (mis my 1p)
Em, mo [.]
K
Moreover,
Bmanugs [ur(m))²] S ||w|loo Byngg [w(mm)]
= =|lw|loo Σ g (m)) po(mm) 1/0/ Ero(mm) 1100 go(mm)
(18)
Combining these yields the desired variance bound. Applying Cauchy- Sauchy-Schwarz gives the final vari- ance bound.
In our setting, the SAB-guided sampler is explicitly designed to approximate the target distribution with diverse but not overly peaky weights, which empirically keeps ||w]|o< small and leads to stable optimization.
A.4 NEGATIVE SAMPLING STRATEGIES ON LLAVA-1.5-77 MODEL
TableB]shows that -LLVA--15577 follows the same trend as Qwen2.5. The single-negative baseline (mDPO) performs the weakest, while adding more negatives through diffusion crop+diffusion gives only moderate improvements.
In contrast, MISP-DPO consistently achieves the best results, notably lowering hallucination rate and improving accuracy across HallusionBench. POPE, WildVision, and MMVP. This confirms that our SAE-guided multi-negatio sampling generalizes across model families, providing stronger guidance even for smaller-scale ussion-langual models.
Model
Benchmark
MMHHalennch
HallusionBench
POPE
WildVision
MMVP
Score (*)
HalRate (3)
aA
ÍA
(1)
Acc. (t)
Reward
WinRate (1)
Acc. (1)
mDPO
2.99
49,81
47.32
20.52
13.19
83.25
-62.1
58.33
1lava-1.5-bb
diffission
3.49
33.33
52.57
21.38
18.02
83.25
-53.0
18.8
61.33
crop+diffusion
3.44
35,42
53.63
22.54
19.78
83,80
-50.5
20.0
61.00
MISP-DPO
3.51
32.29
57.52
25.43
24.83
83.94
-6.44
20.6
63.00
Table 3: llava performance
A.5 SELECTION COST AND SCALABILITY
Our SAE-based negative-selection framework is designed be computationally efficient and lightweight. Using 5.7k RLHF-V dataset together with 118k COCO images on single A100 GPU, the computation CLIP embeddings takes 4 minutes, SAE training 9 minutes, and the nega- tive selection stage 20 minutes, resulting in total runtime of 33 minutes. This corresponds to only 0.37 seconds per positive sample select three semantically informative negatives, demonstrating the practicality and efficiency of our importance-sampling strategy.
During preference training, incorporating three negatives introduces modest and predictable over- head: MISP-DPO runs at approximately 3.6s per sample, compared to 2.6s for DPO and 3.1s for mDPO, with CHiP exhibiting similar runtime. The overall pipeline scales linearly with dataset size and remains stable under the -lackett-Luce objective and importance-sampling estimator.
16

---

positioned way Published as a conference paper at ICLR 2026
LLaVA
A standing sand looking painting bird.
DPO
A vicuna standing the sand is not looking at anything. In image, black and white bird with yellow and red beak standing in front of green leafy plant. The bird is not looking at the plant, but rather at the
CHiP
vicuna standing in sand is looking a painting bird.
MISP-DPO
In the image, there is no sand or any other background visible. The painting black yellow beak, not vicuna. The bird is surrounded by green and black brush. and the painting described as painting bird not standing the sand, and the image does not provide any context or information about the bird looking at something.
(Ours)
features colorfu
with large beak.
bird
standing in the sand looking at?
bird with yellow beak. The
LLaVA
hands wearing the gloves are large.
gloves are black color.
DPO
The hands wearing the gloves are black and have a logo on the palm. The gloves made black material, that they are facing upwards.
CHiP
In the image, the hands wearing the gloves are black color.
MISP DPO (Ours)
image shows two
glaves,
lago the back. that they appear to be two hands. The
Describe the hands wearing the gloves.
They are positioned in such gloves are black and have close-up view. making it difficult discem any other details. The gloves are not attached any hands, but they are positioned in way that they appcar to be hands.
LLavA
the image, there are two watches placed next to each other. One watch is positioned of the other, with the watch being G-Shock watch and the bottom watch being Garmin watch. The two watches are placed side by side, showcasing their differences design and features.
the watch with the band is positioned on top of the watch black band. The watch with the red band while the watch with the black band on the left side of the image as well.
the left side of the image,
CHiP
In the image, one watch is in the foreground, and the other is the back- ground. The foreground one is a close-up while the other blurry.
Describe how the two watches are placed.
MISP-DPO (Ours)
The two watches are placed side by side, with the Garmin watch on the left and the other watch the right. The Garmin watch displaying map, while the other watch has red band. The two watches are positioned such a way that the Garmin watch's screen is visible, and the other watch's red band also visible.
Figure 6: Comparison of responses from LLaVA model training through different methods: pre- trained, DPO. CHiP, and MPSP-DPO. Blue text indicates faithful descriptions; red text marks hallu- cinated or unsupported content.
17


Reviews:
[{"review_id": "mSHirDvqv9", "reviewer": "Reviewer_HwrY", "summary": "The paper proposes MISP-DPO, a listwise preference-optimization framework for multimodal LLMs that combines (i) a Plackett–Luce (PL) objective over a *winner + multiple negative* candidates, (ii) importance sampling with a learned proposal (q_\\phi) to reduce training cost and variance, and (iii) a semantically diverse negative-construction pipeline built in CLIP embedding space using sparse autoencoders (SAE) and feature-level perturbations. The method is applied to both *visual-preference* (image conditioned) and *textual-preference* (response conditioned) settings. Experiments report consistent gains over DPO/mDPO/CHiP-style baselines on datasets such as MMHal-Bench, HallusionBench.", "questions": "1. **Cost accounting:** Please add wall-clock/GPU-hour comparisons vs. DPO, mDPO, CHiP to demonstrate the promised efficiency gains of importance sampling.\n2. **Negative construction controls:** How do you ensure that CLIP-SAE-driven negatives are not trivially separable (e.g., distributional artifacts), and that they **stress visual grounding** rather than language priors? Any human spot-checks? \n3. **Unbiasedness & variance:** Is the IS gradient strictly unbiased under your training scheme? Do you apply **weight clipping** or **self-normalized IS**? Please report effective sample sizes or variance diagnostics across training.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **Clear problem framing & practical motivation.** The paper demonstrates clear problem formulation and presentation. The PL-based listwise loss directly optimizes rankings rather than isolated pairs. \n- **Good objective with scalable estimation.** Using **importance sampling** and a learned proposal (q_\\phi) to approximate the PL gradient is a sound strategy to keep many-negative training tractable while emphasizing informative (“hard”) negatives. \n- **Semantically diverse negatives grounded in CLIP space.** The SAE-based feature editing and “mix-and-match” construction pipeline plausibly increases negative diversity without requiring extra human labels. \n- **Broad evaluation.** The method is tested across multiple hallucination/factuality benchmarks frequently used for MLLMs (MMHal-Bench, HallusionBench, POPE, WildVision, MMVP) and with a modern evaluation toolkit (VLMEvalKit).", "weaknesses": "- **Novelty concern.** Beyond pairwise (1 chosen, 1 rejected) multimodal DPO, there already exists listwise optimization in the context of multimodal DPO, such as LPOI[1]. Therefore, there may be certain novelty concern, **especially when the authors claim** their paper is \n\n  - \"the **first** framework to incorporate *multiple*, semantically *diverse* negative images in multimodal DPO\" (line 16, in abstract)\n  - \"the **first** framework to incorporate multi-negative supervision into multimodal DPO\" (line 89), \n  - \"However, such techniques **remain underexplored** in vision-language models\" (line 117). \n\n  **So please double-check your claims in the submission.** Also, the use of Plackett–Luce objective is also not really a novelty for listwise DPO, as there are already some prior work such as PLPO[2].\n\n- **Insufficient experiments against former baselines.** As there are already many works published on multimodal DPO, it's not enough to just incorporate mDPO and CHiP as baselines apart from Random and basic DPO in experiments. Please at least incorporate and run experiments for latest methods OPA-DPO[3] and SymMPO[4]. Any more baselines are also welcome. \n\n- **Estimator properties insufficiently analyzed.** The text would benefit from formal statements or empirical diagnostics of *bias/variance* under finite negative sampling, any **weight clipping** or self-normalization, and the stability of (q_\\phi) training (e.g., divergence from target leading to high-variance importance weights). (I did not see explicit guarantees/ablation in the provided pages.) \n\n- **Ablation depth.** While the framework has several moving parts (PL listwise loss, IS with (q_\\phi), SAE-based negatives, textual-preference branch), the paper would benefit from *systematic ablations* that isolate each contribution and report uncertainty (std/CI over seeds).\n\nReferences:\n\n[1] Fatemeh Pesaran zadeh et al. \"LPOI: Listwise Preference Optimization for Vision Language Models\" In ACL 2025 Main Conference.\n\n[2] \"Plackett–Luce Preference Optimization (PLPO): Listwise Ranking for Preference Optimization\" Preprint 2024.\n\n[3] Yang et al. \"Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key\" In CVPR 2025.\n\n[4] Liu et al. \"Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization\" In NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aWd8XwXCxe", "reviewer": "Reviewer_hiMy", "summary": "This work introduces MISP-DPO, a novel framework to leverage a sparse auto encoder to identify diverse negative images for multi-negative preference optimization, building on multimodal DPO.\nThe authors demonstrate the efficacy of their method on multiple multimodal models across a range of benchmarks.", "questions": "* Considering that $d_i$ is dependent on $m_p$, $x$ and $m_n$, how scalable is the retrieval of negatives at training time, if one assumes potentially scaling up the distractor pool and the data used for alignment?\n* In the reproduction of mDPO, section 5.4 mentions “mDPO, which relies on a single diffusion-generated negative”. But mDPO constructs the negative image via random cropping (0-20%). Is this a typo?\n* It is not clear to me why selecting more than 3 negatives would be detrimental to performance as presented in figure 2 and briefly discussed in 5.4. The authors propose this may be “due to noise introduced by redundant or low-quality samples”, but then redundancy may be directly addressed through the diversity-promoting selection and COCO may not have substantial amounts of “low-quality samples”?\n* The reported numbers for MMHalBench for at least LLaVA 1.5 7B seem surprisingly strong, even for reported baselines? Earlier works such as [MDPO] has baseline LLaVA 1.5 7B at 2.19 (in this paper: 2.78) and with their method they achieve “only” 2.39, whereas the “mDPO” reproduction in this paper reports 2.99. Are the evaluation protocols comparable?\n\n\n[MDPO] Wang, Fei, et al. \"mdpo: Conditional preference optimization for multimodal large language models.\" arXiv preprint arXiv:2406.11839 (2024).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors report non-trivial improvements over respective baselines across benchmarks and models. They re-implement DPO, mDPO, and CHiP for a direct and fair comparison (matching data and base models).\n* The main contributions are the introduction of multi-negative preference optimization and the method proposed for selecting counterfactual images, building on CLIP retrieval, a sparse autoencoder, and a greedy algorithm to achieve diverse negatives.", "weaknesses": "* Section 4.2, describing one of the main contributions of the work, is perhaps a bit limited in detail. For example, the training (data, recipe) for the SAE is not described. And while the math presented in the negative selection may be sufficient, some discussion behind the intuition of sampling for reconstruction error and activation may make the paper more accessible, particularly to casual readers less familiar with using SAEs for interpretability.\n* Another recent work [S-VCO] also argues for negative images that are substantially similar to the request image under alignment. In this work, the authors acknowledge this work and argue that this method is expensive (as [S-VCO] relies on image generation method to generate counterfactuals) while the proposed method is more efficient. However, the respective efficacy is not further discussed. An ablation comparing the retrieval + SAE based approach directly to the generative approach proposed by [S-VCO] would further enhance the contributions of this paper. ([S-VCO]’s data (MVC) appears to have been made available.)\n* In table 2, the caption implies that the main difference is how negative samples are chosen, but another difference seems to be the number of negative examples being used as per the description in the text. Perhaps this could be clarified?\n* For the ablations in table 2, “diffusion” and “crop+diffusion” have two or one negative images still selected by the proposed method as described in the text. I believe this may make comparison a bit more difficult? I understand that multiple negatives based on diffusion or cropping may not have enough diversity, but if that is the concern, perhaps an ablation with 1 negative sample for all methods could be made fairly, further separating the improvements achieved from the targeted selection method from the multi-negative proposal?\n* Minor notes for table 2: typo “mdpo” (instead of “mDPO”); Missing average improvement as in table 1 for easier comparisons.\n\n\n[S-VCO] Wu, Shengguang, et al. \"Symmetrical visual contrastive optimization: Aligning vision-language models with minimal contrastive images.\" arXiv preprint arXiv:2502.13928 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vh8Pu2g5x7", "reviewer": "Reviewer_h9st", "summary": "The paper presents MISP-DPO, a framework for multimodal Direct Preference Optimization that uses multiple semantically diverse negative samples instead of a single one. It combines a Plackett–Luce ranking objective with importance sampling guided by a Sparse Autoencoder trained in CLIP space to select informative negatives. Experiments on several benchmarks show improved multimodal alignment and reduced hallucination compared to existing DPO methods.", "questions": "1. How does the proposed multi-negative sampling strategy affect training efficiency and scalability when applied to larger datasets?\n\n2. Could the authors provide a more detailed analysis of computational overhead introduced by the sparse autoencoder and importance sampling modules?\n\n3. Beyond hallucination reduction, has the method been evaluated on reasoning or instruction-following tasks to assess generalization across multimodal objectives?\n\nI will adjust my score based on the authors’ response.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper clearly identifies a weakness in current multimodal DPO frameworks—the oversimplified single-negative setup—and proposes a principled multi-negative formulation to address it.\n\nExperimental evaluation is extensive, including comparisons across multiple models and benchmarks, with consistent quantitative gains in hallucination reduction.", "weaknesses": "The novelty may be moderate: it mainly leverages existing models to extract multiple negative samples, without introducing substantial theoretical or methodological innovation.\n\nThe paper does not deeply analyze computational overhead or training stability when incorporating multiple negatives, which could affect scalability for larger datasets.\n\nIt remains unclear whether the improvements generalize beyond hallucination-oriented tasks (e.g., to reasoning or instruction following).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)