P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Authors: ruipeng zhang
Zhihao Li
Haozhang Yuan
C.L.Philip Chen
Tong Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs ; MLLMs ; Hallucination ; DPO
Primary area: foundation or frontier models, including LLMs

Abstract: Hallucination has recently garnered significant research attention in Large Vision-Language Models (LVLMs). Direct Preference Optimization (DPO) aims to learn directly from the corrected preferences provided by humans, thereby addressing the hallucination issue. Despite its success, this paradigm has yet to specifically target the perceptual bottleneck in attended regions or address insufficient Visual Robustness against image degradation. Furthermore, existing preference pairs are often vision-agnostic and their inherently off-policy nature limits their effectiveness in guiding model learning. To address these challenges, we propose Perceptual Processing Direct Preference Optimization (P$^2$-DPO), a novel training paradigm in which the model generates and learns from its own preference pairs, thereby directly addressing the identified visual bottlenecks while inherently avoiding the issues of vision-agnostic and off-policy data. It introduces: (1) an on-policy preference pairs construction method targeting Focus-and-Enhance perception and Visual Robustness, and (2) a well-designed Calibration Loss to precisely align visual signals with the causal generation of text. Experimental results demonstrate that with a comparable amount of training data and cost, P$^2$-DPO outperforms strong baselines that rely on costly human feedback on benchmarks. Furthermore, evaluations on Attention Region Fidelity (ARF) and image degradation scenarios validate the effectiveness of P$^2$-DPO in addressing perceptual bottleneck in attended regions and improving Visual Robustness against degraded inputs.

Published as a conference paper at ICLR 2026
Pº-DPO: GROUNDING HALLUCINATION IN PERCEP- TUAL PROCESSING VIA CALIBRATION DIRECT PREFERENCE OPTIMIZATION
Ruipeng Zhang Zhihao Li C. L., Philip Chen 1.233 Tong Zhang 1.23+
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science Engineering, South China University of Technology
Pazhou Lab, Guangzhou, China
3 Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China (ruipengzhang, zrp, lizhihaohen @mmai 1 com [philipchen,
edu.cr en
ABSTRACT
Hallucination has recently garnered significant research attention in Large Vision- Language Models (LVLMs). Direct Preference Optimization (DPO) aims to learn directly from the corrected preferences provided by humans, thereby addressing the hallucination issue. Despite its success, this paradigm has yet to specifi- cally target the perceptual bottleneck in attended regions address insufficient Visual Robustness against image degradation. Furthermore, existing preference pairs often vision-agnostic and their inherently off-policy nature limits their effectiveness in guiding model learning. To address these challenges, we propose Perceptual Processing Direct Preference Optimization (P²-DPO), novel train- ing paradigm in which the model generates and learns from its own preference pairs, thereby directly addressing the identified visual bottlenecks while inher- ently avoiding the issues of dision-agnostic and off-policy data. It introduces: (I) an on-policy preference pairs construction method targeting Focus-and- and-Eneance perception and Visual Robustness, and (2) well-designed Calibration Loss to pre- cisely align visual signals with the causal generation text. Experimental results demonstrate that with comparable amount of training data and cost, P2-DPO outperforms strong baselines that rely on costly human feedback on benchmarks. Furthermore, evaluations Attention Region Fidelity (ARF) and image degra- dation scenarios validate the effectiveness of P²-DPÓ in addressing perceptual bottleneck in attended regions and improving Visual Robustness against degraded inputs]
INTRODUCTION
The rapid evolution of Large Language Models (LLMs) (Zhao al.) 2023a Achiam et al.) 2023 Touvron et al. 2023 Guo et al. 2025) has catalyzed the development of Large Vision-Language Models (LVLMs) (Li al.) 2023a] Zhu et al., 2023 Liu et al., (024b). which have demonstrated impressive capabilities in cross-modal understanding. Despite these advances, hallucination re- mains a critical challenge: the generated text lacks faithful grounding visual input (Liu et al., 2024a Guan et al. 2024). Existing research (Xie et al. [2024] Leng et al.) (224)) attributes this to a fundamental architectural imbalance where powerful linguistic priors often suppress visual sig- nals, resulting in hallucination. We can broadly categorize the visual sources of hallucination into two distinct classes: failures in Perception and Perceptual Processing. The former represents a failure that pushes the model to its upper limits, typically stemming from insufficient knowledge
Corresponding author.
'The code for model training and testing is available at: anonymous. w0p.n....n......... .cc enceec//r
1PP0-CF7A

---

Published as a conference paper at ICLR 2026
gained during pre-training or the inherent limitations of the visual encoder. The latter, in con- trast, is considered a failure where the model possesses the latent potential to answer correctly without external knowledge, it has already captured the key visual evidence but still fails in the subsequent processing stage, leading to hallucination (Yang et al.) 2023). While Perception has received significant attention, we argue that Perceptual Processing remains an equally critical but underexplored bottleneck for mitigating hallucinations. Specifically, we identify two core mani- festations of this Perceptual Processing failure. The first is Perceptual Bottleneck in Attended Regions, where the model accurately localizes its attention but fails to get the right answer (Fig- ure I Left). The second is Lack of Robustness, which reveals the model's high sensitivity to image quality. As shown in Figure [ (Right), even minor noise or image degradation can cause the model to generate a hallucinatory output. A crucial insight from these failure modes is that the model is often on the cusp of correct answer, failing not in fundamental failure in Per- ception, but in its final processing step, which makes it an ideal candidate for self-correction.
The predominant approach to mit- igating hallucinations in LVLMs is preference optimization (Rafailov jet al.) 2023: Yu al.] (024b), which reduces model hallucination using human or synthetic preferences. Ex- isting data sources follow two main approaches: (1) post-hoc textual re- visions of model outputs guided by human or AI feedback (Schulman
Deggn ug
Thu joke of Worl vars 1ack
vadispatins
et al.] 2017] Yu et al.) [2024b] Lee
Har
jet al.); ) and (2) synthetic injection of
Orgie
hallucinations to generate contrastive
Emsige
pairs (Zhou et al., [2024). Despite their demonstrated efficacy (Ouyang eet al. 2022 Rafailov et (002)). we contend that these methods, which
Left: Pereoptual Bottleneck. Attention heatmaps visualize that the model correctly focuses on the key entily. yet still hallucinates. Right: Lack of Robustness. The model hallucinates when presented with an image containing noise impercoptible humans. shown answers are generated by LLaVA-1.5-7B.
Figure I:
we term Post-hoc Semantic Correction (PSC), suffer from a critical limitation: by constructing preference pairs based on textual discrepancies alone, they are inherently vision-agnostic. Conse- quently, the resulting gradients are poorly targeted, failing to teach the model how to resolve the Perceptual Bottleneck or how address the Lack of Robustness. Moreover, their reliance on ex- ternal feedback makes the data inherently off-policy, which is inefficient for DPO's KL-constrained objective (Azar et al., 2024).
To address the gap, we propose a novel self-correcting method: Perceptual Processing Direct Prefer- ence Optimization (P²-DPO). Our core idea is to have the model generate its own on-policy, vision- aware preference data, which provides a direct and targeted gradient signal for its vision-dominant parameters. Specifically, we introduce two new types of preference pairs, each tailored to one of the core deficits: (1) Focus-and- Baass-nn-EEnaanee Enhance Preference Pair to resolve the Perceptual Bottleneck by contrasting outputs from enhanced versus degraded fine-grained details; and (2) a Visual Robustness Preference Pair boost Visual Robustness by contrasting outputs from processing clean versus noisy signals. To further enhance training efficacy, our framework incorporates an adaptive opti- mechanism and an auxiliary Calibration
mization process, featuring Dynamic Deficit-Weightin loss. This results in an efficient learning loop that improves the visual grounding of LVLMs with high precision, obviating the need for costly human feedback.
In summary, our main contributions are as follows.
We categorize visual causes of hallucination in LVLMs into Perception and Perceptual Processing failures, and identify two overlooked manifestations of the latter. We also critique the existing preference pairs construction methods as vision-agnostic and off-policy.
We propose Perceptual Processing Direct Preference Optimization (P²-DPO). self-correction framework that synergistically combines the generation of on on-policy, vision-aware preference pairs with purpose-built optimization strategy, including a dynamic weighting mechanism and novel Calibration loss, to directly and efficiently rectify perceptual deficits.
2

---

Published as a conference paper at ICLR 2026
We experimentally show that P'-DPO achieves competitive results on challenging hallucination methods without using any such annotations.
benchmarks, outperforming costly
RELATED WORK
General Capability Enhancement A significant body of research aims mitigate LVLMs hal-
lucination Yue et al. 2024; Han et al. 2024: Yu et al. 2024a; Huang et al. 2025). ). One prominent perceive visual evidence by integrating more powerful vision encoders (Lin et al., 2024 Wu & Xie, 2024, Shen et al.) (024), leverag-
line of work focuses on improving the model's 's ability
ing higher-resolution imagery (Li et al., 2022 Hong al.) 2024), or designing refined attention mechanisms (Zhong et al. 2024; Wu et al., 2024). While these methods can reduce hallucination, they often incur substantial computational costs and engineering overhead due to their architecture- level modifications. More importantly, such architecture-cente LVLMs backbones and deployment constraints, and they constitute general-purge purpose enhancements rather than targeted interventions for the specific visual processing failures we address.
solutions are less portable across
DPO-based Alignment and Its Data-Centric Limitations Direct Preference Optimization (DPO) (Ratailov et al.] 2023] An al. 2023 Zhang al. (0224) has become the primary paradigm for aligning LVLMs, with multimodal extensions such as mDPO (Wang et al. 2024) and fine-grained data strategies like RLHF-V (Yu al. 2024b) further advancing the field. However, the efficacy of DPO is fundamentally contingent on the origin and construction of its preference pairs, Current data generation largely falls into two categories: (I) post-hoc textual revision, where model's 's halluci- natory output (y)) is corrected by humans
stronger Al to form the winning response (y"") (Wang) al. 2023b Liu al. 2023 Sun al. 2024), and (2) synthetic injection, where textual errors are deliberately introduced to create contrastive pairs (Zhou et al. 2024).
While these strategies have achieved some success, they are insufficient for resolving the specific visual deficits we target. This insufficiency stems from core methodological issue: both the DPO loss and its accompanying data construction methods are often inherited from text-only domains without critical adaptation for multimodality. This results in three critical flaws. First, the corrective signal, derived from textual differences, is not targeted at the core visual failures we identify-thy Perceptual Bottleneck and insufficient Visual Robustness. Second, the data is inherently off-policy due to its reliance on external sources, which is inefficient for DPO's KL-ccostrriieed objective. Finally, this dependence external feedback makes the process prohibitively expensive and difficult to scale.
3 PRELIMINARIES: CRITICAL ISSUES IN PREFERENCE DATA
The efficacy of Direct Preference Optimization (DPO) is highly sensitive to the properties of the preference data used for training. Below, we dissect two critical axes of concern that motivate our work: the visual grounding the data and relation to the model's own policy.
3.1 VISION-AGNOSTI VS. VISION-GROUNDED DATA
We formalize why Grounded Contrastive Preference Generation (VCPG) strategy provides a more effective optimization signal for the vision-dominant parameters 0 compared to traditional Post-hoc Semantic Correction (PSC) strategies, which are vision-agnostic.
Gradient of Vision-Dominant Parameters. We decouple the model parameters 0 into two distinct sets: 0,, the vision-dominant set, and 0z. the language-dominan set, as detailed in Appendix A The gradient for O2 in DPO is driven by the difference hetween the log-likelihood gradients of the winning and losing texts:
(0))) A Ve, log To Jongg P) Vo, log To (y' Jonig P)
(1)
Thus, the effectiveness of a preference data strategy hinges on its ability to maximize the expected norm of this gradient difference term, (()))(
Information-Theone Analysis via Visual Dependency. To understand how the properties of preference data influence the gradient norm [(]))]] we introduce the concept of Visual Infor-
3

---

Published as a conference paper at ICLR 2026
mation Dependency (VID): Let (I. D, F. go(T), and Y To(. I, P); define VID T(Y; F. 0). This quantity quantifies how much additional information the visual features Fo provide for generating the text Y, beyond what is known from the prompt.
The expected norm Vo,, (Y I, P) is positively associated with VID (motivated by In- formation Bottleneck views (Alemi et al. 2016)). This proposition reveals limitation of PSC methods: since the preferred and dispreferred outputs are typically induced by similar visual evi- dence, we have T(Y F. P,0) R T(Y P,O), so gradient cancellation occurs, leading to smaller value for (0:)) In contrast, VCPG generates pairs with significantly different VIDs, specifically by creating Visual Information Disparity (I(Y"; F F. P,0) > ; P,O)), which maximizes || (6:))l. providing a stronger optimization signal.
||]
Information Geometry Analysis via Fisher Information. Next, we connect the gradient dynamics the Fisher Information Matrix (FIM), a key tool for measuring parameter sensitivity. [Information- geometric interpretation (heuristic)) The Fisher information for the vision-dominant parameters O,, as measured by Tr( (o,,) ), tends to increase with the KL-divergence between the representation pos- teriors induced by the preference pair (y"), y²):
Tr( Fo,) tends increase with EDD [DRc(po(FE) y0,, Po .))]
This provides a key insight: increasing Fisher information in parameter space corresponds in- creasing the separation in the representation space. By generating preference pairs with large visual disparity between y" and y'. VCPG tends to push the posteriors Po y P) and Po y P) farther apart, thereby increasing the KL-divergence and the induced Fisher information on 0,. De- tails and assumptions are provided in Appendix A
3.2 ON-POLICY Vs. OFF-POLICY DATA
Given that DPO ensures the trained policy remains close to the reference policy (TT²),, define response y as on-policy it has a non- toiiiaiival probability under the reference model: Teery I, P) > where c is small positive constant (Yang et al. 2025).
The core of this issue lies in DPO's objective function, which constrains the trained policy Te to remain close to the reference policy Tee via Kl.-divergence penalty. This introduces fundamental limitation for strictly off-policy data. If a preferred response yd lies outside the support of the reference model, denoted as supp(Tret). such that "y P. O, then the KL-divergence term D (Tell (Ter) + oc. This effectively confines the optimized policy's support within that of the reference model, i.e., supp(To) S supp(Tret). making it mathematically impossible learn responses that Net would never generate.
This theoretical limitation manifests as a vanishing gradient during training. The DPO gradient is proportional to a sigmoid weighting factor, VgL x afi - f..)) Av. where A is the log- likelihood gradient difference. For strictly off- off-policy y"", the implicit reward f.. _ (og(Too
( I, P)) diverges to infinity as its denominator approaches zero:
" I, P) + 0 = Aw +00
Consequently, the argument of the sigmoid function becomes infinitely negative, driving the weight- ing factor to zero:
- af f..)
This effectively nullifies the entire gradient update for the most informative, off-policy samples, rendering the learning process highly inefficient. This analysis highlights the critical need for on- policy data generation, a cornerstone of our P2-DPO framework.
4 METHODOLOGGY
Recognizing that Perceptual Processing failures are often "last-mile" errors amenable to self- correction, we introduce P2-DPO, fully self-driven DPO framework that operates without reliance on external feedback. It couples a mechanism for the autonomous generation of preference pairs with tailored optimization process. This approach enables the targeted construction of on-policy, awsee data, which circumvents the core inefficiencies of conventional, post-hoc preference collection.
4

---

Published as a conference paper at ICLR 2026
Stertee strual
Constratted Preferenee
pAppP Training
SEW
An overview of nur proposed p² DPO Framework. The process flows from left right: (I) Derive Visual Inpurs: Based on initial forward pass obtain an attention map (A), we create an enhanced input via cropping (Tug)) degraded inpur via crasure (Tag)) and noisy imput (Lasia), alongside the original image (f), (2) Generaae Preference Pairs: The reference model generates two orthogonal preference pairs. (3) E3PPO Training: Diflerence losses are dy dymamically combined.
Figure 2:
4.1 VISION-AWARE ON-POLICY DATA GENERATION
To address the targeted visual deficits, our approach moves beyond indirect textual revisions by directly intervening on the visual input to generate preference pairs. This causal generation pro- cess inherently yields data aligned with our Visually-Grounded Contrastive Preference Generation (VCPG) strategy. Furthermore, because these pairs are derived entirely from the model's own re- sponses without external feedback, the resulting data is on-policy by definition. Our pipeline is designed to generate two distinct types of preference pairs from single image-prompt instance, strategy that maximizes data efficiency and obviates the need for human annotation. These generated pairs are then subjected to rigorous filtering process to ensure their quality.
4.1.1 FOCUS-AND-ENIANS PREFERENCE PAIRS
To overcome the Perceptual Bottleneck, we build upon the crucial observation that a model's at- tention maps often correctly localize the relevant region even when it is hallucinating. Inspired by previous work (Zhang et al. 2025), which shows that supplementing the input with cropped salient regions can mitigate such errors, we adopt similar strategy to generate our preference pairs.
Our process begins by using carefully crafted prompts (see Appendix for details), designed to guide the model's focus towards key visual areas, to perform forward pass through the reference model Met. We obtain the final head-averaged attention matrices. Specifically, for both the answer-to-token attention (Aua) and the token-to-image attention (Aing), we aggregate the scores from all H heads at each layer via averaging, i.c., Á = 1/ E A.. The final attention map, quantifying the relevance of each of the 2 input image patches to the answer, is then computed via their tensor product: A = Auo & Amng IR 1gLx x1x where L and Le denote the number of layers in the LLM and the connector, respectively. From these maps, we employ an adaptive cropping algorithm to extract a salient region, Tempp that optimally isolates the most prominent visual entity (see Appendix S [for algorithm details). Based on this crop, we derive two distinct visual inputs. The Enhanced Input (Taug) is formed by combining the original image with its salient crop Lerop denoted as lang = Combine Jaro)) ). Conversely, the Degraded Input (Jaeg) is created by applying a mild erasure within the bounding box of the crop on the original image, denoted as lagg = Erase( I, Bbox (Iuop)).
attention maps A by composing two intermediate,
With these two visual contexts established, we generate the preference pair (V Miscaa) The winning response, Viens Meer(Lagg Paahh). ) generated by conditioning the model on the enhanced input and meticulously designed prompt template. The losing response, Vlogus generated from the degraded input and the original prompt. This procedure yields a preference pair where the winning response is derived from state of heightened perceptual clarity, while the losing response stems from a state of targeted visual impairment.
MeeeLaage P), is
4.1.2 VISUAL ROBUSTNESS PREFERENCE PAIRS
We follow carefully designed three-stage process. The procedure begins with single prompt P and an original image
First, we create a degraded version of the image, Jnsise Noise(I), by perturbing it with Gaussian noise. The losing response, s/op = Matt (nnosse: P), is then generated by conditioning our reference model Mar this corrupted visual input. This response represents the model's output when faced
5

---

Published as a conference paper at ICLR 2026
with low-fidelity signals. Next, initial, higher-qualie response, yiah Me,(I,,), is generated by conditioning the model on the original, high-fidelity image I.
To refine this initial response into the definitive winning response yoob introduce Contrastive Amplificcatio. This decoding strategy, building on contrastive principles, has been shown to reduce output hallucinations (Leng al. 2024). The method designates the model conditioned original image as an expert (EP) and on the degraded image noise as an amateur (AT). At each decoding step i, amplifies the difference between their logits, but critically, only within candidate set Vhead of plausible tokens pre-determined by the expert:
the
yt softmax ((1 + Xca) logits (v:) Ama (r
subject to Ue E Vhead (Ver), where Ama a hyperparameter. This process steers generation towards visual fidelity while preserving linguistic coherence.
The output of our data generation pipeline for single input (1. is a set of two orthogonal pref- erence pairs: the Focus-and- Enhance Pair, denoted Diocus (Vieuss Mroens) and the Visual Ro- bustness Pairs, Dmb (webb V/ub)) To ensure both the quality and utility of this raw dataset, subsequently apply principled filtering process, A pair is retained only if it meets a two-fold crite- rion: (1) the perplexity (PPL) of both responses must fall below fluency threshold Tppl. and (2) its log- bog-robbbiility margin, .M log Pre((y")) log Pter(y)) ), must lie within an effective learning range [Olow, Bhigh/- Further threshold choices are provided in Appendix D
4.2 CALIBRATION DIRECT PREFERENCE OPTIMIZATION
Grounded in the insight that Perceptual Processing failures are on-policy errors requiring proba- bility redistribution, our training objective is a composite DPO loss. It synergistically integrates signals from two orthogonal preference pairs- targetnng the Perceptual Bottleneck and Visual Ro- bustness respectively- and unifies them via dynamic weighting mechanism that adapts to the model's 's evolving deficits.
4.2. 1 OBJECTIVE FOR THE PERCEPTUAL BOTTLENECK
To address the Perceptual Bottleneck, we begin with the standard DPO loss Focus-and- Enhance pairs (Discus, denoted as Lodpo_focus:
Te (Vtnons I, p) Teat (yocus I, P) ) Even though our Focus-and And-Ennance pairs provide some visual grounding, the standard DPO loss learns purely correlational signal, rewarding the preference for viccus over vlocus without attributing it to the causal impact of the visual intervention. To instill this causality more robustly, we introduce a complementary Calibration Loss (Ccali). This objective derived from a preference model over the Perceptual Confidence Gain, Ax A log ( Lang) - log π (dg) As formally proven in Appendix E minimizing LCamb is equivalent to maximizing ANo (Vi which in turn maximizes the Visual Information Dependency (VID) of the winning response, defined as the mutual information I(Y focus F+ P.O). The resulting loss function is:
log (Blog Te (reieen P)
foposs -E ( Voocs
Tref (yeeass I1, 3 log
(4)
Divans
To Toissous U, Terop, P) B3 log
Tree (yocuss Tamps P)
Lcaib (
(Dricn Wlrea)) )~ Discos log
Blog
To To Ideys P)
Tree Ideg P)
(5)
The complete objective for the Perceptual Bottleneck, Lincus, a weighted sum of these two com- ponents, balanced by a hyperparameter Acalib:
Lioeus Lapo focus + Aealih LCalib
4.2.2 OBJECTIVE FOR VISUAL ROBUSTNESS
Symmetrically, to tackle the Lack of Robustness, we apply the DPO loss to our Visual Robustness pairs (Dob). This loss, Ldpo.rob. trains the model see through noise by favoring the ideal response Whab over the corrupted one Viep even when conditioned on the same noisy input Inoise:
ddparrob = -R (y Dea log
(Blog log
Te (y Ti I Inoise, P).
13 log
Tol ( Incise P)
P2)
Tee Tnoise- P)
Anvises P)

---

Published as a conference paper at ICLR 2026
4.2.3 DYNAMIC DEFICIT- WEIGHTING
While both preference pairs originate from the same input (4, P), they address different scenarios. Focus-and-Enhance excels at improving local perception, whereas Visual Robustness targets the global robustness, meaning their training signals should be dynamically balanced. To address this, we introduce novel mechanism called Dynamic Deficit- Weinchtinggg (DDW). The idea behind DDW that if the cropped key region has higher CLIP score with the text compared to the original image, it indicates that the cropped region is highly relevant to the query, and thus its training proportion can increased. This mechanism first computes a Perceptual Gain Ratio, r, using a pre- trained CLIP model to diagnose the primary deficit for a given sample, where r = CLIPScore P. CLIPScorc
This ratio, which indicates if the bottleneck (r I) robustness is the dominant issue, is then mapped to an adjustment factor a = O'max tann ("=7-0) that dynamically allocates weights as Wtocushobust Wbuse - Our final unified objective, Luxat, is the dynamically weighted sum over the mini-batch, enabling tailored corrective pressure for each sample:
Luvial
E
Wlocus
Liocus + Wrobust Lego rob
(8)
EXPERIMENTS
Our comprehensive evaluation assesses both the effectiveness and the underlying mechanisms of P?_ DPO. First, we perform targeted validation of our method's s ability to rectify the specific Perceptual Processing deficits identified. Second, broaden the scope uating P2-DPO against prominent alignment methods under fair and controlled conditions. Third, we assess data efficiency by examining whether our on-policy, vision-aware data more sample- efficient for DPO than traditional off-policy sources. Finally, to understand the contributions of each component within our framework, we perform an ablation study to isolate their individual effects.
benchmark comparison, eval-
5.1 EXPERIMENTAL SETUP
Base Models. Our experiments are primarily built upon the open-source LLAVA-1.5-78 (Liu et al., 2024b). To demonstrate the generalizability of our method across different model architectures and sizes, we also conduct experiments on the recently released Qwen2.5-VL-7BB3B (Bai et al. 2025).
Data Generation. Our preference pairs are generated using prompts (image-question) sourced from the RLHF-V dataset (Yu al. 2024b). We do not use any of the provided human preference labels, ensuring that our data generation requires no human feedback or annotation.
Evaluation Benchmarks. We conduct comprehensive evaluation on suite of standard hallucina- tion and trustworthiness benchmarks, strictly adhering to the official evaluation protocol for each.
POPE (Li al. 2023b): Probes for object-level hallucinations through binary (yes/no) questions
about object presence.
HallusionBench (Guan al. 2024 Evaluates fine-grained discrimination, testing model's
ability to distinguish objects from their contexts and attributes.
MMHal-Bench (Sun al. 2023): A question-answering benchmark that leverages GPT-4 for nuanced scoring of factual consistency regarding detailed object properties.
AMBER (Wang al. 2023a): multi-dimensional benchmark that assesses model trustworthi- ness by evaluating its generated text against rich, object-level annotations.
TA (Singh et al., 2019): Leveraged for our targeted Perceptual Bottleneck analysis, its ground-truth bounding boxes are essential to quantitatively measuring attention alignment.
5.2 TARGETED VALIDATION OF PERCEPTUAL DEFICITS
Table I: Evaluation on AFR and Processing Ac- curacy (AFR > 14.0).
We first conduct two targeted experiments to directly validate method's 'sss efficacy in resolving the specific Perceptual Pro- cessing failures identified in the introduction.
Model
AFR (Avg. P-Ace, (%
LLaVA-1.5-BB
14.73
66.29
Validating Perceptual Bottleneck Mitigation. To evaluate the Perceptual Bottleneck, we select TextVQA (Singh et al, 2019) for its ground-truth bounding boxes, which facilitate
DPOS
15:57 10.84
65.71 J0.58
+ P'-PPO
18.71 13.98
70.10 13.81
7

---

Published as a conference paper at ICLR 2026
The BoL
WARLY
Toe Burt spert wilh
Figure 3: TTe model's response effect for blurry imagesie 0.20)
precise quantitative measurement. Inspired by prior work for evaluating visual grounding (Zhang al., 2025). we use the Attention Focus Ratio (AFR) the ratio of mean attention inside vs. outside the answer's s box-to quantify visual perception. To isolate subsequent processing failures, we then measure Processing Accuracy (P-Acc.) in high-focus subset where the AFR already exceeds our baseline's average (AFR > 14.0). The results in Table reveal baselines exhibit high AFR but poor P-Ace., indicating that attention is accurately localized, but it still gives a wrong answer. In stark contrast, P2-DPO significantly boosts both metrics, effectively bridging this perception-processin gap and confirming it rectifies the bottleneck.
Robustness to Visual Perturbations. We evaluate Vi- sual Robustness via a systematic stress test on the POPE benchmark (Li et al., 2023b). Gaussian noise with in- creasing standard deviation (a) is added to all images, and the resulting F1 scores
87.5
POPE Scove
48.5
measured. Figure shows that performance degrades with stronger pertur- bations, but robustness to light noise-common in real- world settings-in especially critical. We compare orig- inal LLaVA-1.5-7B, standard DPO on RLHF-V data, and our method. While all models decline under heavy noise, p2-DPO demonstrates significantly higher stability at low-to-moderate noise levels, outperforming LLaVA- 1.5-7B by over 4 Fl points at a = 0.20. As shown in Figure 3 we find that the model provides correct answers even when visual degradation is difficult for humans to recognize. These results val- idate our design improves early-stage robustness and mitigates premature degradation under minor perturbations.
20.2
15.5
370
Larootan Nous inturaity
Figure 4: Model accuracy on the POPE benchmark under increasing levels of Gaussian moise (or).
5.3 BENCHMARK COMPARISON
The key advantage of P2-DPO is its ability achieve competitive performance without relying on costly human or external Al annotations. To ensure fairness, we compare against prior work with similar data scales and compute budgets. As shown Table 2 our method consistently improves hallucination-sensitive metrics. On LLaVA-1.5-7B, p2-DPO not only achieves top scores across POPE, BaallusionBBenhh, and MMHal-Bench, but also demonstrates particularly dramatic improve- ment AMBER's 's relational reasoning, boosting the FlR score by substantial +8.5 points over the base model.
We further validate the generalization ability on Qwen2.5- where P²-DPO again shows con- sistent gains. Notably, it achieves the most significant improvement on the challenging MMHal- Bench, reducing the hallucination rate by -3.13% while simultaneously boosting the AMBER rela- tional reasoning score to 80.9 (+3.0). These results demonstrate three key advantages: (i) consistent gains hallucinationn-snsii benchmarks, (ii) stable improvements across different architectures,
8

---

Published as a conference paper at ICLR 2026
Table 2: Evalinatoon of P'DPOO against prominent alignment methods onne hallucination benchmarks, Baselines include the training-free VCD (Long et al. 2024) the Al-foodback-driven HA-DPO (Zhao et al. 2022,, and methods trained on human preferences (DPOxuev. V- RLHF-V dataset Yuu al. 2024bb. method, using only self-geuerated data, achieves highly competitive superior results. Arrows show dellas vs. the curresponding Bave in each block: green imprrvvmennt red degradation: for 4 metrics (MMHla
DPO
AMBER CHAIRi/Hll), smaller better so green arrows point downward.
Made
Phiro Seurce
POPE.
HalloinBackk (Ace
MMiied Beach
AMBRER
9/acc
Hall.
PlA
TTaSK 5776
$1.96
11.90
08.16
0167
1.9
9.1
66.1
67.4
HA-DPO
$2.54
$3.81
99.25
56.35
0.60
1.97
63
50.9
88.3
66.1
18.5
$4.1
87.91
00
14:01
0.56
7.66
R4
77.1
91.1
73.7
66.1
Thonos
M4/12
1M.12
M6.38
16:71
30:11
51.31
0.04
2366
64
58.
1417
724
MA
19-pet cheeno
Selt
$4.55
87.90
19.97 -13
87.64 11
26,37
30.85
15A2 mmm
12:6 cm
244
367
78.7
780
Oveen VL-318
$6.26
05.05
79
03
28.15
34.10
50.10
0.42
3.30
36.1
65.2
17.
Barea
$6.41
$8.89
90.40
88, 66
25,99
33:53
94.66
0.40
1.41
6.8
59.5
03.0
85.5
78.
DPO Ohers)
Saf
66 88 ve.r
88, 66 14.
91.50 1111
$6.95 m
91.11 PM
34.68 .m
59.66 cM
pm
3.40 mm
100
16.3
92.7 um
44.3 mm
Qownd.5 5-V0 VI-1I
Baw
$1:03
86.7%
01
17-57
1447
90.
K04
124
Thrron
$1.00
60.51
19-12
34.05
74
62.63
04345
7651
6.3
261
82.1
12.
10-pe1 (Cheno
Selt
66.79
87.
WUB5
19.21
96.70 mM
41.02
65.19
0.33
340
674 p0
73
and (iii) annotation efficiency, since all benefits are obtained without external supervision. We also evaluate Swwe2.55VL--2 and observe similar trends. Taken together, they confirm that by enhanc- ing Perceptual Processing, P²-DPO enables models to better exploit inherent capabilities and deliver robust improvements at no additional annotation cost.
5.4 ANALYSIS OF ON-POLICY DATA
As argued in the preliminaries, preference data from off-policy sources (e.g., human feedback) creates significant policy gap. which is challenging for DPO's KL-constrained objective to bridge. To ver- ify this, we employ the Implicit Preference Strength (IPS) metric (Zhao et al., (023b) to measure the pol- icy divergence between various preference pairs and the base model. A large negative IPS significs a sub- stantial policy gap, predicting difficult and inefficient learning process for DPO. Table 3 provides a stark contrast. The standard Off-Policy data exhibits a severely negative average IPS of -58.52 with a 68.3% % negative ratio, quantifying the substantial learning difficulty it imposes. In contrast, our on-policy data with positive IPS and low variance is not particularly difficult to learn and offers a stable learning signal.
Table 3: IPS anulysis reveals a severe policy in off pulicy data, contrasted with strong alignment our on-policy methods.
Metric
Off-Policy (RLHF-V)
Focus- Enhance
Visual Robust.
Avg. IPS -58.52
12.35
45.58
Std. Dev.
147.08
25.61
60.56
Neg. Ratio 68.3%
8.5%
44.2%
5.5 ABLATION STUDY
We evaluate our two-part design-data con- struction. consisting of foeus-and-Enhanee pairs (FEPs) and Visual Robustness pairs (VRPs), and training strategy, including the calibration loss (Leath) and Dynamic Deficit- four benchmarks (Table Training only with VRPs under stan- dard DPO leads to performance drop, and similarly, training only with FEPs also degrades results, indicating that both data types are nec- essary and complementary. On the optimiza- tion side, discarding LCalib also leads to overall performance degradation, demonstrating the effec- tiveness of our loss design, while replacing Dynamic Weiiiit-eiggtingg with equal static weighting consistently degrades results, validating that different losses should carry different weights for dif- ferent types of errors. Overall, each component contributes positively to the final performance.
Table 4: Ablation study of our method's components. report per- formance key hallucination and reasoning benchmarks. Each com- ponent contributes positively to the final performance.
POPE AMBER MMHal
HallusionBench
Configuration
F1
Hal
Score
gAce
fAce aAce
Weighting (DDW)-acros
P'-DPO
87.42
29.27
2.43
26.37
30.05
55.62
Ablatine Pairs:
w/o FEPs
85.84
34.70
2.30
23.56
26.62
51.10
w/o VRPs
85.27
33.60
2.26
21.02
25.71
52.10
Ablating Strategy:
w/o Ceais w/o DDW
86.17
33.10
2.33
23.56
27.75
53.41
86.68
31.20
2.39
23.96
28.61
52.70
CONCLUSION
We identify that LVLMs have a failure in Perceptual Processing- a critical "last-mile" problem. To address this, we propose P2-DPO, self-correcting framework that generates on-policy, vision-

---

Published as a conference paper at ICLR 2026
aware preference data to directly target these failures. Experiments confirm that P2-DPO achieves performance comparable to or exceeding other training paradigms that rely on human annotation, across a range of ballucination benchmarks, while demonstrating significant advantages in learning efficiency and robustness.
7 ACKNOWLEDGEMENT
This work was funded part by the National Natural Science Foundation of China grant under number 62536004, and in part by the Key-Area Research and Development Program of Guangdong Province under number 202380303030001, and in part by the Science and Technology Program of Guangzhou under number 2024A04J63 10, and part by the Fundamental Research Funds for the Central Universities 2025ZYGXZR021.
STATEMENTS
LLM Usage Statement In adherence to the ICLR 2026 policy, we disclose that Large Lan- guage Models (LLMs), including services based GPT-4, were utilized in the preparation of this manuscript. Their role was primarily that of a writing assistant, grammar correction, and polishing the phrasing of sentences improve clarity and readability. The core scientific ideas, experimen- tal design, results analysis, and final conclusions were conceived and articulated exclusively by the human authors.
Ethics Statement We have considered the ethical implications of this work. Our research is built upon publicly available datasets and open-source models, and we do not anticipate any direct neg- ative societal impacts. The preference data generated by our P--DPO framework is derived from the model's own outputs on these public datasets and does not involve any private or sensitive user information. While our work aims mitigate model hallucination, which is step towards safer AI, we acknowledge that no method is perfect, and the potential for misuse of Large Vision-Language Models remains broader community concern. To foster transparency and enable further research into the safety and reliability of LVLMs, We commit to releasing the implementation of our method and the evaluation scripts upon publication.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 161iv: 2016.
Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. American
Mathematical Soc., 2000.
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song- Direct preference-basee based policy optimization without reward modeling. Advances in Neural Infor- mation Processing Systems, 36:70247-70266, 2023.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandricllo. A general theoretical paradigm to understand leam- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Masane Fuchi and Tomohiro Takagi. Erasing concepts from text-to-image diffusion models with few-shot unlearning. arXiv preprint arXiv:2405.07288. 2:1, 2024.
10

---

Published as a conference paper at ICLR 2026
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pp. 14375- 14385, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint (tti::2501.1294/4 2025.
Tianyang Han, Qing Lian, Rui Pan, Renjie Pi, Jipeng Zhang, Shizhe Diao, Yong Lin, and Tong Zhang. The instinctive bias: Spurious images lead to hallucination in milms. CoRR, 2024.
Wenyi Hong, Weihan Wang, Qingsong Lx, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang. Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Lei Huang. Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 1352) 2025.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing- Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 13872-13882, 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre- tre-trrining with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- languae models. arXiv preprint arXiv:2355.10355, 2023b.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label important things for large multi-modal models. In proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 26763-26773, 2024.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15%. 2024.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint
00253, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 26296-26306, 2024b.
Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312. 15997, 2023.
11

---

Published as a conference paper at ICLR 2026
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728 -37441 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multi- modal experts for generalist multimodal large language models. Advances in neural information processing systems, 37:42048 42070, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang. Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv: 14525, 2023.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13088 131 130 Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 00.18653/v1/024., fin.244.iininn--c..... URL Findings acl. 7757
org/2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 3377
Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang. Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arxiv:2406. / /839, 2024.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, al. Amber: An llm-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023a.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng. Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: survey, arXiv preprint arXiv:2307.12666, 2023b.
Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, Guannan Jiang, Xi- aoshuai Sun, and Rongrong Ji. Controlmlim: Training-free visual prompt learning for multimodal large language models. Advances in Neural Information Processing Systems, 37:45206-45233 2024.
Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognition, pp- 13084-13094, 2024.
Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yer Kan. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. arXiv preprint arXiv:2411.02712 2024.
12

---

Published as a conference paper at ICLR 2026
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXhv:2309.174211 9(1):1, 2023.
Zhihe Yang. Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 16610-10620, 2025.
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang. Wentao Ye, Bosheng Qin, Siliang Tang. Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 11944-12955 2024a.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024b.
Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective. arXiv preprint arXiv:2402.14555 2024.
Jiarui Zhang. Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where to look: Training-fiee free perception of small visual details with multimodal llms. arXiv preprint arXiv:2502. 02.002.77422 17422, 2025.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero- init attention. arXiv preprint arXiv:23003. 16199, 2023.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng. Keyang Xu, Yuanhan Zhang, Di Fu, Chun- yuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01255, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303.18223 1(2), 2023a.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023b.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hal- lucinations: Enhancing Ivlms through ballucination-awer direct preference optimization. arXiv 2023c.
preprint arXiv:2311.
Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Zitnik, and Pan Zhou. Moextend: Tuning new experts for modality and task extension. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 80-91, 2024.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402. 11411, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS: TARGETED PARAMETER OPTIMIZATION BY MAXIMIZING VISUAL INFORMATION DISPARITY
This section aims formally show that the Visually-Groundee Contrastive Preference Generation (VCPG) strategy more effectively concentrates the optimization pressure of Direct Preference Opti- mization (DPO) onto the targeted vision-dominant parameters 01 than the Post-hoc Semantic Cor- rection (PSC) strategy. We begin with formal definitions of the model and DPO gradients, introduce anformation-theorette tools, and then connect the statistical properties of preference pairs param- eter sensitivity measured by the Fisher Information Matrix (FIM).
A.1 FORMAL DEFINITIONS: MODEL, PARAMETERS, AND DPO GRADIENT
A.1.1 PARAMETER DECOUPLING
We assume IVL.Ss exhibit functional specialization: different parameter subsets tend to serve dif- ferent functions. For tractability, we adopt coarse two-way partition: vision-dominant subset that is more responsive to visual evidence and thus more plastic under vision-heavy supervision, and language-dommnant subset that predominantly supports textual reasoning. This specialization is supported by evidence from three complementary perspectives architecturally, via purpose-built bridging modules such as the Q-Fomeer BLIP-2 Li et al.] (2023a); empirically, via PEFT results (e.g.. LLaMA- Zhang et al. )2023) suggesting that visual integration can be localized to a sparse parameter subset; and mechanistically, via interpretability analyses that associate visually grounded knowledge with specific neural pathways /uchi & Takagi (2024). Guided by this evidence, we partition the parameters 8 into two sets:
O1 (Vision-Dominant Parameter Set): the subset of parameters whose updates predomi- nantly driven by visual evidence, i.c., they exhibit larger log-likelihood gradients when condition- ing the image (and the corresponding fused visual representation F,) varies.
Parameter Set): the complementary subset whose updates are pre- dominantly driven by linguistic signals, primarily supporting textual understanding and reason- ing. These parameters operate over linguistic inputs as well as the fused visual representation Fo; while chiefly governing text generation, they may still interact with visual information depending on task demands.
02 (Language-Dominal
A.1.2 DPO LOSS AND GRADIENT DERIVATION
For single training instance comprising an original image Tanig- a text prompt P, and a preference pair (y",, yy),, the DPO loss is:
[(0) - -log (frr ft),
where the implicit reward is
fg log To Inrig: P) Tarig. P)
(10)
Tee( /
Applying the chain rule:
VoL(O) a llog (2)
Vo(fi ft).
(11)
dz
1==
Using dog (x) - a2)) a(-20 and noting is fixed, we obtain:
VoL(O) aa(f: fw)) 3 (Vo log To ( Long P) Va log (p Long. P))
(12)
Let 1 = 3 (n - )w)) > 0. Projecting onto the 0 subspace yields:
Ve, [(0) = A [Ve, log Te Long. P) - Vo, log (o( Tong. P)]
(13)
Define the graddintttt difference term
(0)) Vo, log To Lorigs P) Ve, log (o Torig P).
(14)
Thus, the effectiveness of a preference strategy hinges on its ability to increase the magnitude of A (01) (in expectation over preference pairs).
14

---

Published as a conference paper at ICLR 2026
A.2 INFORMATION- THEOOOTTITTEEICC INTERPRETATION OF GRADIENT DYNAMICS
[Visual Information Dependency, VID] Let P) D be an image prompt pair drawn from the data distribution, F, go( ) be the fused visual representation. and Y .o(. P) be the model output. We define the Visual Information Dependency (VID) under parameters 0 as the conditional mutual information
VID & F.Y:F.IPP
where (1, ~ D. F = go(1), no II
(15)
Equivalently,
VID(O) = E log PY Fo, P)
(16)
Po P)
under the induced joint distribution of F.,
[Assumption: Gradient-VID association] Under the induced joint distribution of (Y, Fø, P), the expected norm of the vision- vision-paramenn parameter log-likelihodd gradient positively associated with VID:
V log (Y (Y I, P)]] tends to increase with VID(O) I(Y; F, P,O).
(17)
Setting of the two strategies (basic cases). We define the two preference construction strategies follows.
PSC: yi ~ TT(. Torig P) is generated from the original image and prompt; Y = Edit(Y')) is obtained by post-hoc editing without access additional visual evidence.
VCPG: There exist an enhanced image I+ and degraded image / such that Y Te(- I+, I+,Pi P) and y no(- I-, P). We assume a degradation channel /- D(I+) (noise/lossy transform), and Torig is equal or close to I+ (the training image). When optimizing DPO, we form the training tuple (Twigg P, y yt) and score both responses under the same context (Jang, P) in the DPO loss.
[DPl-based VID ordering]
(a) (PSC) Let F, go (Fung)) Assume the edit operator is conditionally independent of F, given YI, i.e., Fg yi yw forms a Markov chain under (P,0). P). Then by the data processing inequality (DPI),
I P,0) s I(Y P,0).
If the edit is small (token-level perturbation), we assume the VID reduction bounded:
I(Y'; I(Y"; P,0) εVID
for a small EVID > 0. (b) (VCPG) Let I = D(I+) be degradation channel and F# (I+)). Let y+ ~ Tel: It,P) and Y- ~ ne I- P). We assume degradation reduces visual dependence in expec- tation:
F P,0) 2 ;( P.0)).
If Tarig- I+ (or is sufficiently close), evaluating the quantities against Tong preserves this ordering up to small mismatch dimg-
[Local smoothness gradient field] Assume Vo, log Toly Torig P) is L-Lipschitz y under task-appropriate distance d(..)):
|| Vo, log ( Torigs P) Ve, log mo(u2 Tong: P)]l < L d(n).12))
Ctradient-differeent behavior under PSC vs. VCPG) & Ve, log To( Longg P) and define the random gradient-difference
()))) -
-
where (Y"",, Y!) are constructed by the corresponding strategy and then both are scored under (Torig: P) in g()). Under the basic cases specified above:
15

---

Published as a conference paper at ICLR 2026
(i) PSC: IfY" EY) with small edit distance d(Y Y.) S E, then by Lemma A.2
|| (PSS)) (0:) (lg(()) - g(V")) S Le.
(18)
Moreover, Lemma A.2 implies I(Y F. P,0) is close to I(YY, F, P.O) up to EVID,
and by Assumption A.2 the corresponding expected gradient norms are close (up to a constant factor). Hence PSC yields small gradient-differencs
equation [8]
i VCPG: Let I be enhanced and I degraded with - = D(I+), and assume Long = I (or
the mismatch is bounded by ding). Then by Lemma (b) and Assumption A.2 there exist > 0 and small & 2 0 such that
[[())( 2 a, E [Ig(())) S E,
(19)
where tends to increase as the preferred output becomes more visually dependent, and tends to decrease as the degradation strengthens. Therefore, by the triangle inequality.
E (ACCP))
(01)1 [l g(Y')l] > Ellla E[Il((())) 2 - (20)
In the limiting case > O, we have E EJA (V)())))) + gY i.e. the gradient- difference magnitude is significantly amplified as the degraded sample becomes visually un- informative.
Proof. (i) For PSC, the Lipschitz smoothness in Lemma A.2 directly gives equation 18 when the edit distance small. The DPI statement in Lemma A.. a) formalizes that post-hoc editing without extra visual evidence cannot increase visual dependence; with small edits, the VID gap is bounded by EVID- Assumption A.2 then connects this bounded VID gap to a bounded gap in expected vision- gradient norms.
(ii) For VCPG, Lemma bb2 states that degradation reduces visual dependence in expecta- tion. Combined with Assumption A.2 this induces gap between [Y]](( and EY")( as Applying the triangle inequality yields the expectation-level lower bound equa- QED
in equation 19
tion 20
Consequences for DPO updates. Combining Theorem A.2 with the DPO gradient O1. Ve, Ve, C(0) = -A (())), we obtain: PSC produces a small update due to equation 18] VCPG yields a larger-magnitude update due to equation 20, thereby increasing [|)(),) (in expectation) compared to PSC.
A.3 LINKING STATISTICAL PROPERTIES OF PREFERENCE PAIRS TO THE FIM
We now connect the above analysis to the Fisher Information Matrix (FIM). For preference obser- vation (y" > y/) under (Torig P), define the DPO preference likelihood as
Poly" > y Jonig: P) A afw -1).
(21)
The score function restricted to 01 is
81 A Vo, Po > y Tonyy P)-
(22)
We are interested the trace Fo, From the DPO gradient derivation, we have (0) A0)(0)))
where A = of - -ww). Therefore.
Tr(Fo,) Ep || (0) ED [4 (6))))
(23)
To relate Tr(Fo,) to the statistical structure of preference pair, consider conditional distribu- tions over the visual representation space induced by the model and data, denoted as (F, | y, P).
cntemmation-gommetiii interpretation (heuristic)) Within the DPO framework, Tr(Fo,) tends in- crease when the preference pair induces a larger separation between the representation posteriors:
(F, tends to increase with Ep DDi(Po(FF , (F v',
(24)
Definition of po F. P). Under the induced joint distribution (T, P) ~ D. F, =o((T), Y To(- 1, P). we use
po (Fo. y, P) x po(y) F,, P) po P)
the posterior over fused visual representations associated with generating y under prompt P.
16

---

Published as a conference paper at ICLR 2026
Heuristic explanation. The term A())) can be interpreted as the parameter-space force that shifts the model toward representation patterns consistent with yd rather than y/. From an information- geometric viewpoint (Amari & Nagaoka (2000), larger KL separation between the two posteriors typically implies a larger required parameter displacement, which manifests as a larger |A(0))
and thus larger Tr(Fo,
The VCPG strategy, by design, generates preference pairs (y", y²) that are separated by a wide visual-dependency gap. visually-rich preferred text yd and visually-agnostic dispreferred text y' tend to require substantially different ideal visual features, meaning the posteriors pe (P y", P) and po(F, P) are typically farther apart. Therefore, VCPG increases the Fisher information in the vision-parte parameter subspace by enlarging the separation in the representation space, yielding more targeted and effective optimization.
B OPTIMIZING ATTENTION MAPS VIA PROMPT ENGINBERING
The efficacy of our adaptive cropping algorithm critically dependent on the quality of the answer-to-image attention maps produced by the reference model Mee. To elicit concentrated and well- aligned maps, we first run Mar with care-
Comparison Attention Fecus Ratee (AFR) Across Layers
Maaktn boa Pate saro
fully crafted prompts that explicitly ask the model to locate the relevant region
before answering. From this forward pass we extract two intermediate, head- averaged attention tensors: the answer- to-token attention Asok and the token-to- image attention Aung- For each module and layer we average over the H heads, Á # Σi= An, and compose the fi- nal answer-te cotwer-to-iman to-image map via their tensor product:
12 1.5
Figure Layer-wise comparison the baseline and our optimized prompt. The optimized prompt (in red) sistently achieves a higher AFR, particularty in the mid-to high layers (7-26), indicating substantially improved ability localize relevant visual informa- tion. This enhanced focus provides a high-quality signal for our subsequent cropping algocithm.
Attention Focus (AFR) between
Am @ Ame E I.Xx.XIIXN
where L and L. denote the number of layers in the LLM and the connector, respectively, and N2 the number of image patches. This A quantifies the relevance of each of the N patches to the predicted answer.
Given A, we derive saliency map over the x N patch grid and feed it to our adaptive crop- ping procedure to obtain salient region Temp (Appendix C. Concretely, a multi-scale sliding win- dow searches over candidate boxes and maximizes sharpness score-the normalized difference between the summed attention inside window and that of its adjacent neighborhood-whhi en- forcing image-bound constraints. The selected window is then mapped from patch indices to pixel coordinates to yield a tight bounding box. This crop is subsequently used to construct the Focus-and- Enhance visual contexts introduced in the main text: the enhanced input lagg = Combine [ Iewo) and the degraded input Ideg Erase( I, Bbox(Temp))). which together form vision- grounded prefer- ence pairs for PP-DPO.
B. EXPERIMENTAL SETUP
Dataset and Metric. To quantitatively evaluate attention localization, we use the TextVQA (Singh al. 2019) dataset, which provides ground-truth bounding boxes for textual answers. Our evalua- tion metric is the Attention Focus Ratio (AFR) (Zhang et al., 2025), defined as the ratio of the mean attention intensity inside the ground-truth box to the mean attention intensity outside. A higher AFR score indicates a more accurate and focused localization.
Prompting Strategies. We compare two strategies:
Baseline Prompt: The default user-assistant conversation template used by LLaVA (Liu et al. 2024b), where the question is presented directly.
17

---

Published as a conference paper at ICLR 2026
Optimized Prompt: A structured, task-decomposing prompt designed to explicitly guide the model's focus before answering.
B.2 RESULTS AND ANALYSIS
Our experiments demonstrate that the optimized prompt significantly enhances the model's ability focus on relevant visual evidence. The comparative results are visualized in Figure which plots the average AFR score across all 32 layers of the vision-languaa backbone for both prompting strategies.
The analysis of the plot reveals several key insights:
Superior Overall Performance: The "Optimized Prompt" curve (in red) lies substantially above the "Baseline Prompt" curve for the vast majority of layers, indicating a consistent and significant improvement attention focusing.
The most dramatic improvements occur in the mid-to-high layers of the model (approximately layers 7 through 26). These layers are critical for abstract reasoning and semantic understanding. The high AFR in these layers suggests that prompt is not just affecting low-level visual processing, but is successfully guiding the model's deeper cognitive processes to ground themselves in the correct visual evidence.
Impact on Deeper Layers:
Justification for Cropping:
The substantially higher AFR scores validate our approach. By using reliably generate high-quality. focused attention maps. These maps
the optimized prompt, serve as an excellent foundation for our adaptive cropping algorithm to accurately isolate the most salient visual entity, which the cornerstone of our Focus-and- Ennhanccee data generation pipeline.
B.3 FINAL OPTIMIZED PROMPT TEMPLATE
Based on this quantitative validation, we adopted the following template for generating all attention maps in our pipeline. It modifies the standard LLaVA instruction format to include "chain-o thought" style directive.
USER: <image>
Your task is answer the question by first locating the relevant region the image. Question: {question}
ASSISTANT:
Here, (question] the placeholder for the original question from the dataset. This structure compels the model to prioritize the localization task, resulting in the improved attention focus shown above.
C CROPPING ALGORITHM
In this section, provide the detailed methodology for adaptive cropping algorithm, which is responsible for extracting the salient region Teoop from a given image based on the model's internal attention maps A.
C.1 ALGORITHM MOTIVATION AND OVERVIEW
The primary goal of this algorithm is identify and isolate the most prominent visual entity that the model is attending to. A simple approach of taking the single point of maximum attention is often too noisy and fails to capture the full extent an object. Therefore, our algorithm employs more robust, multi-scale sliding window approach. It evaluatee candidate bounding boxes of various sizes, selecting the one that not only contains high attention scores but also exhibits the sharpest contrast with its surrounding regions. This sharpness criterion, measured difference, is key to ensuring that the crop is tightly focused on a single, salient entity rather than a diffuse area of mild attention. The entire process is detailed in Algorithm
normalized attention
18

---

Published as a conference paper at ICLR 2026
C.2 KEY STEPS EXPLAINED
Algorithm Adaptive Cropping from Attention Map
Require: Attention map A. image size (W, H). base box shape (b bn)
Require: Scale ratios R (1.0, 1.2.... .2........))
Ensure: Bounding box coordinates (#1, 91. 22, 1/2)
1: Initialize Dbest -00, Buest f- None
2: for each ratio E R do
3: // Compute crop dimensions
4: wr bw rr; hy t- b₁,
5:
Nw +
(A))]
6:
No &- {
7:
I/ Sliding window attention sum
8:
Auum + Sliding SeehingiinoSSum((( (Nv,, Nw))
9:
// Locate attention peak
(Nmmx Fmax) &- arg max m
11:
Omax + Asum Mmmxx Fmax Ondj < www.Acccaanttttetion..ooooo
12:
(Mmax Ummax)
13:
I/ Score atlention sharpness
14:
Dr x- (Amax (gd))/ (N Nn)
15:
if D, > Doest then
16:
Dress + Dr
17:
Buest + ([Vauxx Tmma)) ) (Na, Nw (hy, we))
18:
end if
end for
20: // Decode best crop box
((y", (N R. Nw), (h", w²)
Beest
22: t xx= x== "W//di,, (A)) + w²) 1/
24: Je + ty" - (H/dimg (A)) + h2t22
clip(Te, w² /2, W u=t /2)
ye + clip(ye, h² /2, H h²12))
26: // Convert to box coordinates
27: x1 IIc w² /2: /1 + Wc h2/22 /2
28: x2 f w² /2; 1/ Mc h² /2
29: return (int(21), (nt(t))... int( int (u())) =0
Lines 2-6: The core of the algorithm is a loop over several scale ratios. For each ratio, it calculates the target crop dimensions (w,, h2)) and converts them into the corresponding number of blocks (Nw, N2.) in the attention map. This allows the algorithm to search for objects of different sizes.
Line 8: A 2D summed-aree area table (or a direct sliding window sum) used efficiently calculate the total attention within every possible rectangular window of size (N,, N..). This creates new map Asm where each point represents the attention sum of a candidate region.
This is the crucial step for selecting the best scale. Instead of picking the window with the absolute highest attention sum (which would favor larger windows), we calculate normalized difference score D, (Line 14). This score measures how much "sharper" the peak attention is compared its immediate neighbors. By dividing by the area of the window (N x No), we normalize for size and the subsequent if block (Lines 15-18) to select the scale that produces the most focused, unambiguous attention peak.
Lines 14-18:
Once the best-performing scale ratio and window position are identified, the algorithm unpacks this information (Line 21) and converts back into image- coordinates. It calculates the center of the bounding box (Lines 22-23), clips it to ensure it stays within the image boundaries (Lines 24-25), and finally computes and returns the top-left and bottom-rigg right coordinates (Lines 27-29).
Lines 21-29:
19

---

Published as a conference paper at ICLR 2026
D DATA FILTERING IMPLEMENTATION DETAILS
To ensure the quality and utility of our self- generated preference pairs, we applied a prin- cipled, data-driven filtering process. The raw dataset, consisting of 5.733 pairs, was analyzed based two key metrics derived from the ref- erence model (Tett): Perplexity (PPL) for flu- ency, and Log-Probabilityy Margin for learning utility. The goal was to remove outliers and re- tain approximately 95% of the data, focusing on samples that were both fluent and presented meaningful learning challenge. The 95% re-
Table 5: Statistical summary of filtering metries for the 5,733 raw preference pairs. Thresholds were deter- mined based on this distribution to retain approximately 95% of the data.
PPL (Win)
PPL
Log-Prob Margin
Statistic
(Lose)
Mean
1.76
1.79
0.017
Std. Dev. Min
0.16
0.15
0.094
1.34
1.24
-0.294
25% (Q1)
1.66
-0,039
50% (Median)
1.74
1.77
0.015
tention target was chosen to balance two com- peting goals: filtering out low-quality or unin- formative samples, while preserving sufficient data diversity and volume for effective training. This range also aligns with standard statistical practices ge.g.. 2.5th-97.5tt percentiles) for outlier exclusion, ensuring that the retained dataset re- flects the core distribution without being skewed by extreme values.
75% (Q3)
1.85
1.87
0.075
Max
2.48
2.78
0.402
Final Threshold max(.) < 2.50
[-0.174, 0.208]
Our filtering process is based on two-fold criterion applied to each instance:
Fluency Control via Perplexity (PPL). Perplexity measures the fluency of a generated re- sponse. A high PPL often indicates incoherent grammatically incorrect text. To remove such outliers, we specified a maximum perplexity threshold, Tppl and discarded any pair where either the winning losing response exceeded this threshold. Based where the 99.9th percentile for the maximum PPL was 2.48, we set a conservative threshold to avoid removing valid samples.
data distribution (see Table (5),
Criterion: max (ax((P(()")
Tppl
Selected Value: Tppl 2.50
2. Learning Utility via Log-Probability Margin. The log- prg-pobaability margin, defined as M log ")) log (eer""))"" quantifies the reference model's preference strength and indicates the learning difficulty of pair. Pairs with extreme margin values (either very large and positive, or very negative) are less effective for training. To select for samples in the sweet spot of learning, we retained only those pairs whose margin fell within the central 95% of the data distribution. This was achieved by setting the lower and upper bounds, 0jo and hhgh to the 2.5th and 97.5th percentiles of the margin distribution, respectively.
Criterion: Orow S. .M < Ohigh
Selected Values: 6jow 0..174, Bhgh 0.208
D.1 SUMMARY OF DATA DISTRIBUTION AND FILTERING
summarizsss the statistical properties of the key metrics our raw dataset. The final filtering criteria were chosen based on this analysis be robust and reproducible. After applying both criteria, 5,446 pairs (94.9%) were retained for the final preference pairs.
E DERIVATION OF THE CALIBRATION LOSS (LCALIE)
The standard Direct Preference Optimization (DPO) framework is derived from preference model over pairs of textual responses (y" y/). Our novel Calibration Loss, Leali. extends this principle to domain: it models preference perceptual confidence gains. This appendix provides first-principles derivation of Leath from a reward modeling perspective, analogous to the derivation of the standard DPO loss.
20

---

Published as a conference paper at ICLR 2026
E. A REWARD MODEL FOR PERCEPTUAL CONFIDENCE GAIN
First, we define the core quantity of interest: the Perceptual Confidence Gain. For a given policy and response y,, we define its gain (v) as the increase in log-probability when the model is provided with the salient visual crop Terop versus the degraded context dog"
(v) = log π ( Lero:: P) - log π (yy P)
(25)
This value, (y), quantifies how much the policy's 's confidence in response y is calibrated by the specific visual evidence in Leop-
Our goal is train the policy Th to achieve a high confidence gain on the correct response (y"",, while ensuring this gain is meaningfully larger than any spurious gain the model might have on an incorrect response (y'). To formalize this, we define a hybrid preference model. We posit that the confidence gain from our policy To the winning response y" should be preferable to the confidence gain from the reference policy The on the losing response y'.
Using Tradley-Terry-liie model, we can express this preference p" as:
p" (A (v") (v)) = (r(AA,, (v r(A mm (v')))
(26)
where r(.) is an implicit reward function over the confidence gains.
E.2 DERIVING THE Loss OBJECTIVE
For simplicity and directness, we define the reward of given confidence gain as being directly proportional to the gain itself, scaled by the DPO temperature parameter B::
r (v)) 3 (v)
(27)
Substituting this reward definition back into our preference model, we get:
p" (A (v") > And (2/)) = o(3 ANo (1 B A (s/))
Now, we expand the gain terms A using their definitions:
p (...)=o (1(10s To (1 Larop, P)
(log
Trerly I,o Lerop. p)) Teer |Teg P)
To(y"" TTag P)
The final step is to formulate the training objective using Maximum Likelihood Estimation (MLE). We aim to find the policy xa that maximizes the liglliiel of this preference data. The negative log-likelihood loss for single preference pair is therefore:
L = - log (A.o (v")) > ATA (3'))
(28)
This gives us precisely the form of our Calibration Loss, LCalib. averaged over the dataset Diocus:
To [1, lexop P) 3 log Tearly Frerly"\Aace | P)
Frer(y/ (1, Acrops P)
Ccalb = (y .f ) log o [Blog Te(y" [Lagg P)
(29)
This derivation demonstrates that Lealib is not an ad-hoc objective, but principled loss grounded in reward model tailored to instill visual causality by maximizing the confidence gain on correct responses, regularized by the bascline gain on incorrect ones.
E.3 CONNECTION TO VISUAL INFORMATION DEPENDENCY (VID)
We now analyze how minimizing Lealib implicitly maximizes the Visual Information Dependency (VID) of the winning response nh Recall that VID is defined the conditional mutual information F( P. 9), where ~ TT | P) and F, = (I). The objective of Lcalbb is to maximize the term inside the sigmoid, particularly the perceptual confidence gain for the winning response:
A (y ") log (y Tempp P) log no (y) | P)
(30)
Let's 's analyze the two components of this term. The enhanced context Terop) provides a rich visual representation, which we denote Ft. Conversely, the degraded context ldey provides noisy and
21

---

Published as a conference paper at ICLR 2026
uninformative representation, Fø Therefore, maximizing AT, (v") is equivalent maximizing the difference in log- likelihoods given these two representations:
max (log ne (v (F,)) - log Ne (1 (F7))
(31)
From information -eotmmttion-thoretii viewpoint, maximizing log To (T) F.) is closely related to increasing the dependence between the output random variable and the visual representation, i.e., T(Y; Fg P, () (Alemi et al., 2016). Thus, the optimization forces the model to learn policy mo that satisfies two conditions simultancously:
High Information Utilization: To maximize log The (y" the model must learn extract and utilize large amount of information from the rich visual features FT.. This directly pushes for higher conditional mutual information I(;; F P.0).
2. Low Information Reliance on Noise: To minimize log (F) the model must learn to distrust the noisy features Fg and reduce its reliance on them for generating y"
By jointly optimizing for both conditions, the training process explicitly rewards the model for making its generation yd more dependent on the high-quality visual evidence present F This directly translates to an increase in Visual Information Dependency (VID), effectively teaching the model to "see" rather than merely "guess".
F IMPLEMENTATION DETAILS
This section provides a comprehensive overview the experimental setup and hyperparameters used obtain the results presented the main paper.
F.1 GENERAL TRAINING CONFIGURATION
All models are fine-tuned using the AdamW optimizer (Loshchilov & Hutter 2017). We employ cosine learning rate schedule with a peak learning rate of 2 x 10 ", linear warm-up phase of 100 steps, and final learning rate annealed zero. The weight decay set to 0.1. To ensure computa- tional and memory efficiency, all fine-tuning is performed using Low-Ram bow-Rank Adaptation (LoRA) (Hu al. (022). We apply LoRÁ adapters with rank of
and an alpha scaling factor of 64 to all linear projection layers within the attention blocks of the language model. For all Direct Pref- erence Optimization (DPO) based loss components, the temperature parameter 3 consistently set to 0.1.
All experiments were conducted on system equipped with NVIDIA A100 (80GB) GPUs. Our implementation is built upon the PoTorc framework and leverages the Hugging Face Transformers and PEFT libraries.
F.2 P2-DPO SPECIFIC HYPERPARAMETERS
The key hyperparameters unique to our P2-DPO framework are detailed below. These values were selected based preliminary experiments and held constant across all main results.
Calibration Loss Weight (Aeal):: The balancing coefficient for the Calibration Loss is set to
Acalib 0.3.
Contrastive Amplification Coefficient (Aa): For generating the Visual Robustness Pairs, the coefficient for Contrastive Amplification is set Àaa = 0.1.
Dynamic Deficit- Weeicit-Weiggtigg (DDW) Parameters:
Base weight (Whase): 0.5
Maximum adjustment factor (Omaa) : 0.2
Temperature (T): 0.1
G ADDITIONAL EXPERIMENTS ON DATA QUALITY
In this section, we present two additional experiments that specifically examine the quality of our constructed training data: (1) external validation of the PPL Margin filtering strategy using
22

---

Published as a conference paper at ICLR 2026
Table 6: CLIPScore statistics for attention- guudedd vs. inverse cropping.
Group
Metric
Count
Mean
Median
Std
Min
Max
att.crop
improvement
500
1.22
1.16
0.20
1.06
3.17
att.crop
improvement.diff
500
3.64
3.09
1.85
1.85
12.25
att.inverse
improvement
500
0.93
0,93
0.18
0.22
1.55
att.inverse
improvement.diif
500
-1.76
-1.29
3.75
-15.15
8.95
human-annotated references, and (2) analysis of the reasonableness of CLIPScore dynamic weighting signal for attention attention-guided guided crops.
G.1 EFFECTIVENESS OF PPL + MARGIN FILTERING
Our preference pairs are constructed from the prompts and images in the RLHF-V dataset. Dur- ing training.
only the prompt and image and do not use the human preference annotations provided by RLHF-V. To validate the quality of our self-constructed preference pairs and to check whether the PPL Margin filter truly identifies "problematic pairs", we construct an external evalu- ation task based on RLHF-V's human annotations.
Evaluation task with human references. Each RLHE-V sample contains an image, a question, and two human responses: high-quality answer Aно (High-Quality Answer) and low-quality answer ALO Loo--uuaitty Answer). In this experiment, these human annotations are used only as evaluation references and do not participate training.
For each self-constructed preference pair (y", y²), we construct an evaluation prompt that contains: (i) the original question; (ii) the human high-quality answer as golden reference; (iii) the human low-quality answer negative reference; and (iv) two model responses, denoted as Response A and Response B, corresponding to y and y randomly assigned A/B.
A strong general-pune purpose language model (Qwen3-Max) is then used evaluator to assign quality score in 2,333) to each and B, based on their semantic proximity to the human high- /low-quality answers:
3 points: semantically close to Auq. with only minor descriptive differences and correctly capturing the main idea;
2 points:
basically reasonable but not fully aligned, e.g., missing some information or adding harmless verbosity;
1 point: closer to AL. or clearly wrong / hallucinatory.
Standard-answer-barr invalid-sample criterion. Within this framework, we label preference pair an invalid sample that should be removed if either of the following holds:
the response we mark as the winner, yd, receives the lowest score point), indicating that it is semantically closer to the human low-quality answer;
n receives strictly lower score than y indicating that the evaluator believes that the true preference direction opposite to our constructed win/lose relation.
This "standard-anseee answer filter" depends only on the question, the human high-/low-qualing answers, and our generated responses. It is completely independent of PPL, Margin, or attention maps, and thus serves as an external semantic-level reference.
Quantitative comparison with PPL. Margin. According to the above standard-answer filter:
the Qwen-based evaluator marks 510 preference pairs invalid samples;
PPL. + Margin filtering alone removes 287 samples;
the intersection between these two contains 236 samples.
23

---

Published as a conference paper at ICLR 2026
Thus, among the 287 samples removed by PPL + Margin, have
236
2 82%
287
that are also judged invalid by the independent standard-answer filter (Qwen + human high/low- quality answers).
Two observations follow:
High overlap indicates that the rules are not arbitrary. More than 80% of the samples removed by PPL. + Margin are simultaneously flagged as problematic by an independent evaluator that explicitly compares against human high-/low-qualin answers. This suggests that the PPL + Margin filter is highly consistent with a human-anchored semantic criterion, rather than pruning examples random.
The thresholds are intentionally conservative. Although the standard-answer filter iden- tifies 510 suspicious samples in total, PPL + Margin removes only a subset of them (287 samples) and retains roughly 95% of the overall data. PPL + Margin therefore acts fallback cleaner for extremely had samples, instead of aggressively deleting all borderline cases, and thus achieves balance between data quality and data diversity.
Overall, this external evaluation confirms that PPL Margin filtering strategy reliably iden- tifies and removes a large portion of low-quality or weeeerene-dircction-nooeee samples, while remaining conservative in scale.
G.2 REASOSABLENESS OF CLIPSCORE- BASED DYNAMIC WEIGHTING
We further design controlled experiment to verify the reasonableness of using CLIPScore dynamic weighting signal in our data construction pipeline.
Setup. We randomly sample 500 examples from the training set and compute CLIPScore between the prompt P and: (i) the original image I, and (ii) the attention-guided cropped image Lump. We define
diff CLIPScore(P. (P, Iamp) CLIPScore (P,
ratio CLIPScore(P,, Lero))
CLIPScore(P,
At the same time, we construct reverse-cropping control group: for the same set of examples, we crop from the complementary (low-attention)) region of the attention map to obtain Livv and compute the corresponding diff and ratio in the same way. We denote the ceantion-uuided cropping group as att_crop, and the reverse-cropping group att_inverse.
Results. Table sulmarrzzee the statistics, and the full distributions (omitted here for space) show consistent trends.
In the att.crop group, the mean of improvement ratio is about 1.22, and the median is about 1.16, indicating that cecentionguuide cropping systematically improves CLIPScore, with roughly
20% relative gain on average. The mean of improvement _iff is about 3.64, and the median is about 3.09, showing that the semantic similarity between the cropped image and the prompt is consistently increased for most samples.
In contrast, the att_í nverse group, the mean of improvement_ratee is about 0.93, and the median is also about 0.93, both below 1, meaning that when deliberatcly crop from regions not attended by the model, CLIPScore decreases about 1..7 and the median is about - 1.29, exhibiting clear negative shift.
systematic way. The mean of improvement.diff
Although the standard deviations indicate some variation across individual samples, the signs and magnitudes of the means and medians make the overall pattern clear: when cropping along high- response attention regions (att crop), CLIPScore typically improves; when cropping from com- plementary regions (att inverse), CLIPScore typically degrades.
24

---

Published as a conference paper at ICLR 2026
Implications for dynamic weighting. These results show that CLIPScore is highly sensitive to whether the crop focuses on semantically relevant regions. When the crop aligns with high-response attention regions, most samples satisfy ratio > 1 and diff > 0; when the crop lies in complementary regions, most samples satisfy ratio < 1 and diff < O. Using ratio and diff to drive the dynamic weighting of Wlocus and Wrabust therefore allows us, in statistical sense, to amplify the contribution of "well-cropped" samples and suppress the impact of "badly cropped" samples, which is precisely the intended behavior of our weighting mechanism.
H ANALYSIS OF ATTENTION REDISTRIBUTION AND FOCUSING
To quantify how training reshapes the balance between textual and visual evidence inside the LVLMs, we analyze (i) global and layer-wise shifts in attention from text tokens to image tokens, and (ii) the evolution of the Attention Focusing Ratio (AFR), which measures how tightly attention concentrates on ground fround-truth object regions.
H.1 GLOBAL AND LAYER- WISE SHIFT FROM TEXTUAL TO VISUAL ATTENTION
We first measure how much attention answer tokens allocate to visual tokens versus textual tokens across training. For each checkpoint (base model and epochs 1-4), compute two aggregated statistics over the entire greference-triiiig split (more than 5,000 multimodal examples):
Lise mean image ratio: for each layer, we compute the fraction attention mass from answer tokens image tokens (image sum sum. ratir,,, then average this ratio across
all layers.
Global image ratio across all layers:
image-sum
Global Image Ratio
image-sum sum + text-sum
where image- sum and text-stteeeeeee sum aggregate attention weights image/text tokens over all layers and heads.
Both aggregation schemes show a consistent global shift toward visual evidence: the mean image ratio increases from 0.1971 (base) to 0.2055 (epoch 4), a relative gain of about +4.3%; the global image ratio increases from 0.2204 to 0.2291, relative gain of about +4.0%. Although the absolute change (<x 0.0087) is small on the [0, 1] scale, these metrics are averaged over all layers, heads, and answer tokens, and computed on the full training split, indicating stable redistribution of attention probability mass from text to image tokens rather than noise on a small subset. The epoch- wise trend remains consistently above the base level and aligns with the reduction of hallucination metrics reported in the main results, suggesting that improved visual grounding is accompanied by measurable shift in attentional reliance toward image tokens.
To better understand how this shift is organized inside the network, we further break down the visual attention ratio layer-wise by grouping the 32 Transformer layers into low (0-7), mid (8-17), and high (18-31) segments. The change is not a uniform "flat boost" of visual attention, but a structured redistribution:
In the low segment (layers 0-7), the average visual ratio increases only modestly (about +1.5% 1.5% relatively), and in some shallow layers (e.g., layers 2-5) it even slightly decreases.
In the mid segment (layers 8-17) and high segment (layers 18-31), the average visual ratio increases more strongly, by roughly +5.5% and +6.1 % respectively.
* Several mid/high layers exhibit the most prominent gains, for example
layer 21: 0.2535 - 0.2710, layer 30: 0. 0.1058 - 0. 0283,
For some upper layers (e.g.. layers 22, 27, 30), the visual attention ratio exhibits a nearly monotonic increase from epoch 0 to epoch 4, while shallow layers remain comparatively stable. This pattern indicates that the model is reallocating attention budget from early layers toward mid/high layers, which have been identified key stages for visual-language fusion and hallucination mitigation
25

---

Published as a conference paper at ICLR 2026
in prior analyses of LVLMs internals [2,3,4]. Importantly training objective does not impose any explicit layer-wiss wise rescaling, the observed redistribution emerges from optimization under the calibration-bass preference objectives, which are designed to make preference differences depend on visual interventions rather than purely textual paraphrasing.
H.2 ATTENTION FOCUSING RATIO (AFR): FOCUSING ON GROUND-TRUTH OBJECT REGIONS
Global shifts toward image tokens do not, by themselves, guarantee better grounding. To verify that the model is increasingly focusing on correct regions, we revisit the Attention Foeusing Ratio (AFR) defined in Sec. 5.2 of the main paper. Using TextVQA, which provides ground-truth bounding boxes. Higher AFR, indicates a stronger concentration of attention on the ground-truth object region relative to background irrelevant areas.
In the main paper, AFR was reported only for the base model and final model. Here we include intermediate checkpoints at epochs 1-3 to track the full training trajectory. The average AFR over all layers increases from 5.54 (base) to 5.64 (epoch 4), a relative gain of about +1.8%, indicating moderate but consistent strengthening of attention key object regions. Grouping layers into low/mid/high segments yields:
Low: 0.75 + 0.79 - 0.79 + 0.78 + 0.78 (absolute change Rs +0.03);
Mid: 7.05 - 7.03 + 7.10 + 7.12 - 7.16 (absolute change F: +0.11);
High: 7.20 - 7.30 7.32
7.32 + 7.33 (absolute change = +0.13).
The largest absolute gains thus occur in mid/high segments, while the low segment fluctuates slightly around its base value. The overall shape of the AFR curve over layers (i.e., which layers act as peaks and valleys) remains almost unchanged: "visual- semantic" layers such as layer 14 already have high AFR in the base model and remain the main peaks, but their peak values further amplified. For example,
layer 14: 14.74 17.10 18.02 18.44 18.72,
relative increase of about +27%. Thus, the layers that originally most responsible for fo- cusing on ground-truth object regions become even more concentrated on these regions under our training, rather than being replaced by different subset of layers.
Taken together, the global, layer-wise, and AFR analyses indicate that the proposed training scheme preserves the original hierarchical division of labor across layers while gently but directionally real- locating attention from shallow layers and background regions toward mid/high layers and ground- truth object regions. The model not only "looks a more structured and semantically focused manner, consistent with the reductions in ballucination observed in the main quantitative results.
the image" in aggregate, but does in
ADDITIONAL EXAMPLES
We present several additional examples visually demonstrate the effectiveness P2-DPO in rec- tifying the specific perceptual failures we identified. These case studies highlight our model's ability overcome the Perceptual Bottleneck, its enhanced robustness against visual noise, and its overall superior performance in complex reasoning compared to the baseline LLaVA VA-.5-..
26

---

Published as a conference paper at ICLR 2026
Aedes mesen rock Tirced Meeeiiaa
Figure 6: Example of improved overall performance in complex scene. Given a general prompt such as "Describe this image", PP-DPO produces a more accurate and detailed description than the baseline, showcasing the holistic improvements brought by our method.
legar lagg
Cuevervatse
Ispal teag:
Casroration
Figure 7: Visual robustness under motion blur. P--DPO correctly rejects hallucinated objects (e.g.. "car", "book", "truck") that the baseline model incorrectly predicts, demonstrating improved resilience to visual degradation.
27


Reviews:
[{"review_id": "uWwlrmllGA", "reviewer": "Reviewer_m9BD", "summary": "This paper introduces P²-DPO, a self-correcting framework for reducing hallucination in Large Vision-Language Models. It targets perceptual processing bottlenecks rather than perception failures, generating on-policy, vision-aware preference pairs to improve attention and robustness. With a Calibration Loss and Dynamic Deficit-Weighting, P²-DPO enhances visual grounding and robustness, outperforming human- and AI-feedback DPO baselines without external supervision.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a novel perspective on hallucination in LVLMs by distinguishing between perception failure and perceptual processing failure. It highlights the latter as an overlooked yet solvable issue that can be addressed through self-correction within the model, offering new insight into the root causes of hallucination.\n\n- The proposed on-policy strategy, which generates vision-aware contrastive preference pairs from the model itself, effectively removes the dependence on human or AI feedback used in traditional DPO frameworks.", "weaknesses": "- Limited experimental scope: The experiments are primarily conducted on LLaVA-1.5-7B and Qwen2.5-VL-3B, but it is unclear why the authors did not include the more commonly used Qwen2.5-VL-7B model. This omission limits the completeness of the evaluation and raises questions about the method’s scalability and consistency across model sizes.\n\n- Potential self-reinforcement bias in on-policy data: Although the on-policy preference generation strategy avoids the off-policy issue, self-generated preference pairs in the early training stage may inherit the model’s existing biases or hallucinations.\n\n- Lack of qualitative interpretation of improvements: While quantitative results are comprehensive, the paper lacks intuitive visual evidence explaining how the Calibration Loss and Dynamic Deficit-Weighting improve the model’s attention distribution or semantic grounding. Visualizations such as attention map changes would make the effectiveness of these mechanisms clearer and more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4R1ataP1U", "reviewer": "Reviewer_cpy5", "summary": "In this paper, the authors propose Perceptual Processing Direct Preference Optimization, a new method to address hallucination, mainly focusing on the visual end and self-generated preference data.", "questions": "1. How does this method perform on Qwen-VL?\n2. During training, how does the attention pattern change? Does it tend to favor visual tokens rather than textual tokens?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The writing is clear, and the results are solid.", "weaknesses": "1. The model architecture is LLaVA, and the version is relatively old. It's unclear how it performs on current new models.\n2. Lacks extensive comparison with related work, such as [1], which also uses model self-generated data and has a consistent idea—namely, focusing on hallucination in visual perception. Therefore, I have doubts about the novelty.\n\n[1] Calibrated Self-Rewarding Vision Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5cabkf8eBx", "reviewer": "Reviewer_LaWn", "summary": "The paper proposes P2-DPO (Perceptual Processing Direct Preference Optimization), a self-supervised alignment method for Large Vision-Language Models (LVLMs) designed to mitigate hallucinations by focusing on Perceptual Processing rather than Perception failures.\nUnlike prior DPO or RLHF approaches that rely on human or synthetic preferences, P2-DPO generates on-policy, vision-aware preference pairs through two mechanisms: (1) Focus-and-Enhance pairs (for perceptual bottlenecks) and (2) Visual Robustness pairs (for degraded inputs) and uses them for Reinforcement Learning training. To enhance training efficacy, they further introduce a Calibration Loss to align visual evidence with text generation and a Dynamic Deficit-Weighting (DDW) scheme for adaptive balancing. Experiments on hallucination benchmarks (POPE, HallusionBench, AMBER, MMHal-Bench, TextVQA) show consistent improvements over DPO baselines and comparable results to human-feedback-based training.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper offers a compelling and underexplored perspective: separating Perceptual Processing failures (i.e., hallucinations despite correct attention) from Perception failures. This framing adds analytical depth and could motivate new diagnostic tools for LVLMs.\n\nThe idea of generating on-policy, vision-grounded preference pairs directly from the model’s own attention maps is interesting.\n\nThe experiments span several challenging hallucination benchmarks (POPE, HallusionBench, AMBER, MMHal-Bench, TextVQA) and include targeted validation (e.g., AFR, noise robustness), with clean tables and consistent results.", "weaknesses": "Although the method avoids human labels, generating multiple augmented views and attention-driven crops for each sample adds nontrivial computational overhead.  Could the authors provide results to demonstrate what is the runtime overhead (per training sample or epoch) compared to standard DPO?\n\nThe entire method depends on the quality of attention maps and cropping heuristics. If attention localization fails, the generated preference pairs may reinforce spurious regions or noise. How does the model behave when the attention maps are themselves incorrect—does this lead to reinforcing spurious crops? How would the approach handle multi-object scenes where attention maps have multiple disjoint foci?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)