Cat-PO: Cross-modal Adaptive Token-rewards for Preference Optimization in Truthful Multimodal LLMs
Authors: Zhixiao Zheng
Zheren Fu
Zhiyuan Yao
Dongming Zhang
Zhendong Mao
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models; Preference Optimization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multi-modal Large Language Models (MLLMs) have shown remarkable generative capabilities across multi-modal tasks, yet remain plagued by hallucinations where generated textual contents are semantically inconsistent with the input images. This work reveals that existing multi-modal preference optimization methods exhibit shortcomings at the preference data decoding stage. Specifically, different response tokens exhibit varying degrees of association with visual content, and consequently, their contributions to reducing hallucinations and generating high-quality responses differ. Nevertheless, most existing methods do not distinguish in their treatment, often handling them uniformly. To address this challenge, we propose a novel preference alignment method: Cross-modal Adaptive Token-rewarded Preference Optimization (Cat-PO). Building upon direct preference optimization, Cat-PO calculates hierarchical visual relevance rewards for each response token at global, local, and semantic levels. It then organically integrates these three rewards to construct a smooth reward mechanism and designs an innovative KL-based customized loss for rewarded tokens, thereby enabling fine-grained correction of hallucinatory outputs. Extensive experiments on various base models and evaluation benchmarks demonstrate that our Cat-PO can significantly reduce hallucinations and align with human preferences to enhance the truthfulness of MLLMs.

Published as a conference paper at ICLR 2026
CAT-PO: CROSS-MODAL ADAPTIVE TOKEN-REWARDS FOR PREFERENCE OPTIMIZATION IN TRUTHFUL MUL- TIMODAL LLMS
Zhixiao Zheng', Zheren Fu', Zhiyuan Yao', Dongming Zhang", Zhendong Mao
1 University of Science and Technology of China
2 State Key Laboratory of Communication Content Cognition, People's Daily Online (zhixiac. zheng, yaozhi yuan) @mail 1 ustc. edu. cn [fzr, zdmao) @uste. cn, zlangdanggih@@peppee..m
ABSTRACT
Multi-modal Large Language Models (MLLMs) have shown remarkable generative capabilities across multi-modal tasks, yet remain plagued by hallucinations where generated textual contents semantically inconsistent with the input images. This work reveals that existing multi-modal preference optimization methods exhibit shortcomings at the preference data decoding stage. Specifically, different response tokens exhibit varying degrees of association with visual content, and consequently, their contributions to reducing hallucinations and generating high-quality responses differ. Nevertheless, most existing methods do not distinguish in their treatment, often handling them uniformly. To address this challenge, we propose a novel preference alignment method: Cross-modal Adaptive Token-rewarded Preference Optimization (Cat-PO). Building upon direct preference optimization, Cat-PO caleulates hierarchical visual relevance rewards for each response token at global, local, and semantic levels. It then organically integrates these three rewards to con- struct smooth reward mechanism and designs innovative KL-based customized loss for rewarded tokens, thereby enabling tine-grained correction of hallucinatory outputs. Extensive experiments on various base models and evaluation benchmarks demonstrate that our Cat-PO can significantly reduce hallucinations and align with human preferences enhance the truthfulness of MLLMs.
INTRODUCTION
The success of Multimodal Large Language Models (MLLMs) marks a significant advancement in artificial intelligence research[Lin et al. (2024b); Amirloo et al. (2024). By integrating visual information with Large Language Models (LLMs), MLLMs have demonstrated unprecedented capabilities in multimodal understanding, reasoning, and interaction Xiao et al. (2024); Pi et al. (2024); Zhang et al. (2024). However, MLLMs exhibit notable hallucination problem, where the generated textual descriptions are inconsistent with the input visual content. This phenomenon includes describing non-existent objects, incorrect object attributes, or relationships [Bai al. (2024); Gunjal et al.] (2024). The hallucination issue causes a disconnect between outputs and visual facts, severely degrading user experience and undermining the reliability of downstream applications, thereby limiting their deployment in real-world scenarios Liu et al. (2024)): Liang et al. (2024).
To alleviate this issue, strategies incorporating preference learning, such as Reinforcement Learning from Human Feedback (RLHF) Christiano et al. (2017), have been widely investigated as a form of fine-tuning. The core idea is to leverage preference feedback to align model outputs with desired expectations. Recently, Direct Preference Optimization (DPO) Rafailov et al.] (2023) has gained prominence for achieving excellent results without a separate reward model by simplifying complex reinforcement learning steps. Existing work demonstrates that DPO, by efficiently incorporating preference data, mitigates hallucinations in MLLMs, and improves the alignment with human
'Correspondngg author. author..
https:///itth// htt.:/ittt.b...o.o.oooooaaaaa wttpp//.ccmo//g/ii/////caagggggg

---

Published as a conference paper at ICLR 2026
token
token laptop
token cup
token the
Cross- Attomtion
Cross- modal Samlrrtty
0.336
0.673
0.635
0.160
laptop
a cup are on the taible.
DPO
maximal
Gradlent Distribution: Flat
objects
Cat-PO
madimal Ikolhood
Gradient
Distribution:
Rewarded Tikens Rario
What dbRect are on The sabl??
Targeted
Figure The motivation of our framework. (a) A visual question answering example where the model identifies "a laptop and cup" on table, with cross-modal attention heatmaps and cross-modal similarity scores indicating the model's visual focus and word importance the response. (b) performance comparison of token-rewarded DPO, showing AMBER F1 (1) improving and MM-Hal Hallucination Rate (L) declining as the percentage of rewarded tokens increases. (c) A comparison of standard DPO versus Cross-modal Adaptive Token-rewarded Preference Optimization (Cat-PO). The former uses a flat gradient distribution for maximal likelihood optimization. And the latter employs targeted gradient distribution, suggesting potentially superior performance for the latter in refining a pre-trained MLLM.
preference Li (2023b); Zhu et al.] (2024); Chen al. (2024). However, in current human preference alignment processes, different response tokens processed by the model have varying degrees of relevance to the visual content, and their contributions reducing hallucinations and generating high-quality answers also differ. As shown in Fig. (a), When MILMss process different tokens, the cross-modal attention they allocate to the image varies, and the token-image similarity also differs, indicating that different tokens exhibit distinct degrees of association with the visual content. Nevertheless. most existing works suffer from two primary limitations: (1) They overlook the varying degrees of association between different tokens in the response and the visual content, as well as their differing contributions to high-quality outputs, treating all tokens uniformly and thus lacking fine-grained correction, as depicted in the upper part of Fig. (c). (2) They rely external visual detection models, additional noise injection techniques, expensive closed-source LLM API, or even external tools, thereby neglecting the intrinsic capabilities of MILMs and leading waste of existing resources and increased costs.
Therefore, how to deeply exploit token-level fine-grained alignment signals, construct more refined DPO feedback mechanism, and fully leverage the inherent multimodal capabilities of MLLMs to reduce additional costs and overhead remains critical issue. Motivated by this. we conducted series of explorations. As shown by the statistical experiment in Fig. (b), when we applied DPO only to the top 50% rewarded tokens in chosen responses, we observed significant improvements in hallucination metrics AMBER-FI -1 and MM-Hal compared to the original DPO. Furthermore, applying DPO with all rewarded tokens yielded even more outstanding results.
Building upon these explorations, propose Cross-modal Adaptive Token-rewarded Preference Optimization (Cat-PO). This framework fully leverages the multimodal capabilities and advantages of MLLMs deeply mine token-level fine-grate grained alignment signals between text and images, using token-rewards for Cat-PO, with the aim of more effectively mitigating hallucinations. simplified pipeline is shown in the lower part of Fig. (c). Specifically, within the MI.MMs, before the image
2

---

Published as a conference paper at ICLR 2026
features (projected by CLIP Radford et al. (2021) and ViT) fed into the LLM's transformer layers, we first calculate the cross-modal semantic similarity between response tokens and the image, representing the semantic relevance of tokens to visual content. Subsequently, within the transformer layers, based on the cross-modal attention of response tokens to the image, we compute the global and local relevance of each token the visual content. Furthermore, we normalize and aggregate the three hierarchical relevance scores, map the result through an activation function, and compute the final reward for each token. Finally, we design novel Cat-PO loss based on token-level rewards and KL divergence for further optimization. Experiments on open- source datasets and benchmarks demonstrate that our Cat-PO achieves excellent performance, significantly reducing hallucinations and improving response accuracy, thereby enhancing model truthfulness. Concurrently, this work offers new perspective on mitigating hallucinations by fully exploiting the inherent multimodal capabilities of MILMS without introducing external technologies or tools.
Our main contributions are summarized as follows:
1. We propose Cross-modal Adaptive Token- rewards for Preference Optimization (Cat-PO) in MLLMs. By assigning token-rewards to highlight visually critical tokens and incorporating fine-grained KI. regularization, Cat-PO effectively reduces multimodal hallucinations.
2. We introduce a hierarchical token-rewards calculation method that relies solely on the model's inherent multimodal capabilities, without introducing any external tools or tech- nologies. Specifically, it first computes global relevance based on cross-modal attention between text and image, then calculates local relevance based on patch entropy, and finally uses cooal semantic similarity for further refinement.
3. We conducted extensive experiments across multiple datasets and benchmarks to evaluate the effectiveness of Cat-PO. Notably, on the our proposed Cat-PO outperforms existing state-of oftth--r-t methods by 7% 15% metrics.
and MM-Hal benchmarks,
2 RELATED WORKS
2.1 MLLMS HALLUCINATION
MILMMs hallucination refers outputs that are factually inconsistent with the input image, such as identifying non-existent objects, misdescribing attributes, or misinterpreting relationships. For example, mentioning "dog" in a landscape image that contains animals Bai al. (2024).
To address hallucinations in MLLMs, researchers have proposed a variety of strategies, which broadly classified as training-free or training-based Xiao et al. (2025). Training-free methods, including decoding strategies like Opera Huang et al. (2024) and VCD'Leng et al. (2024). training-based approaches reduce hallucinations through further training. Among these, preference learning such as RLHF Christiao et al. (2017) are prominent for their efficiency and effectiveness.
2.2 PREFERENCE LEARNING FOR HALLUCINATION
Preference learning was initially applied LLM alignment via methods such as RLHF with PPO. These approaches typically necessitate an explicit reward model and involve complex reinforcement learning. Recently, DPO has gained widespread adoption as a simpler and more stable alternative to traditional alignment techniques. HA-DPO Zhao et al. (2023) constructs high-quality sample pairs for preference learning. POVID Zhou et al. (2024a) creates fine-grained DPO dataset by injecting hallucinated text and adding noise to images. MDPO Wang et al. (2024) addresses "unconditional preference," where the model may ignore image. CSR Zhou et al. (2024b) iteratively constructs a preference dataset by self- seffraenen responses, integrating visual constraints into reward modeling.
Recently, RLHF-V Yu et al. (004)) collects segment-level human preference data and performs dense DPO training. TPO Gu et al. (2024) explores token-level information in DPO for LVLMs. V-DPO et al. (024) pairs response preferences with image-contrast preferences and employs vision-guided DPO reinforce visual context learning.
3

---

Published as a conference paper at ICLR 2026
(2) Extract the Features from MLLM Itself
(4) Our Cross-modal Adaptive rewarded Prefarence
Input
(3) Compute the Token-Rewards
(Without External Tools/ / Models)
Optimization (Cat-PO)
Attention Scores
Feature
Local 19g
Pawoards
Pawcards
token E Chosen:
Projector
Crosommol))
Api JantyAA Scores)
Question:
Festure Space
Languago Model
Token Rewards
Scores
Please
Image.
Sarr
The
token E Rejected:
Responses:
Aur F-AAAAA Scoresh
Chosen:
bed
Lot-
cat lies on
1-8
the floor.
200
Chower
Rejected:
cal runs an the best
Semaars
Similarity Scores
Telectand
Figure 2: Overview of our proposed Cat-PO framework: into the feature space via CLT,+ViT, and the textual question/response tokens are embedded by LLM tokenizer. (2) Cross-modal attention and semantic similarity are extracted the multi-modal transformer to hierarchically form the global, local, and semantic relevance scores. (3) Token weights are computed by normalizing these scores with positive/negative sample formulas. (4) The weights integrated into the standard DPO loss enhance alignment and mitigate hallucinations.
(1) The visual images are first projected
3 METHODOLLOGY
The overview of our proposed Cat-PO is illustrated in Fig. 2 We first introduce traditional DPO in Sec. [.I]] Then, we introduce the Hierarchical Visual Relevance of Tokens in Sec. 3.2 and
Token-rewards in Sec. 3.3 Lastly, describe novel Cat-PO Loss in Sec. 3.4
3.1 PRELIMINARIES: DIRECT PREFERENCE OPTIMIZATION (DPO)
DPO directly optimizes the model through a contrastive learning objective, making it more inclined to generate human-prefen preffered responses while reducing the probability of generating dispreferred responses. DPO learns from preference data (x. y ~ DD where x the input prompt, y the human-preferred /chosen response, y is the dispreferred /rejected response, and D is the dataset.
The DPO objective function originates from Bradley- Terry model, which assume that the human preference probability p" (y+ > .)) can be modeled via a latent reward function r" r" y): py > y z) = (x, r rz )). DPO further relates the reward function to the model's policy me and reference policy Teef T- y) ((log(TT (3) z)) Log(Te((y) z))).
where / a hyperparameter controlling the ratio between reward function and policy deviation. DPO's loss can directly optimize the model to maximize the probability of generating y* and minimize generating y For given preference pair (I, (xr,y,.)) y DPO loss function is defined as:
Looo - loo -lo (s (1(18g
=o(y= x:) Tref (g+
log To 2:) (x))
(1)
Tre (g
By minimizing this loss function, the model Ty is trained to increase the difference between the log-probate probabibliiies of preferred and dispreferred responses, relative to the reference model Tref- This direct method makes DPO simpler and demonstrates comparable or superior performance to RLHE.
3.2 HIERARCHICAL VISUAL RELEVANCE OF TOKENS
Without any external tools or techniques, we leverage the intrinsic multimodal capabilities of MLLMs, to hierarchically compute each token's 's global, local, and semantic relevance the visual input.
4

---

Published as a conference paper at ICLR 2026
3.2.1 CROSS-MODAL ATTENTION BASED GLOBAL RELEVANCE
When MLLMS process DPO training data within the Transformer architectures, the feature representa- tion of each token in a response interacts with visual features via a cross-modal attention mechanism. The activation distribution of these attention scores intuitively reflects the focus of specific text tokens on different image regions. Leveraging this, define and compute global relevance score for each token concerning the visual content, thereby quantifying its overall association with the image.
Specifically, for given image I and its corresponding preferred response yo rejected response yi (collectively denoted y) from the dataset, the representation the 1-th token yt in serves as the query. The set of N, visual token features, {v1, U2,,... ..., UN, "N,): derived from image / via visual encoder, acts as the keys and values. The sequence of cross-modal attention scores from token to all N, visual tokens is denoted by {ar,1, 0t.2, ......... Nt,N,,.
The global relevance (ytoha( is defined as the sum of the attention scores for token y::
Sglobal =>o at
higher Sylob((())) indicates that the model attends more intensively to the entire image when processing token yt, implying stronger global alignment between the token and the visual content.
3.2.2 PATCH ENTROPY BASED LOCAL RELEVANCE
Although the global relevance score Sgloba captures the overall association between response tokens and visual content, it does not reveal whether attention is concentrated on key regions or dispersed across the image. Typically, focused attention indicates a strong link specific local information, while dispersed attention suggests higher uncertainty or weaker visual grounding.
To accurately characterize the focusing pattern within this attention distribution, we leverage the con- cept of information entropy to compute the patch entropy scores for each token y/: based on its image attention distribution. First, we extract the cross-modal attention vector a [ar.1, at.2,...5 Nt,N,] for token Y/ with respect to all p image patches, where Ct,j represents the attention strength of y towards the j-th image patch. Next, we normalize the attention strengths in 0, to form a probability
distribution Pt = [Pr.1, Pr......, N.. J. and Pt,j at / Σn ar,k- We then compute the patch entropy (P:)) of this probability distribution. To ensure numerical stability in the logarithm, a small epsilon value (e.g., 10-12) is introduced:
N₂ Σ
=
PLs Log((L,))) +c)
This entropy value (P.)) measures the uncertainty or dispersion of the attention distribution. Sub- sequently, for Np > 1, we normalize the computed entropy H(P:), and the Patch Entropy Score, Sentrpy (yt), , is then defined as minus this normalized entropy:
H (Pt))
(yea(()))
(for N, 1)
log N₂
A higher Steat (Vr) score indicates lower entropy in the attention distribution, implying that attention is more sharply focused on a few image patches. This generally signifies a stronger degree of association between the token yt and specific local regions of the image.
3.2.3 CROSS-MODAT. SIMILARITY BASED SEMANTIC RELEVANCE
Beyond the global and local relevance, we exploit prior semantic signal obtained from a pretrained cross- modal encoder to quantify the semantic alignment between response tokens and visual content.
Given sample (I, v) let the embedding of the t-th token be e(gr). The image is divided into N, patches, each encoded as a visual feature (v1,..., NN, ). With the cross- attention weights
at.j (normalized over patches), we obtain a context-awarc visual vector: va(yi) = Σ at,j Vj The semantic relevance score is then defined as
5

---

Published as a conference paper at ICLR 2026
e(vr)) vo(3k)
Ssemantic - cos(e(ve), ve(uu))
(
[ )1 Ve
This score captures the semantic relevance hetween the token representation and the visual content of its most attended region, complementing the attention-basee global and local relevance.
3.3 TOKEN WEIGHTING SCHEME
Unified Visual Relevance Score: After obtaining hierarchical visual relevance scores for every response token token yi, we fuse them into a unified visual relevance score.
8 af 0.5 Sglobal. + 0.5 Stoeat. + (1 - a) Ssemanicc, a E (0, 1].
(6)
Here, a balances the attention branch (global & local) against the semantic branch.
Smooth Mapping to Token Weights: Directly injecting 8; into the loss may yield unstable gradients.
We therefore apply a smooth non- -in-lineaitt:: Т; = tanh(s;) E (0, 1), and introduce a base weight Aet > 0 to maintain controlled dynamic range:
Xvef + Tú
E y²
Aet (T),.), Yt E y.
This design (i) rewards tokens in the preferred response that strongly align with the image (T; t), and (ii) penalises hallucinated weakly aligned tokens in the dispreferred response ((1 T2) f).
3.4 WEIGITED INTEGRATION AND KL- EL-REGULARRSED CAT-PO LOSS
Incorporating token weights {wit, wT and token-level KL into the DPO loss yields the Cat-PO loss.
Weighted DPO. For preference pair (y+, ")) we weight the log ligeliidl ratio of the policy and the reference Tref- The weighted policy T
(w) is defined as T E, (w, log ( t 1 (w) w, log (v hn,)), and the weighted reference </1e) is defined as TTre = E.(w) (wi log A (v:" | htt - wi log Tre (Bi h )). Thus, the weighted DPO loss is defined as: Lwneo - log af (T( (w)
(8)
weightweighted KL regulariser. To keep the policy close to the reference and to stabilise training, with regularisation strength À > O, we add token-level. weight-modulater KL penalty:
LKI. > Σ ast KT(ToO. | (t) Tres(: (11 + Σ wi KT[[=o- (ht) Wref(: mr)]]
(9)
The final Cat-PO Loss objective is:
LCa.PO = LADPO + LKL.
(10)
Minimising equation 10 enables the policy model encode human preferences and fine- Ane-grained loken-vision alignments, suppressing hallucinations while preserving generation quality.
EXPERIMENTS
4.1 DATASETS AND METRICS
Training Data: Our experiments primarily employ the widely used RLHF-V dataset Yu et al. (2024). comprises 5,733 samples, each including an image, a question, a high-quality response, and relatively low-quality response. We use these data compute weights and train our model.
Evaluation Benchmarks: To comprehensively evaluate the model's performance in reducing hallu- cinations and enhancing general capabilities, employ several widely used benchmarks:
For hallucination evaluation, AMBER Wang et al. (2023) a LLM-free benchmark which consists of two main sub-tasks: (a) Discrimination: Determining whether a given statement about image

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on the Discrimination and Generative of AMBER Wang al. (2023), MM-Hal (Sun et al. (2023), LLaVA-Bench Liu et al. (2023b) and SEED Li et al. (2023a)
benchmarks. All methods are based on LLaVA-v1.5-77 and 13B Liu 2003b models with the
RLHE-V Yu et al. (2024) dataset, with the best results highlighted in bold.
AMBER-DAE D
AMBER-Gent
MM-Hal
Method
LLaVA SEED 1
Ace f
f
CHAIR +
Hal + Cog
Score
Rate
L.LVVAvv..--BB
71.7
74.3
7.8
36.4
4.2
2.01
61.4
65.6
66.1
+ DPO Rafailov et 2023
77.5
82.1
5.7
27.3
2.6
2.14
58.3
69.1
+ CSR Zhou ct al.
2024b)
73.2
76.1
5.4
25.5
2.6
2.05
60.4
68.9
65.9
+ POVID Zhou el (2024a)
71.9
74.7
5.7
26.9
3.0
2.26
55.2
68.2
66.1
RLHIF- V
Yu
2024)
74.8
78.5
5.5
26.3
2.5
2.02
60.4
68.0
66.1
+ V-DPO Xie et al.
81.6
27.3
2.7
2.16
+ TPO Gu ct al. (2024
79.3
85.0
2.47
51.0
70.2
66.6
+ Cat-PO (Ours)
78.0
85.3
4.8
23.7
2.1
2.74
42.0
70.3
67.0
LLLVA-v1..5-133
71.3
73.1
7.0
33.1
3.3
2.38
53.13
73.1
68.2
+ DPO Rafailov et al.
2023
77.5
82.1
6.1
26.3
2.7
2.47
51.04
72.8
68.6
TPO (Gu et al 2024
83.9
88.0
2.72
45.83
72.8
68.7
+ Cat-PO (Ours)
82.9
88.00
22.0
1.6
2.85
42.0
69.2
correct or not. (b) Generation: Generating a descriptive based on the image and question. MM-Hal (Snn et al. (2023) evaluates response-level hallucination rate and informativeness. It requires GPT-4 to compare model outputs with human responses and object labels for evaluation.
For general capability evaluation, ALaVe-Bench Liu et al. (2023b) is comprehensive benchmark that uses GPT-4 scoring to evaluate model generalization. SEED-Bench [i al.) (2023a) large-scale multimodal benchmark assessing visual understanding and text/image generation.
4.2 IMPLEMENTATION DETAILS
In experiments, leverage the widely adopted L. al. (2023a) and Qwen2.5- VL et al. (2025) series models to evaluate the scalability and effectiveness. The training of main experiment was performed over 6 epochs with effective batch size of 32, implemented through gradient accumulation. And the DPO hyperparameter 3DPO set to 0.1.
4.3 MAIN RESULTS
We compare Cat-PO with advanced preference alignment methods, which include: DPO Rafailov ct al. (2023). CSR, Zhou et al.) (2024b) A calibrated self-rewarding method. POVID, Zhou et al. (2024a) A GPT-4V based alignment method. RLHF-V, Yu et al. (2024) A method that segments human preference collection. TPO. Gu et al. (2024) A DPO variant employing self- self-calibrated, visually anchored rewards. V-DPO, Xie et (2024) vision-guides DPO variant.
In table I we evaluate existing methods and Cat-PO across mul- tiple benchmarks. On the AMBER- AMBER-Generation Generattion and MM-Hal, Cat-PO significantly improves response quality while effec- tively reducing hallucinations. On the AMBERRDiscriminanl it achieves competitive performance, highlighting its ability to evaluate image-related statements. Furthermore, on general Cat-PO
benchmarks such LLaVA- -enchhh and also remains outstanding.
-235
OMET URO
12.4
To verify the cross-model generalization of Cat-PO, we also conduct experiments on Qwen-2.5VL-33 Bai et al. (2025). As shown in Fig 3 Cat-PO achieves improvements Qwen and Qwen+DPO on MM-Hal and AMBER benchmarks, demon- strating its strong generalization ability.
Figure 3: Performance comparison of different Qwen2.5-VL models in terms of AMBER and MM-Hal Benchmarks.
7

---

Published as a conference paper at ICLR 2026
3.00
402
130
MM4-ral Score MS4-ril Rate
MK-ita Score
MK--fai Seore MM--tll Mate 196
MKK-ta Score 1
MK-rtl Rate 183
MKK-fal Rate (%0
Score
2.75
NW Foi Storo
Sate
Scowe
2.73
ree Pe zune
2.2
NVV- nb Tulio
2.50
2.33
Mnd Hot-/aa
3015
5.0
100%
0.03
(a)Weighted Token Ratio
(b]Alpha (a) Value
(c)A(KL) Value
(d)Data Category
Figure 4: The performance comparison of (a) different weighting proportions, (b)(c) important hyper-parameters AKL, and (d) weighting positions in our proposed Cat-PO framework.
4.4 ABLATION STUDY
Modules' Contribution. To further validate Cat-PO's effectiveness, conducted comprehensive ablation study in Table 2 The results show that cross- moosll attention relevance and semantic relevance play critical roles: weighting either alone improves performance, but their combination yields even greater gains. Moreover, removing the KL-based loss optimization causes a performance drop, confirming the necessity of the token-level KL. term.
Impact of Weighted-Tokert Proportion. We further investigate how varying the proportion of weighted tokens in the chosen responses affects Cat-PO performance. As shown in Figure (a), performance steadily improves with increasing weight proportion. However, applying weights to only the top 50% of tokens yields smaller gain than the top 30%, indicating that weighting the remaining 50% also provides notable contribution to overall performance.
Hyperparameter Analysis. We ex- amine two key hyperparameters in Cat-PO: (1) Balance Coefficient a.
Table 2: Performance of individual Cat-PO modules.
MM-Hal
AMBER-Grr Gene
Figure 4 (b) shows that both exces- sively large and small values of im- pair performance, underscoring the need to balance cross-modal atten- tion and semantic relevance; (2) KL- divergence Coefficient AKL- Figure 4 (c) demonstrates that ÀKL 0.03
Modules
Score f
Rate J
CHAIR |
Hal
DPO-only Attention-only Similarity-only Cat-PO without
2.14
58.3
5.7
27.3
2.34
55.0
5.3
25.9
2.51
47.0
5.1
27.7
2.36
53.0
26.9
Cat-PO (Ours)
2.74
42.0
4.8
23.7
achieves the optimal trade-off between maintaining model flexibility and constraining deviation from the reference distribution.
Chasen To Tolens Scores
Reject Takans Scores
Attention Scores Similee Scores Scores
1.0
Attertion Scores Similrity States
0.5
0.8
Scores
0.6
0.6
0.2
0.2
Please describa the image.
c2t
Ics
on
thc
floor
COt
Fut5
0n
thc
be3
Figure 5: Token level attention, similarity, and aggregated scores for a single example. The aggre- gated calculation lifts truly visual tokens (blue) while down-weighting hallucination tokens.
4.5 ANALYSIS EXPERIMENTS
The Effectiveness of Token- Meikhts Calculation. To further validate the effectiveness of our computed token weights, we applied weighting to either the top 30% or the bottom 30% of tokens. As shown in Figure weighting the top 30% of tokens yields significant improvement compared to weighting the bottom 30%. This result confirms the accuracy of our weight computation, and demonstrates that key tokens play decisive role in enhancing the alignment capability.
8

---

Published as a conference paper at ICLR 2026
Table
CatPO performance on overall and adversarial subsets of POPE Li et al.
Table 4: Comparison of Cat-PO (general) and Cat-PO (with learnable fusion).
2023cl.
Model
MM-Hal (1) Hal-Rate (L)
CatPO's Score
Acc.
Precision
F1
Cat-PO (general)
2.76
49%
Average
85.6
95.2
84.0
Cat-PO
Adversarial
(w/ learnable fusion)
2.55
50%
(most difficult)
84.0 (-2%) 91.3 (-4%)
82.5 (-2%)
The Analysis of Training logits. Training logs and visualizations further demonstrate its stability. We track the evolution of the training loss and reward margin throughout optimization. As shown in Fig- [6] the loss curve decreases smoothly, while the reward margin increases steadily without noticeable oscillations. The monitored gradient norms also remain stable. These observations indicate that Cat-PO maintains good optimization stability under token-level reward modulation.
Figure 6: Training dynamics of Cat-PO, showing smoothly decreasing loss and steadily increasing reward margin with stable gradient norms, indicating relatively stable optimization.
Evaluation on Edge-Case Diagnostics We additionally include POPE benchmark Li et al. (2023c) (with Object, Relation, and Adversarial subset). The Adversarial subset explicitly designed to elicit visual hallucinations and is widely regarded as the hardest visual alignment benchmark.
Thole that Cat-PO's s scores on the adversarial subset are slightly lower than average, indicating that our rewards remain relatively robust even under highly biased and adversarial edge-case settings.
Learnable Fusion Weights. We modified the fusion Eq. to be learnable parameters as follows:
5 * Sglobal, - Stocal, + - - 8) * Ssmmanti,
7, 8 E [0,
(11)
with ry and are the learnable parameters. Then, we jointly optimized them within training loss. Table show that introducing learnable coefficients yields the performance below the original one. Learnable fusion showed no benefit, possibly because (1) DPO does not directly supervise weight allocation, causing learnable coefficients overfit noise, and (2) Cat-PO mainly gains from the complementary moosl signals, making the fixed uniform weighting more stable design.
Robustness Analysis of Token-reward Calculation. To further assess robustness in edge scenarios, we visualize rare cases where attention and similarity disagree (Fig. 77. For "horse", misaligned attention is corrected by high semantic similarity, while for "train", an abnormally low similarity score is compensated by sharply focused attention. These examples show that Cat-PO's fused multi- signal scoring mutually compensates single-branch failures instead amplifying isolated alignment errors.
From theoretical perspective, Cat-PQ computes each token-level reward by fusing three complemen- tary cross-modal signals: global attention, local patch entropy, and cross-modal semantic similarity. A token receives high reward only when it simultancously exhibits strong attention strength, stable focus pattern, and high semantic consistency with visual content. This complementarity reduces reliance on any single noisy branch and structurally limits the misalignment.
Training Overhead Analysis. (1) Only one-time pre-computation: Pre-conputing token-level rewards for all positive and negative samples takes 2h16m188 18s. This cost is incurred only once and the results can be reused indefinitely. (2) We compared the training overhead Cat-DPO with DPO. Table shows that Cat-PO matches DPO across all metries, introducing only marginal overhead.

---

Published as a conference paper at ICLR 2026
Data idx 1520, RLHF-V Datset
Data idx 2290, RLHF-V Datset
Edge of Attention Misallgnment, but Similarity Corrects the Rewards
Edge Case of Similarity Misalignment,
but Attention Corrects the Rewards
Token = "horse" Attention Map x
Token = "traln" Attention Map
Attention (global) 0. 0.880065357861790887
Similarity (semantic) = 0.0488228554555427144
X
Similarity (semantic) = 0.7594918817754842
Attention (global) 0.2288050381188903557
Teken-Rewards 0.9820386664504800
Token-Rewards 0ttp:aadd=0011675000000000000 0.7167500575246255
Figure 7: Robustness Analysis in Corner Cases of Token-Reward Computation. Blue highlights indicate regions concentrated attention. All the data are from RLHF-V Dataset Yu al. (2024).
Left Part: When visual attention for the token "horse" is misaligned (rarely distributed in horses), the high semantic similarity effectively rectifies the final reward. Right Part: Conversely, when the semantic similarity for "train" is anomalously low, the focused attention distribution (correctly distributed in train) ensures a reasonable reward. This demonstrates that the fused metrics in Cat-PO mutually compensate for single-metric misalignments, ensuring reliable preference optimization.
4.6 CASE STUDY
Attention-Similarity Fusion for Token Scoring. Figure Siguureilllluraaaa attention scores (global and local), similarity scores, and their weighted sum for sample. Attention or similarity alone can distinguish visually critical from fact-violating tokens, validating each module's utility. However, biases exist: in the chosen sample, "floor" has low attention score, while "on" shows low similarity. But fusing the two signals ranks critical tokens higher, enabling precise token weighting in Cat-PO weighting strategy.
and further supporting
Comparison Cat-PO and DPO Generations. The comparison is pre- sented in Appendix A.6
Table 5: Training comparison DPO vs. Cat-PO: average processing time of per sample and the peak memory usage.
Model
Avg. time (8) Peak Memory Usage (GB)
DPO
2.1s
40.420
5 CONCLUSION
Cat-PO (Ours) 2.9s (+38%)
40.450 (+0.07%)
In this paper, we propose Cross-modal Adaptive Token-rewarded Preference Optimization (Cat-PO) for mitigating hallucinations and improving MLLM truthfulness. Each token's global, local, and semantic relevance is computed from cross- cross-modal modal attention and similarity, fused and incorporated into the DPO loss for fine- fine-grained optimization. Experiments public benchmarks show that Cat-PO effectively reduces hallucinations, improves response accuracy, enhances MILMSs truthfulness.
ACKNOWLEDGMENTS
This work was supported in part by the Artificial Nattiiaance-Nationlll Science and Technology Major Project under Grant 20237ID121200
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, al. Understanding alignment in multimodal llms: A comprehensive study. arXiv preprint ar arXiv:2407.02477, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Gc, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li. Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar Ar1:::2002. 13923, 2025.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXivv2404. 18930, 2024.
Assaf Ben-Kish. Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch-Elow Mocha: Multi-objective reinforcement mitigating caption hallucinations. arXiv preprint 1rXiv:23/2.03631, 2, 2023.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran, Dress: Instructing large vision-language models align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
14239- 14250, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, and See-Kiong Ng. Chip: Cross-modal hierarchical direct preference optimization for multimodal Ilms. arXiv preprint arXiv:2501.1662. 2025.
Jihao Gu, Yingyao Wang, Meng Cao, Pi Bu, Jun Song, Yancheng He, Shilong Li, and Bo Zheng Token preference optimization with self-calibrated visual-and aacul-anchored rewards for hallucination mitigation. arXiv preprint arXiv:2412.1 14487, 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entan- gled language hallucination and visual illusion in large vision-language models. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375-14385, 2024.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18135-18143, 2024.
Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and cntpppecttoonallocaatin on Computer Vision and Pattern Recognition, pp.
Proceedings of the IEEF/CVF Conference 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large languge models through visual contrastive decod- ing. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-1 3382, 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench. Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125. 2023a.
Lei Li, Zhihui Xie. Mukai Li, Shunian Chen, Peiyi Wang. Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312. 10665, 2023b.
11

---

Published as a conference paper at ICLR 2026
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination large vision-language models. arXiv preprint 0rXiv:2305.13333 2023c.
Zijing Liang, Yanjie Xu, Yifan Hong. Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp.
2024.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey hallucination in large vision- vision-language models. arXiv preprint ar 2tt::2402.00253,, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024b.
Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang. Runtao Liu, Rui Pan, and Tong Zhang. Strength- ening multimodal large language model with bootstrapped preference optimization. European Conference on Computer Vision, pp. 382-398. Springer, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7748-8763, PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-5377 53341, 2023.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Lia Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 2wXiv:2309.14555, 2023.
Fei Wang. Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406. 1 1139, 2024.
Junyang Wang. Yuhang Wang. Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free nulti-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.07399 2023.
Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang. and Linchao Zhu. Detecting and mitigating hallucination in large vision language models via fine teeeggraiedd ai feedback. arXiv preprint arXiv:2404. 14233, 2024.
Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. Detecting and mitigating hallucination in large vision language models via fine- grained ai feedback. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 25543-25551, 2025.
Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. arXiv preprint arXiv:2411.02712 2024.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- -rained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024.
12

---

Published as a conference paper at ICLR 2026
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprint WxXi:2244.11199 2024.
Kejia Zhang, Keda Tao, Zhiming Luo, Chang Liu, Jiasheng Tang, and Huan Wang. Tars: Minmax token-adaptive preference strategy for hallucination reduction in milmss arXiv e-prints, pp. arXiv- 2507, 2025.
Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng. Jingdong Wang. et Automated multi-level preference for milms. Advances in Neural Information Processing Systems, 37:26171-26194 2024.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lylms through hallucination-awarr direct preference optimization. arXiv preprint arXiv:2311.168. 2023.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning, arXiv preprint arXiv:2402. 11411, 2024a.
Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang. Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622 2024b.
Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang, Self-supervised visual preference alignment. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 291-300, 2024.
Mingkang Zhu, Xi Chen, Zhongdao Wang, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Tgdpo: Harnessing token-level reward guidance for enhancing direct preference optimization. arXiv preprint arXiv:2506. 145744 2025.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 USAGE OF LARGE LANGUAGE MODELS IN MANUSCRIPT PREPARATION
To ensure the clarity, fluency, and grammatical accuracy of the manuscript, large language models (LLMs) were employed during the preparation process, specifically for two core tasks: text polishing and grammatical егтог correction. In terms of text polishing. LLMs were used to optimize the expression of technical content (including experimental descriptions, result analyses, and discus- sion sections) - this involved refining sentence structure to enhance logical coherence, adjusting terminology consistency to align with academic conventions the field of machine learning (e.g.. standardizing the naming of "Cat-PO model" and "baseline DPO model" throughout the text), and improving the readability of complex statistical interpretations. For grammatical error correction, LLMs assisted in identifying and rectifying potential issues in English expression (including subject- verb agreement, tense consistency, and preposition usage) to eliminate language-relatee ambiguities that might affect the understanding of research findings. It is important to note that all core research content - including experimental design, data collection, model training processes, statistical analy- ses, and key conclusions remained independently completed by the authors, and LLMs were only used as auxiliary tools for language optimization without altering any substantive research content.
A.2 LIMITATIONS AND FUTURE WORK
Our approach relies solely on the intrinsic multi-moda capabilities of MLLMs, without external tools or models. This work primarily validates effectiveness and provides qualitative analysis. Future work will systematically measure resource consumption, expand evaluation metrics, and verify the resource savings from our eliminating external dependencies.
A.3 ADDITIONAL RELATED WORKS.
In more fine-grained DPO explorations. CHiP Fu et al. (2255) introduces visual preference optimization together with hierarchical preference optimization at the response, segment, and token levels on the text side. The token-level component is primarily based sequence-level KL divergence derived from text probability distributions; is not image-aware and functions only as an auxiliary term in the text-side loss. TARS Zhang et al. (2025) reformulates DPO as min-max game. TARS introduces adaptive perturbations on vision-irrelevant tokens to induce controlled distribution shifts and combines them with a frequency-doman improvements in visual grounding and robustness under very low data costs. AMP ZZang et al. (2024) leverages multi-level preferences to construct Aner-grained preference orderings, effectively reducing the quality gap between positive and negative samples. It focuses on response-level alignment by incorporating multiple preference levels.
regularization constraint, achieving substantial
In pure-text LLMs. TGDPO Zhu et al. 2025 introduces reward-guided DPO framework that decomposes sequence-level PPO into loken-level subproblems and theoretically proves the indepen- dence of the partition function. This enables the integration of fine-grained token-level rewards into the DPO objective, improving instruction-folloiing performance and training stability. TDPO Zeng et al. (2024) further decomposes sentence-level rewards into token-level rewards via the Bellman equation and introduces a Sequential Forward KL constraint. Through token-level optimization, TDPO enhances alignment performance in text-only generation.
Other related works. Hallucinations in MILMMs from several factors: training data deficiencies/ Li et al. (223c);; module-specific issues within MLLMs' separately trained components Guan al. (2024): y suboptimal training paradigms Ben- Kish et al. (202);; and inference-stage defects.
A.4 TOKEN-LEVEL LOG- LOG-PROBABILITISS ON CHOSEN RESPONSES: CAT-PO vs. DPO.
To illustrate Cat-PO's advantage in boosting generation confidence, in Figure we present token- level oog-prob case studies. Cat-PO (blue curve) assigns higher log rog-probabilittes to most tokens in chosen responses compared to DPO (green curve), shown by the consistently positive differences (yellow bars). It suggests that Cat-PO better identifies and reinforces key semantic tokens, producing answers that are both accurate and confidently grounded.
14

---

Published as a conference paper at ICLR 2026
Values
UP0 Logmut
Logrud
PO (:32.49)
Difference 1 CHPO DPD
key
features you observe in image
lakens
Dug DPO LAMPUR 1-2.3D
LiTerrice CIsPO Laghtss
How many umbrallas this picture?
Figure Log-Probabile (log-prob) comparison between Cat-PO and DPO on chosen responses. The blue (Cat-PO) and green curve (DPO) represent the log-prob. The yellow bars represent differences, computed as Cat-PO minus DPO. This indicates that Cat-PO not only learns a stronger preference for chosen responses but also generates them with greater confidence.
Improvement Metrics
102
97.62%
Perentage
20
Tonans
Figure 9: Improvement metrics for the model trained with Cat-PO in enhancing positive sample responses. The left bar indicates that, under the Cat-PO model, average of 97.66%6% of tokens within each positive sample response experienced a positive increase in their log-probability. The right bar shows that 99.98% of the samples in the dataset demonstrated net growth in total probbbbbbyylityy under the Cat-PO model.
A.5 QUANTITATIVE EVALUATION: SUPERIORITY OF CAT-PO IN BOOSTING TOKEN-LEVEL LOG-PRORABIILTTES
Our supplementary statistical analysis further quantifies the significant effect of the Cat-PO model in enhancing the generation confidence of positive responses compared to the baseline DPO model. The distributional statistics, illustrated in Figure reveal the widespread positive impact of Cat-PO: on average, remarkable 97% tokens within each positive sample experienced a positive increase in their pro-bbobaa Furthermore, at the overall sample level, 99.98% of the samples in the dataset demonstrated net growth in total log- probaabibtty (sum log- probabiiiyy improvements across all tokens) under the Cat-PO model. These figures robustly demonstrate that the Cat-PO method is not only highly effective but also that its advantages are broadly applicable, covering almost all test data and the vast majority of generated content.
15

---

Published as a conference paper at ICLR 2026
Regarding the magnitude of improvement, statistical results equally encouraging (this can be linked to your second figure if shows magnitude, or stated directly). Specifically, the Cat-PO model achieved
average log- pogorobbbillit increase of 7.911821 per token in positive responses. When converted to a relative percentage, the average percentage increase in log-probability per token reached 29. 99.... This substantial level of improvement, especially in the log-space, signifies that the Cat-PO model exhibits far greater "certainty" than the baseline model when generating key parts of these preferred responses.
A.6 COMPARISON OF GENERATION IN CAT-PO AND DPO.
Our comparative analysis (in Figure 10 illustrates the performance differences between Cat-PO and DPO across four distinct examples. The figure contains four sub-examples demonstrating Cat-PO's consistent advantage in generating more detailed and accurate descriptions, partieularly in identifying subject counts, positional relationships, and contextual details.
16

---

Published as a conference paper at ICLR 2026
likely woman, riding horse beach The rider is wearing a helmet for safety while enjoying the ride. The horse of motion and speed the The beach setting with the ocean in the background creates serene and picturesque backdrop for equestrian activity.
The image captures
DPO
gray colored animal, adding
Examplel
Our Cat-PO
The image features person riding a horse on beach, with the horse running along the shoreline. The person wearing black iacket and and they are skillfully riding the horse. The horse is positioned towards the center of the image, with the person picturesque and serene atmosphere for the rider and the horse.
Question: Describe this image.
back. The beach setting provides
The image features person riding skateboard beach The skateboarder the main subject of the scene, they navigate the sandy terrain while maintaining their balance the skateboard. The backdrop consists sky with clouds, creating serene and open atmosphere.
DPO
Example2
Our Cat-PO
The image features
riding skateboard down sandy hill, enjoying the thrill of the descent. The skateboarder is positioned in the middle of the with the skateboard visible beneath him. The hill appears sandy dune, and the man seems to be skillfully navigating the terrain, The sky the background blue, adding the overall atmosphere the scene.
Question: Describe this image.
The image features two dogs of different colors walking on sandy beach. They appear be enjoying their time outdooor, possibly exploring or taking a stroll along the shore. The beach setting and the presence of the dogs
DPO
sense leisure and freedoom, they freely in this natural environment.
Examplel
Our Cat-PO
The image features sandy beach with three doas walking together. The dogs are different colors, with one being black and white, another brown and white, the third being brown. They walking the black and white dog leading the way. followed by the brown and white dog, and then brown dog. The dogs appear be enjoying their time the beach, possibly exploring or playing.
line,
Question: Describe this image.
The image features two children. a young boy and young girl, playing in crassy field sunny day. They appear to be enjoying their time outdoors, possibly exploring or having fun together. The green grass and the sunlight create pleasant atmosphere for the children engage in their play
DPO
Example2
activities.
The image features lush green field with three children plaving together. Two the children are standing. while the child sitting the grass The two standing children are positioned closer to the left side of the image, while sitting child is the right sidé, the scene, there is toy visible, likely toy car, placed near the left side of the image. The children seem to be enjoying their time outdoors, playing and having fun in the grassy field.
Our Cat-PO
Question: Describe this image.
Figure 10: Four comparative examples showing generation differences between DPO and our Cat-PO. (1) Beach horse riding: Cat-PO provides specific details about rider attire and horse movement, (2) Sand skateboarding: Cat-PO adds contextual information about terrain and activity. (3) Beach dogs: Cat-PO correctly identifies three dogs with distinct color patterns. (4) Children playing: Cat-PO notes precise subject count, positions, and presence of a toy.
17


Reviews:
[{"review_id": "g8XaBsH4ky", "reviewer": "Reviewer_e8nW", "summary": "This paper tackles the persistent issue of hallucinations in multimodal large language models (MLLMs) by introducing Cross-modal Adaptive Token-rewarded Preference Optimization (Cat-PO). \nThe authors observe that existing preference optimization methods, such as DPO, fail to account for varying visual relevance among response tokens. Cat-PO addresses this by assigning hierarchical, token-level rewards based on global, local, and semantic visual relevance, integrated into a KL-regularized DPO framework. \nExperiments on multiple benchmarks show that Cat-PO substantially reduces hallucinations and enhances factual alignment, with extensive ablation studies validating its contributions.", "questions": "1. Learnable Weighting: Would learning the fusion coefficients in Equation 7 improve adaptability or stability?\n2. Comparative Evaluation: How does Cat-PO compare empirically or theoretically with TARS[1], CHiP[2] and AMP[3]?\n\n[1] TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs\n[2] CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs \n[3] Automated multi-level preference for mllms", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Fine-grained Token-level Rewarding: The core innovation lies in Cat-PO’s token-level reward granularity. By assessing each token’s global, local, and semantic relation to the input image, the model updates its preferences more precisely than aggregate reward schemes.\n\n2. No External Dependencies: Cat-PO requires no external detectors or APIs. It fully leverages the pretrained MLLM’s own components (e.g., CLIP+ViT and the base LLM), ensuring simplicity and low overhead.", "weaknesses": "1.Generalization Claims: Although Cat-PO is presented as a general framework, experiments are confined to LLaVA, with no tests across different architectures or open-domain datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bgBHstieU", "reviewer": "Reviewer_k6vM", "summary": "This paper proposes Cat-PO, which introduces a token-based hierarchical reward mechanism on the foundation of standard DPO to achieve more refined hallucination suppression. It fully leverages the MLLM's own architecture (without requiring external tools) to compute the relevance of each response token to the image from three perspectives: global relevance, local relevance, and semantic relevance. These three scores are integrated into a unified relevance score, allowing tokens with strong visual relevance to receive higher rewards in the chosen response and higher penalties in the rejected response. Finally, the authors demonstrate the effectiveness of their approach through a series of experiments.", "questions": "1. The current method is a black-box \"automatic scoring\" approach but does not provide any interpretability analysis. Could it be that high-frequency words such as \"the\" or \"a\" receive excessive rewards due to coincidental high attention scores?\n\n2. The experiments may involve unfair comparisons. For instance, Cat-PO introduces multiple hyperparameters, which might have been meticulously fine-tuned, whereas hyperparameters of other methods may not have received the same level of adjustment.\n\n3. How does the computational cost compare to standard DPO?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors present a clear motivation for their method, with a well-justified problem statement. It effectively addresses the issue of token-level fine-grained alignment in DPO, thereby enhancing its efficiency.\n\n2. The experiments are sufficiently comprehensive, evaluating the method from both hallucination suppression and general capabilities. The ablation studies are also thorough.\n\n3. The paper is clearly written and easy to understand.", "weaknesses": "**My primary concern regarding this work is the potential risk of circular reasoning in its methodological motivation.** If an MLLM has already incorrectly allocated high attention to a hallucinated object (e.g., misidentifying a chair as a dog and focusing on that region), Cat-PO may mistakenly judge the corresponding tokens as \"highly relevant.\" This would inadvertently reinforce the incorrect generation, potentially cementing or even amplifying the erroneous attention pattern. Consequently, rather than mitigating hallucinations, this approach could exacerbate specific types of them. I would raise my score if the authors could address this issue, either through a theoretical discussion or via visualizations that demonstrate how their method remains robust in such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tEtK5CnVjm", "reviewer": "Reviewer_vajx", "summary": "This paper proposes CAT-PO (Cross-modal Adaptive Token-rewards for Preference Optimization), a token-level preference optimization method for multimodal LLMs aimed at reducing visual hallucinations while improving truthfulness. Building on DPO, the authors compute hierarchical token relevance from model-internal signals: (i) global relevance as the sum of cross-modal attention over all visual patches; (ii) local relevance via patch-entropy (lower entropy ⇒ more localized visual focus); and (iii) semantic relevance using a cosine similarity between token and vision-conditioned text/visual representations. These three signals are fused (with a smooth nonlinearity) into per-token rewards that modulate DPO and a token-level KL regularizer (reward-aware KL) to discourage visually ungrounded continuations.\nExperiments on AMBER (disc/gene), MM-Hal, LLaVA-Bench, and SEED primarily with LLaVA-v1.5-7B/13B and Qwen variants show consistent improvements, notably 7–15% gains on AMBER-Gen/MM-Hal and reduced hallucination rates. The approach adds modest overhead (+38% per-sample time; ~+0.07% peak memory).", "questions": "1. **Learned fusion and fixed fusion**. Did you try a learned gate like 2-layer MLP over [S_global, S_local, S_sem] or a Dirichlet-style mixture with simplex constraints? How do gains compare and how stable is training?\n\n2. Several highly relevant works are not covered or discussed in the current manuscript.\n\n[1]Token-level Direct Preference Optimization.\n[2]CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs\n[3]TGDPO / T-Guided DPO: Harnessing Token-Level Reward Guidance for Preference Optimization", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Methodological clarity**. Token-level reward design from internal signals (attention, entropy, similarity) that are cheap to obtain.\n2. **Consistent empirical gains**. 7–15% improvements on AMBER-Gen/MM-Hal, and competitive on AMBER-Disc。", "weaknesses": "1. **Heuristic fusion lacks learning/theory**: The tanh-based smoothed fusion and fixed mixing coefficients are plausible but not justified.\n2. **Limited evaluation**: While standard suites are covered, edge-case diagnostics are missing.\n3. **Reward-gradient stability not analyzed**. No examination of gradient variance when many tokens get high rewards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)