Consistent Noisy Latent Rewards for Trajectory Preference Optimization in Diffusion Models
Authors: Xiaole Xian
Xilin He
Wenting Chen
Wenshuang Liu
wenqi mu
Yancheng He
Liang Li
Yi Zhang
Xiangyu Yue
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Model
Preference Alignment
Text-to-Image
Text-to-Video
Primary area: generative models

Abstract: Recent advances in diffusion models for visual generation have sparked interest in human preference alignment, similar to developments in Large Language Models. While reward model (RM) based approaches enable trajectory-aware optimization by evaluating intermediate timesteps, they face two critical challenges: unreliable reward estimation on noisy latents due to pixel-level models' sensitivity to noise interference, and single-timestep preference evaluation across sampling trajectories where single-timestep evaluations can yield inconsistent preference rankings depending on the selected timestep.
To address these limitations, we propose a comprehensive framework with targeted solutions for each challenge. To achieve noise compatibility for reliable reward estimation, we introduce the Score-based Latent Reward Model (SLRM), which leverages the complete diffusion model as a preference discriminator with learnable task tokens and a score enhancement mechanism that explicitly preserves noise compatibility by augmenting preference logits with the denoising score function. To ensure consistent preference evaluation across trajectories, we develop Trajectory Advantages Preference Optimization (TAPO), which strategically performs Stochastic Differential Equations sampling and reward evaluation at multiple timesteps to dynamically capture trajectory advantages while identifying preference inconsistencies and preventing erroneous trajectory selection.
Extensive experiments on Text-to-Image and Text-to-Video generation tasks demonstrate significant improvements on noisy latent evaluation and alignment performance.

Published as a conference paper at ICLR 2026
CONSISTENT NOISY LATENT REWARDS FOR TRA- JECTORY PREFERENCE OPTIMIZATION IN DIFFUSION MODELS
Xiaole Xian :22
Xilin He
Wenting Chen
Wenshuang Liu²
Wenqi Mu²
Yancheng He
Liang Li
Yi Zhang"
Xiangyu Yue
1MMLab, CUHK
Tencent
CHtyUHK
*MBZUAI
Jiutian, China Mobile
ABSTRACT
Recent advances in diffusion models for visual generation have sparked interest human preference alignment, similar to developments in Large Language Models. While reward model (RM) based approaches enable trajectory- aware optimization by evaluating intermediate timesteps, they face two critical challenges: unreliable reward estimation on noisy latents due to pixel-level models' sensitivity to noise interference, and single-timestep preference evaluation across sampling trajec- tories where single-timestep evaluations can yield inconsistent preference rank- ings depending on the selected timestep. address these limitations, we propose a comprehensive framework with targeted solutions for each challenge. To achieve noise compatibility for reliable reward estimation, we introduce the Score- Latent Reward Model (SLRM), which leverages the complete diffusion model a preference discriminator with learnable task tokens and a score enhancement mechanism that explicitly preserves noise compatibility by augmenting prefer- ence logits with the denoising score function. To ensure consistent preference evaluation across trajectories, we develop Trajectory Advantages Preference Op- timization (TAPO), which strategically performs Stochastic Differential Equations sampling and reward evaluation at multiple timesteps to dynamically capture tra- jectory advantages while identifying preference inconsistencies and preventing erroneous trajectory selection. Extensive experiments on Text-to-Image and Text- to vo-Vidoo generation tasks demonstrate significant improvements on noisy latent evaluation and alignment performance. The code is available TAPO.
INTRODUCTION
Inspired by Reinforcement Learning from Human Feedback (RLHF) advancements in Large Lan- guage Models (SLMs)Schullmn et al., 2017; Rafailov et al., 2023; Shao et al., 2024) and diffu- sion models' success in visual generation(Nichol et al., 2021; Rombach al., 2022), numerous works (Clark et al., 2023; Fan et al., 2023; Black et al., 2023) have emerged for human prefer- ence alignment in diffusion models. These methods fall into families: offline data approaches that learn from human-annotated pairs (Wallace et al., 2024), and rewarl-model (RM) approaches that enable online preference optimization by scoring candidates during training (Liu et al., 2025c; Wang et al., 2025; Liang et al., 2025). While offline methods improve final outputs, they only evaluate clean and fully denoised images, ignoring the noisy latents along the sampling trajectory. This limits their ability to support trajectory- aware optimization. In contrast, RM-based methods have gained traction for their ability to evaluate intermediate timesteps and enable such optimiza- tion (Liang et al., 2025; Zhang et al., 2025).
Although RM-hased methods can provide preference evaluation for intermediate timesteps, they face two primary challenges. The first challenge concerns unreliable reward estimation on noisy latents. Most existing methods (Liu et al., 2025d; Xu et al., 2024a) adapt Eision-Language Models (VLMs) as reward models, which could be sensitive to pixel perturbation and lack principled noise
Corresponding author.

---

Published as a conference paper at ICLR 2026
'A wuttrco poirting of suber oute li weating a hat with mony flowors'
Preforence Optimization
Diffusion Model
AA robet Standing on the wandr, surrounted by butbies and ruboer ducks
Preference S(x7.n) S
ARA Reward Model
Inconsistent 5.
LPO
TAPO
TAPO
TAPO (n=8)
Figure I: (a) Inconsistent reward signals evaluated from different timesteps could lead to incor- rect trajectory preference ordering, disrupting the training in existing DPO-style diffusion model optimization. Specific examples are shown in Appendix C.5. (b) Superior alignment through Trajectory Advantages. Unlike existing methods ge. LPO) that rely on single-step noisy latent evaluation. TAPO leverages multi-steps (n) advantage across the entire sampling trajectory to obtain higher quality training samples, thereby achieving optimal performance.
compatibility. To enhance noise compatibility, recent efforts Liang al. (2025) inject noise the visual encoder inputs, with LPO Zhang et al. (2025) further leveraging the diffusion backbone to incorporate stronger noise-aware priors. However, LPO overlooks that the diffusion model's noise compatibility fundamentally stems from learning score function over the data distribution, which a fundamentally different objective from reward modeling. Therefore, when these backbones are fine-tuned for preference discrimination, their score-learning properties degrade, resulting in poor performance on noisy latents and unreliable reward predictions.
Another challenge involves single timestep preference evaluation. As illustrated in Fig. 1 (a), when comparing noisy latents from two samples, the reward model may prefer the second sam- ple at intermediate timestep (S(x!") < S(x()) while preferring the first sample at the other timestep (S(x))) > S(x(-1)). Current RM-based methods (Liu et al., 2025d; Xu et al., 2024a) typically evaluate preferences at only specific timestep, failing to consider the full temporal context of the sampling trajectory. Although recent work (Yang et al., 2024b) has explored dense re- wards along the trajectory, it redistributes single clean-image preference evaluation all timesteps. This single-timestep evaluation can yield inconsistent outcomes depending on which timestep is se- lected, leading to erroneous preference rankings where trajectories with high intermediate rewards but suboptimal final outputs are favored. Thus, considering trajectory-Ievel rewards for preference evaluation crucial to preventing misleading trajectory selection during training.
To address these challenges, we propose a comprehensive framework for human preference align- ment applicable to diffusion-based models, encompassing both reward model training and the on- line sampling strategy during the alignment phase. First, to achieve noise compatibility in noisy latents, we introduce a Score-based Latent Reward Model (SLRM), which leverages the complete diffusion model as preference discriminator. SLRM introduces learnable task tokens, leveraging self-attention for adaptive, multi-layer aggregation of fine- ane-grained visual and textual features (Pee- bles& & Xie, 2023). Crucially, we incorporate score enhancement mechanism that explicitly pre- serves the model's noise compatibility by augmenting the preference logits with the denoising score function. This design ensures that SLRM maintains stable and accurate discriminative evaluations throughout all timesteps. Building on this noise-compatible reward model, we propose Trajectory Advantages Preference Optimization (TAPO) establish trajectory-level preference evaluation and ensure consistent preference rankings. TAPO strategically performs Stochastic Differential Equations (SDE) sampling for stochastic exploration (Song al., 2020) and reward evaluation at selected multi- timestissssess. retaining the best and worst samples at each evaluation step based on the reward. This dynamically captures trajectory advantages while avoiding the computational over- head of exhaustive evaluation. Moreover, in Fig. 1 (b), it efficiently identifies win-lose sampling data with pronounced quality differences, yielding high-quality training data for more accurate pref- erence alignment. Extensive experiments on Text-to-Iman Image (T2I) and Text-to-Vides (T2V) genera- tion show our method substantially improves visual generation quality. SLRM achieves significant accuracy improvements across all sampling timesteps. During alignment, TAPO demonstrates sub- stantial generation quality improvements when applied to Stable Diffusion-3.5 (T2I) (Esser et al., 2024) and 1 1 (T2V) models (Wan et al., 2025).
2

---

Published as a conference paper at ICLR 2026
Our contributions are as follows:
We introduce SLRM, score-based noisy latent reward model that leverages the diffusion model's inherent score function to maintain noise compatibility throughout all timesteps, addressing the critical limitation of existing pixel-level reward models in evaluating inter- mediate noisy latents.
We propose the TAPO that strategically performs SDE sampling and reward evaluation at multiple timesteps, dynamically capturing the trajectory advantages to generate high- quality training data with pronounced preference differences.
Through extensive experiments across two generation tasks (T2I and T2V), we demonstrate the significant improvements in generation quality, establishing the broad applicability of our approach to diverse diffusion based models.
RELATED WORK
Human Preference Alignment for Diffuusion Models. Motivated by RLHF's success in LLMs (Schulman et al., 2017; Achiam et al., 2023; Shao et al., 2024; Chen et al., 2024), extensive research has explored preference alignment for diffusion models (Rombach al., 2022; Nichol al., 2021; Ramesh et al., 2021; Saharia et al., 2022). These approaches fall into two categories: offline data methods and reward-model (RM) based methods. Early RM-based approaches used PPO-based policy gradients (Fan et al., 2023; Black et al., 2023), formulating denoising as a Markov decision process, while reward-driven fine-tuning methods (Li et al., 2025a; Lee et al., 2025; Xu et al., 2023; Wu et al., 2023; Ma et al., 2025) directly optimize diffusion models to maximize reward signals. However, these methods suffer from reward hacking and expensive computational costs. Following Diffusion-DPO (Wallace al., 2024), which adapted DPO (Rafailov et al., 2023) to diffusion mod- els, subsequent works (Liu et al., 2025d;e; Wang et al., 2024a; Zhang et al., 2024a; Lu et al., 2025; Wu al., 2025) have advanced preference alignment through implicit reward modeling. While offline methods (Wallace et al., 2024) improve outputs by learning from they evaluate only clean images and cannot assess noisy latents along sampling trajectories. Con- sequently. RMM-ased methods are increasingly adopted for trajectory-aware optimization (Liang al., 2025; Zhang et al., 2025), enabling online preference optimization by scoring intermediate timesteps. However, these methods face challenges with unreliable rewards on noisy latents and inconsistent preference evaluation across trajectories.
pairs,
Reward Model for Preference Optimization. Reward models provide crucial feedback signals for preference-based optimization in generative model alignment. Early approaches leverage pre- trained VLMs like CLIP (Radford et al., 2021) and BLIP (Li al., 2022) for zero-shot evaluation, or employ fine-tuned models (Xu et al., 2023; Wu et al., 2023; Ma et al., 2025; Kirstain et al., 2023; Zhang al., 2024b; Li et al., 2025a) for aesthetic and preference assessment. Recently, L.-M-based reward models (Liu et al., 2025c; Wang et al., 2024b; Xu et al., 2024b) have emerged, leveraging MLLMs' contextual understanding for alignment evaluation. However, pixel-level reward models face limitations when evaluating intermediate noisy latents during denoising, as most existing meth- ods (Liu al., 2025d: Xu et al., 2024a) could be sensitive pixel variations and unable handle noise effectively. Recent attempts (Liang et al., 2025; Zhang et al., 2025) train latent reward models on simulated noisy inputs but suffer from noise compatibility degradation due to insufficient under- standing of the diffusion model's score function (Song et al., 2020). These limitations motivate our SLRM proposal, which directly incorporates the denoising score function to maintain noise com- patibility. Furthermore, current methods determine the advantages of the whole sampling trajectory based on a single point-wise timestep, neglecting the temporal context from the global trajectory- level perspective. Thereby, we further propose TAPO consider reward signals from all timesteps when determining the win-lose trajectory.
3 METHODOLOGY
In Fig. 2, we present comprehensive two-stage framework for human preference alignment. First, we propose the Score-based Latent Reward Model (SLRM) in Sec. 3.2 for stable preference discrim- ination across all timesteps. Subsequently, building upon this noise-compatible reward model,
3

---

Published as a conference paper at ICLR 2026
Ditfusion Model Backbone
Denoising Score Enhancement
Windows Profermne Thtose
MM-DIT Block
MM-DIIT Block
MM- DTT Block
MM-DIT Block
Projector
S
0.27
1.38
beoch
eni
Barij
Profictio Spane
(a) Score-based Latent Reward Model (SLRM)
A
SDE Samplin
Saaiing KTS
S.RM
ATE
M
AA
SDE Sempling: Single Step
Selecting
Samplin: Mutti-Steps
SDE Select
(b) Trajectory Advantage Preference Optimization (TAPO)
Figure 2: (a) Training Pipeline of the Score-based Latent Reward Model (SLRM). (b) TAPO sam- pling process. specific timesteps, win and lose latents adapt SDE sampling to get the latents group respectively.
introduce Trajectory Advantages Preference Optimization (TAPO) in Sec. 3.3. It leverages SLRM to evaluate samples along sampling trajectories, efficiently identifying win-lose pairs with pronounced preference differences.
3.1 PRELIMINARY
Flow Matching, Suppose that Zo Xg is a data sample from target distribution and Φ X1 denotes the source distribution. Recent advanced diffusion models adopt the flow matching (Lipman al., 2022) generate Zg starting from F1- Specifically, the flow matching framework defines continuous-time normalizing flow through ordinary differential equation (ODE):
dæ; = v,dt
(1)
where the linear conditional flow defines the æ, (1 t)æ, tæ The core of these methods is to train a neural network 12.0 to satisfy the velocity field by minimizing the Flow Matching objective:
LFM = Eeejo.1). (lx - vo (x:)))??
where the velocity field given by (x = 21 Ty-
Preference Optimization for Diffusion Models. Diffusion- -DO (Wallace al., 2024) extends DPO (Rafailov et al., 2023) to diffusion models by propagating preference orders from clean images (x:) ag)) to latents in intermediate denoising steps (a:"" .².). However, the preference orders may be inconsistent along the all the timesteps, which has motivated subsequent work to directly evaluate the preference orders of latents in intermediate steps (xx" 2z). Accordingly, the optimization objective of Diffuuion- DPO is reformulated as a step-by-step preference optimization (SPO):
po (=(x (1+1. Blog pe po(22/0441,10) 0)
LSPO = -Eg aa upo (wo, (0) log
3 log Pref (x 2)+1, (re (ee11.)
(3)
where e is the input condition, 33 a regularization hyperparameter, and pa and Pref denote the optimized and reference model, respectively.
3.2 SCORE-BASED LATENT REWARD MODEL
In this section, we provide a detailed presentation of the overall Score- based Latent Reward Model (SLRM), including the architecture, loss and the training process.
4

---

Published as a conference paper at ICLR 2026
Architecture Design. To inherit the diffusion model's capability of processing noisy inputs, SLRM directly initialized the complete pre-trained diffusion model as the backbone, as shown in Fig. 3. Previous approaches that compute scores using separate visual and text encoders:
S(x .c) < - (x=) (c)
(Iviaa(**)) (x*))2 [c)xx
{w,()
(4)
where the EVis and Etst denote visual and text encoders respectively, and w, represent winning or losing inputs. However, computing preference scores based on visual-textual similarity primarily measures text-image alignment rather than comprehensive quality aspects like fine-grained details and aesthetics.
Thus, we introduce specific task to- kens (Xu et al., 2024b) to partici- pate in the model's 's self-attention pro- cess to capture comprehensive qual- ity aspects beyond simple text-image alignment.
Nokey Image
Text
Clean Image, Text
Nolky Image, Text
Enooder (OLEP utell
Text Enceser
Larguage Model
Dificiin Model
Text Tachur
Hisden Stote
Reverd Score
benatting Scone
Coana
Ragrnssion
Specifically, randomly initialize these tokens as new embeddings E RR.x* of the text encoder. Subse- quently, these task tokens are con- catenated with text and visual tokens to pass through the diffusion model participate its self-attention, and the attention features can be modified as follows:
Sintarlly
Reward
Resarrd
Reward
Prinarile Test Ienagenggeeeee
Comprchenaive Evalaation
Compeehenier Evsitontin
Compasiiliy Dograde
Exconyasiiityy
Compurbiltty
Existing Noisy
Reward
SLRM Reward
Figure 3: Comparison of Different Reward Model Architec-
tures. SLRM maintains robust noise compatibility while enabling comprehensive evaluation through denoising score enhancement.
Q Po(at,) @ (W((a)) @ (b(())))
K (1 ((((a))) P (v(sa))
(5)
V = (xt,)) o (V(C.)) o (bb(s)))
where the Pk. PK.PPPPp and PQ. PT are the pre-trained linear projections for image and text Ps. P. are the score linear projections. After processing through layers of the embeddings, PS.PS Sr,
MM-DiT blocks (Esser et al., 2024) in diffusion model, we obtain st, which is then mapped to the vanilla reward score via a linear layer:
Through the self-attention mechanism in these DiT blocks, these task tokens adaptively select and aggregate and fuse the multi-level semantic representations in visual and textual features.
Denoising Score Enhancement.
Although our model initially inherits timestep-aware capability from the pre-trained diffusion model, observe critical issue: the model's ability discriminate preferences noisy latents actually degrades as training, as illustrated in Fig. 4.
This degradation occurs because dif- fusion models' noise compatibility stems from their original training ob- jective of learning score functions (Song et al., 2020) across different noise levels. However, when we adapt these models for preference discrimination, the training objective fundamentally shifts away from score function learning to preference rank- ing. Essentially, this results a naive SLRM, similar to previous works (Liang et al., 2025; Zhang et al., 2025; Dhariwal & Nichol, 2021) that attempted to achieve noise compati- bility by simply adding noise to their inputs.
72
70
68
Accuracy
64
SCBR Dacine (1975
62
SLRM (5-250)
(1-20))
60
58
10K
20K
40K
80K
160K
Training Steps
Figure 4: Impact of Score Enhancement on SLRM. It com- pares SLRM with and without score enhancement across different timesteps.
5

---

Published as a conference paper at ICLR 2026
To resolve the gradual degradation during preference learning in existing methods, design the denoising score enhancement mechanism that maintains the model's noise compatibility by incor- porating denoising score matching into preference evaluation. Specifically, we first compute the denoising score matching distance D2(2; y, 8) of the diffusion model. For a DiT-based model with flow matching, this is expressed as:
(6) where s is the score embeddings. For computational efficiency with individual samples, we follow (Lee et al., 2023) and use the estimation for this expectation and modify the distance as:
(1 y. 8) = =,
(t,,,,
-
(1, y, 8) e e Pllloo
where p is the scale logit to ensure the score distance is scale-compatible with S(r,,c). Finally, we use this distance to augment the score logit S(x,; c):
( c) = ((x,,)) Ds(
(8)
The reward score S(x,, e) corrected by the denoising score must not only consider the aggregated semantic information across variant blocks, but also adapt based on the denoising viability of the latent at its current noise level.
Training of SLRM. Building on how the denoising score enhanced the predicted reward score described above, now detail the training loss of SLRM. Following prior works (Liu et al., 2025c; Yang et al., 2024a) that train a Bradley-Terry (BT) style reward model, we adopt a contrastive learning approach for optimization. Given preference dataset P {(x,"" 2/; a)) where x" and z' are win-lose pair images corresponding to the same prompt c. We randomly sample random timesteps / E {1,... T}. where T is the number of timesteps. The paired images are transferred into noisy latents xx and at, through the scheduler. These noisy latents are then fed into Eq. 9 to obtain the predicted scores S(xY e) and S(x), e) respectively, and the training loss of our reward model is formulated as:
LSLRM -R -EA .jep log S(a .c) S(x7. c)" ((z{.v)"
3.3 TRAJECTORY ADVANTAGE SAMPLING FOR PREFERENCE OPTIMIZATION
During the alignment phase, existing optimization methods typically sample two trajectories to form win-lose pair (æ:" a and determine preference order based on the reward on single intermedi- ate timesteps. Since previous work (Liu et al., 2025a; 2024) indicates that diffusion models focus on different dimensions at different timesteps (e.g., layout and composition in early stages, content coherence in middle stages, and visual details in late stages), our SIRMM is designed to effectively evaluate intermediate latents along sampling paths, thereby capturing comprehensive trajectory ad- vantages beyond single-step signals. However, fully leveraging these advantages through exhaustive evaluation presents a critical trade-off with computational efficiency.
To balance this, we propose Trajectory Advantages Preference Optimization (TAPO), a sampling strategy that efficiently identifies and amplifies trajectory advantages by strategically performing multi-step evaluations and progressively pruning less preferred samples. Specifically, within the total T sampling timesteps, we first designate n sampling steps where SDE sampling is performed to introduce randomness for stochastic exploration. They are uniformly distributed across the sampling trajectory, with the corresponding timestep set Wr defined as:
Wz (m1 T2....... T {1, 2.., and T, tinit (i- (i-1) iE (1.2...,,, (10)
n
where linit denotes the initial evaluation timestep. While the remaining steps use strategy of ordi- nary differential equation (ODB) provide deterministic path for efficient sampling. The sampling process can be formulated as:
(0(4))1))) + ot (x141 + (1 (2vo 1))] d(t) + olyo(t)e, if Wz Ct v)(2),))((t),
It+11
otherwise
(11)

---

Published as a conference paper at ICLR 2026
where (t)) denotes the timestep interval determined by the scheduler, (00 I) injects stochas- ticity. o, is the parameter controls the level of stochasticity during generation. At each SDE sam- pling step, obtain their respective sets of winning latent candidates X = (()))) and losing latent candidates x = { (0)) We evaluate them using SLRM and further retaining only the highest and lowest scoring samples:
i -0 2. i [I,..., PJ, * E [,
(12)
2t xxx (t)()) x!. argmin $6.))
After selecting the samples produced by the SDE, the two samples are selected for the subsequent ODE sampling phase. This process repeats iteratively, ultimately yielding the optimal and worst samples at the end of the sampling trajectories.
Notably, the SDE sampling steps uniformly distributed across the trajectory, covering diverse noise levels from early to late denoising phases. This enables progressive identification of distinct advantages from coarse to fine-grained across generation phases. Based on this insight, we dynami- cally capture trajectory advantages and identify win-lose pairs with pronounced quality differences while avoiding the computational burden of exhaustive per-step evaluation. This approach provides stronger training pair samples for preference optimization.
Following the SPO (Liang al., 2025) framework, we then optimize the model using the win-lose pairs have been collected:
CTAPO - -EE W( "! -po (e,t,are)
po (2/7" 1c,
Blog po (a// Ic, 22a)
(13)
log a Blog
Pref (x 1o-,lc,,4 v.)
Pref (a) lc, t, Di
We summarize the training procedure of TAPO in Algorithm. 1, can be found in Appendix. B.4.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Implementation Details. For T2I, we employ SD3.5-medium (Esser et al., 2024) as our base model for both SLRM and TAPO, while we utilize Wan2.1-1.3BB (Wan et al., 2025) as our base model for T2V. More training details and comparison methods setting can be seen in Appendix B. We pri- marily validate the effectiveness of our preference alignment method on T2I, while also conducting experiments T2V to demonstrate the effectiveness of our approach.
Datasets. For T21, SLRM is trained on Pick-a-Pic vl dataset (Kirstain et al., 2023) (580k prefer- ence pairs) and evaluated its validation/test sets (28k) for win-lose discrimination accuracy. For fairness, TAPO uses 4k prompts in SPO for online sampling with 20 timesteps. For T2V, due to the lack of high-quality video preference datasets, we collected preference dataset (10k pairs) for SLRM training and evaluate on GenAl-Bench (1.9k samples). Dataset details are in Appendix B.2.
Evaluation Metrics. For T2I, we evaluate TAPO (1) text-image alignment using CLIP (Radffoddo et al., 2021) and Ghhov et al., 2023), and (2) general preference using al., 2023), HPSv2.1(Wu (Zhang et al., 2024b). All metrics are evaluated on Pick-a-Pic vl validation set. Result of additional benchmarks of T2I and T2V can be seen in Appendix C.2.
PickScore(Kirstain
al., 2023), HPSv3(Ma al., 2025), and MPS
4.2 QUANTITATIVE EVALUATION
Comparison with SOTA Alignment Methods. Our method demonstrates substantial performance improvements across multiple evaluation dimensions. As shown in Tab. 1, TAPO achieves state- of-the-art performance across most evaluation metrics. Our method attains the best results on both general preference and text-image alignment metrics. Particularly on HPSv3, the latest preference alignment metric, TAPO outperforms the preference alignment method LPO and the latest FLUX.1 Dev by 0.79 and 0070, respectively. It demonstrates that our method achieves the best overall perfor- mance in visual quality and aesthetic preference. The user study is presented in the Appendix C.I.

---

Published as a conference paper at ICLR 2026
SD 3.5-M (Base Model)
DenseReward
Diffusionn DPO
Inversion-DPo
SPO
LPO
TAPO (Ours)
smiling. painting style."
"A man standing ancient jungle temple with mysterious glowing
husky wearing chef hat and apron, cooking, stirring a pot of soup the stove."
Figure 5: Comparison with SOTA (T2I). Qualitative comparison among various preference opti- mization methods based on SD 3.5-M (Esser et al., 2024). TAPO achieves superior overall gener- ation quality, outperforming baseline methods (DiffusionDPO, InversionDPO, SPO, LPO) in text alignment, visual quality, and aesthetic preference.
Table 1: Quantitative comparison results on Pick-a-Pic validation unique set.
Method
Preference
Alignment
HPSv2.1 (1)
HPSx3 (1)
PickScore (f)
MPS (?)
C CJ.P--Scare
GenPral (
SD-XL (Podell 2023)
26.05
7.52
21.94
0.89
21.73
52.29
SD3.5-M al., 2024)
25.18
FLUX.11 Dev (Labs, 20144
30.08
9.19
22.72
26088
58.20
Show-a (Xie et al., 2021)
25.17
8.05
20.96
0.83
21.32
46.92
Diffusion DPO (Wallace al., 2024)
28.23
7.59
22.64
1.93
25.71
$1,93
Alignment
DemsReward (Yang et 2024b)
Inversin-DPOO
2025h)
29.99
8.05
22.83
2.51
26.18
55.27
SPO (Liung et al., 2022))
31.52
8.71
22.70
2.21
21.72
52.75
LPO (Zhang al., 2025)
31.89
22.86
3.12
26.15
59.85
(Ours)
Comparison with SOTA Reward Model. To verify the noise compatibility of our SLRM reward model, we compare it with existing reward models at different timesteps. As shown in Tab. 2, the results demonstrate that our method achieves significantly higher accuracy under noisy inputs and perform better on evaluate the noisy latents. Our SLRM maintains high accuracy of 62.09% and 65.50% under these conditions. Compared to the diffusion-based LRM-3.5, SLRM achieves supe- rior performance across all timesteps, validating the effectiveness of our proposed ecore-enhanced learning strategy. Although HPSv3 and PickScore achieve higher accuracy of 72.80% and 71.93% respectively on clean images, SLRM's discrimination under noise input far exceeds theirs.
Noisy Compatibility of SLRM. To validate SI.M''s discriminative capability across the denoising trajectory, we compare with existing noise-compatible methods across comprehensive timesteps. As shown Tab. 4, our method achieves the best performance with 70.21% accuracy at 501, demonstrating SLRM's ability to assess noisy latents. This robust cross-timestep performance is crucial for effective reward scoring subsequent optimization.
4.3 QUALITATIVE EVALUATION
T2I. We qualitatively compare our method with the SOTA DPO-style approaches, including Dif- fusionDPO (Wallace et al., 2024), Inversion-DPO (Li et al., 2025b), SPO (Liang et al., 2025) and LPO (Zhang al., 2025). While DiffusionDPO and InversionDPO improve detail and color quality, they exhibit insufficient text alignment (3-rd row). SPO and LPO enhance text alignment, but SPO over-emphasizes subjects (1-st row) with degraded visual quality (2-nd row), while LPO shows in- sufficiency in aesthetic quality. Our method successfully balances text alignment, visual quality, and aesthetic preference, generating superior overall quality across diverse scenarios.
8

---

Published as a conference paper at ICLR 2026
WanZ, I-1-b (ilase Model
TAPO
"A N0V2 d nolae in wareng."
Staty pourr8 crEry Muee 1-1 rutte Motte son roturd Egving, NeY Alsmanrre, Hggey Frisiee, nes
Figure 6: Results of TAPO in Text-to-Video.
Comparison with Existing Reward Models. Accuracy comparison of VLM-based and diffussio-based reward models different timesteps on Pick-a-Pic validation and test sets.
Table 2:
Table 3: Ablation Study. Ablation study of SLRM and TAPO. "n" indicates the num-
ber of SDE sampling. "w/o Task Tokens": use image-text similarity score."()) s/iter de- notes time for sampling a pair.
Model
ZE [301, 10000
F [1.00M]
Orelaa Image)
CLIP
Strategy
VOAScure
46.55
Val-Tes
GenEval
HPS-2.1
VLM--Baed
HPS+3
HiSe2.1
PickScore
ImpgrReward
49.51
45.25
44.25
47.26
48.21
47.92
$3.06
72,50
65.58
67.66
71.93
SLRM (win Sumt Ealunasi)
SLRM (wis Tiak Tokens)
SLRM
62.49
65.33
67.52
3,07
2.08
1.92
55.27
64.93
54.83
28.37
30,71
26.49
Diftusion Based MM
LRM-3,5
59.55
64.12
66.41
(a- 3.0kiurr TAPO (a 39b//uzl
2.27
2.03
36.20
$3.50
27.30
23.31
TAPO (a 4:53/tur)
3007
65.93
30.01
TAPO - 16, (.08siter)
2.09
69.18
31.29
Table 4: Preference Prediction Accuracy across Timesteps. Results of SLRM's robust perfor- mance across detailed denoising timesteps compared to existing methods on Pick-a-Pic.
Variant Timestep
Method
= 101
201
(= 300
t= 101
= 501
= 601
701
801
901
SPM
63.75
62.41
62.97
62.58
61.74
61.50
60.82
58.92
56.21
53:46
LRM-3.5
65.42
63.78
64.25
64.03
63,12
62.89
62.15
60.28
57.64
54.83
(wlo Score Enhanced)
64.07
60.17
SLRM
65.81
63.88
67.88
66.15
69.68
70.21
69.55
66.65
67.24
58.12
T2V. We qualitatively compare our method with the base model Wan2.1-1.33 (Wan et al., 2025) across diverse video generation scenarios. The base model demonstrates basic generation capa- bilities but exhibits limitations in temporal coherence, text alignment and aesthetic preference. In contrast, our TAPO consistently generates the videos that are more natural temporal dynamics, en- hanced visual details, and superior text-video alignment.
4.4 ABLATION STUDY
Effectiveness of SLRM. We conducted ablation study evaluate the effectiveness of SLRM's different components in Table 2. For SLRM, removing the score enhancement mechanism results in a significant accuracy drop from 67.52% 62.49% (5.03% decrease), demonstrating its critical role in maintaining noise compatibility. This validates that both components are essential for effective preference discrimination.
Influence of Evaluation Numbers in TAPO. We analysed the influence of the number of SDE sampling and reward evaluation steps n (Eq. 10) in TAPO. Table 3, larger steps allow the model to explore more possibilities, increasing the quality of training sample pairs. Notably, when 8, GenEval shows a significant improvement, reaching 64.93. When we attempt to sample with more steps, there is no significant improvement in general preference, while sampling time increases sub- / 'iter + 6.08s/iter) iitee. Thus, we choose = as the setting our main result. More stantially (4.52s/
ablation study between the candidate size P and evaluation number T is presented in Appendix C.4.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We address two critical challenges in diffusion model preference alignment: unreliable reward es- timation on noisy latents and inconsistent preference evaluation across sampling trajectories. Our solution introduces SLRM, a score-based reward model that maintains noise compatibility through denoising score enhancement, and TAPO, trajectory-awart -ware optimization strategy that captures multi-timestep advantages for effective preference learning. Extensive experiments on T2I and T2V tasks demonstrate significant improvements, with SLRM achieving superior performance in noisy latent evaluation and TAPO attaining state-of-the-art results on HPSv3 and GenEval.
6 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, animal experimentation was in- volved. All datasets used, including Pick-a-pic and our collecting video preference dataset, were sourced in compliance with relevant usage guidelines, ensuring violation of privacy. We have taken care to avoid any biases or discriminatory outcomes in our research process. No personally identifiable information was used, and experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the re- search process.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we have made the following efforts: (1) We will release our code and the collecting dataset. Additionally, the dataset Pick-a-pic are publicly available, ensuring consistent and reproducible evaluation results. (2) We provide experiments setup in Sec, 4 and the more details about training process are presented in Appendix. B. including training steps, model configurations, and hardware details. (3) We elaborate on our evaluation protocol detail in Sec. 4. We believe these measures will enable other researchers reproduce work and further advance the field.
ACKNOWLEDGMENT
This work is partially supported by the National Natural Science Foundation of China (No. 62306261), HK RGC-Farly Career Scheme (No. 24211525), ITSP Platform Project (No. ITS/600/24F)), Funding from China Mobile, and the SHIAE Grant (No. 8115074). This study was supported in part by the Centre for Perceptual and Interactive Intelligence, CUHK-led In- noCentre under the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government. This work is also partially supported by Hong Kong RGC Strategic Topics Grant (No. STG1/E-403244NN, and CUHK-CUHI((ZSSGDST Joint Collaboration Fund (No. YSP26-4760949).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 2023. 1,3
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:231.11117. 2023. 16
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,, 2024. 3
10

---

Published as a conference paper at ICLR 2026
Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint SxXIv:2309.144000 2023.
Josef Dai, Tianle Chen, Xuyao Wang, Ziran Yang, Taiye Chen, Jiaming Ji, and Yaodong Yang. Safesora: Towards safety alignment of text2video generation via a human preference dataset, 2024. 16
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8798 8794, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024. 2, 8,8,17 17
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning cext-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858-79885, 2023. 1,3
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating coxtto-imaae alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023. 7
Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. c2i-comphench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025. 20
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2024. 20
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for eext-to-image generation. Advances in neural information processing systems, 36:36652-3666 2023. 3,
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 17,20
Black Forest Labs. Flux. https: /ttt::///gttttt http://ttttuub..mmbllaaa.c forest - Labs Sabs//E0 flux, 2024. 8
Jaa-Yeon Lee, Byunghee Cha, Jeongsol Kim, and Jong Chul Ye. Aligning text to image in diffusion models is casier than you think. arXiv preprint vrxiv:2503.08256 2025. 3
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint 1212, 2023. 6
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision language understanding and generation. In International conference on machine learning, pp.
PMLR, 2022.
Xiaomin Li, Yixuan Liu, Takashi Isobe, Xu Jia, Qinpeng Cui, Dong Zhou, Dong Li, You He, Huchuan Lu, Zhongdao Wang, et al. Reneg: Learning negative embedding with reward guidance. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 23636 23636-23645, 2025a. 3
Zejian Li. Yize Li, Chenye Meng, Zhongni Liu, Yang Ling, Shengyuan Zhang, Guang Yang, Changyuan Yang, Zhiyuan Yang, and Lingyun Sun. Inversion-dpo: Precise and efficient post- training for diffusion models. arXiv preprini arXiv:2507. 1554, 2025b. 8

---

Published as a conference paper at ICLR 2026
Zhanhao Liang. Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generic preferences with step-by- step preference optimization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13199-13208, 2025. 1, 3, 5, 7,8
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02247,, 2022. 4
Haofeng Liu, Chenshu Xu, Yifei Yang. Lihua Zeng, and Shengfeng He. Drag your noise: Inter- active point-based editing via diffusion semantic propagation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6743-6752, 2024. 6
Haozhe Liu, Wentian Zhang, Jinheng Xie, Francesco Faccio, Mengmeng Xu, Tao Xiang, Mike Zheng Shou, Juan Manuel Perez-Rua, and Jürgen Schmidhuber. Faster diffusion via tem- poral attention decomposition. Transactions on Machine Learning Research, 2025, 2025a. 6
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint ar arXiv:2505.054:: 2025b. 17
Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint ar 2xx1:25001139//1 2025c. 3,
Runtao Liu, Haoyu Wu, Ziqiang Zheng, Chen Wei, Yingging He, Renjie Pi, and Qifeng Chen. Videodpo: Omni-preference alignment for video diffusion generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8009-8019 8019, 2025d. 1, 2, 19, 20
Yunhong Lu, Qichao Wang, Hengyuan Cao, Xierui Wang. Xiaoyin Xu, and Min Zhang. Inpo: Inver- sion preference optimization with reparametrized ddim for efficient diffusion model alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28629-28633) 2025. 3
Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. arXiv preprint arXiv:2508.03777 2025. 3,7
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12. 2021. 1,3
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 4195 4205, 2023. 2
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01992. 2023. 8
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8763. PmLR, 2021. 7.7
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-5374 2023. 1, 3,4
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmir, 2021.
Hannes Risken. Fokker-planck equation. In The Bokker-Plannce equation: methods of solution and applications, pp. 63-95. Springer, 1989. 17
12

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 100844106955 2022. 1,3
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-3644 2022. 3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017. 1,3
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 1,3
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020. 2, 2,
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 8228-8238, 2024. 4,8
Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503. 203 2025. 2, 7, 9, 20
Changyuan Wang. Ziwei Wang, Xiuwei Xu, Yansong Tang. Jie Zhou, and Jiwen Lu. Towards accu- rate post-training quantization for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16026- 16026-16035, 2024a. 3
Fu- Yun Wang, Yunhao Shui, Jingtan Piao, Keqiang Sun, and Hongsheng Li. Diffusion-npo: Nega- tive preference optimization for better preference aligned generation of diffusion models. arXiv preprint arXiv:2505. 1245, 2025.
Yibin Wang, Zhiyu Tan, Junyan Wang, Xiaomeng Yang, Cheng Jin, and Hao Li. Lift: Leveraging human feedback for text-to-video model alignment. arXiv preprint arXiv:2412.04814, 2024b. 3, 19
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint 3xix:2306.0034/ 2023. 7,7
Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, and Aliaksandr Siarohin. Densedpo: Fine-grained temporal preference optimization for video diffusion models. arXiv preprint arXiv:2506..33: 03517, 2025. 3
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024. 8
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 15:03993-1 -0.559993355,.....11111111999335 2023.
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, and Yuxiao Dong. Visionre- ward: Fine- ganener mullti-dimensionlll human preference learning for image and video generation, 2024a. URL https: / aarxiv org/ abs/ 24 222122242 21 059. 1, 2,
13

---

Published as a conference paper at ICLR 2026
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang. Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine- sinengranned multi-dimensional human pref- erence learning for image and video generation. arXiv preprint arXiv:2412.21059, 2024b. 3, 19
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5 5-math technical report: Toward mathematical expert model via self- relf-improvement. arXiv preprint arXiv:2-09.122223 2024a. 6
Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with preference. arXiv preprint arXiv:2402.08200 082265.5.. 2024b. 2,
Jiacheng Zhang, Jie Wu, Weifeng Chen, Yatai Ji, Xuefeng Xiao, Weilin Huang, and Kai Han. On- linevpo: Align video diffusion model with online video-centric preference optimization. arXiv preprint urXiv:2412.15159 2024a.
Sixian Zhang, Bohan Wang. Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning multi-dimeesional human preference for text-to-image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8018-8027, 2024b. 3.7
Tao Zhang, Cheng Da, Kun Ding, Huan Yang, Kun Jin, Yan Li, Tingting Gao, Di Zhang, Shiming Xiang, and Chunhong Pan. Diffusion model noise-aware latent reward model for step-level preference optimization. arXiv preprint arXiv:2502.0105// 2025. 1, 2,
14

---

Published as a conference paper at ICLR 2026
Appendix of "Score-based Enhanced Latent Reward Model for Diffusion-based Visual Generation"
A The Use of Large Language Models (LLMs)
15
B Technical Details
16
B.1
Training Details
16
B.2 Video Preference Dataset Collecting
16
B.3 Details of Stochastic Differential Equations Sampling.
17
B.4 Training Algorithm of TAPO
18
C
Extended Experimental Results.
18
C.I User Study
18
C.2 More Benchmark Evaluation
19
C,3 Sensitivity Analysis of Regularization HyperParameter of B
19
C.4 Ablation of Evaluation Steps and Candidate Latents
19
C.5 Analysis of Reward Inconsistency during Sampling
20
C.6 Analysis of The Score Enhancement
22
D More Visualization
23
Section provides detailed technical details of methods, including training parameters, construction of the video preference dataset, and derivation of key SDE sampling.
Section provides extended experimental results of SLRM and TAPO, including additional benchmark evaluations of both methods on T21 and T2V tasks, as well as ablation studies on hyperparameters during TAPO training.
Section provides more visual results of our TAPO.
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models (LLMs) were used to aid polish the writing of this manuscript. Specif- ically, we used Claude-4-Sonnet solely for language polishing and grammatical refinement of the written text. All research contributions, including the main ideas, technical approaches, experimen- tal work, and scientific insights presented in this paper, entirely the work of the human authors. The LLM usage is limited to improving the clarity and readability of the already- wriiten content without altering the substance meaning of our work.
15

---

Published as a conference paper at ICLR 2026
B TECHNICAL DETAILS
B.1 TRAINING DETAILS
To ensure fair comparison, we retrained the diffusion-based reward model LRM based on SD3.5- medium which was originally based on SDXL-base. Similarly, SPO and LPO were also retrained on 3D3.5-medium All the hyperparameters of our training are shown in the Tab. 5. All the experiments conducted 8 NVIDIA A100 GPUs. The optimizer SLRM and TAPO are both AdamW with default parameters: b0tal=0.9, beta2=0.999, weight decay=0.01. The in Eq. controls the level of stochasticity is set 0.7.
Table 5: Hyperparameters of Training. The batch size represents an batch size implemented via gradient accumulation.
SLRM
TAPO
Hyperparameter
SD3.5-M
1-12,1-1.31
Hyperparameter
SD3.5-M
Wan2,1-1.3
Training Resolution
512 x 512
49 832 x 480 (16 FPS)
Training Resolution
512 512
81 x 832
480 (16 FPS)
Learning Rate
Learning Rate
10
Training Batch Size
32
16
Training Batch Size
8
Training Epach
5
Training Epoch
Datatype
BF16
BF16
Datatype
FP16
BF16
P
4
In
1000
500
LoRA Rank
Evaluation Steps (x)
8
Latents (N)
Sampling Timestops (T)
20
40
B.2 VIDEO PREFERENCE DATASET COLLECTING
To evaluate the effectiveness of our method on text-to-video, we require high-quality video prefer- pair datasets for training the reward model SLRM. However, existing open-source datasets (Dai et al., 2024) generally suffer from low quality (short duration, poor motion coherence), collected from UNet-based models like SVD (Blattmann et al., 2023). In contrast, current video generation models are predominantly based on DiT architectures with relatively better generation quality.
Categories:
Hunyuan y 1yp Wan Video Diffusion Model
Animal
City
Human
Car
Prompts
Human Scoring
Random Selection
Human Input:
Motion
Text Alignment
Visual Quality
fellow the formusa belbe, Ti...
Video-1
Consistency
Afouused cheffnn a lilp whol voomm and slight stheen of swest on Irs brow
7
4
Video-2
2
proparing in
Prompt List Collection
Preference Video Generation
Human Annotated
Figure 7: Video Preference Data Collection Pipeline.
We constructed pipeline for col- lecting paired video datasets and ob- tained a preference-annotatte dataset of 10,141 pairs through filtering.
2017
Number of Samples
831
870
905
030
832
640
Specifically, as shown in Fig. 7, we first establish list with 8 meta ele-
ments. Subsequently, we use LLM to extend the element category informa- tion into prompts for specific scenar- ios, ultimately obtaining prompt list of 10.1k items. And the distrubution shown in Fig. 8.
6
Content Categories
Figure 8: Distribution of Video Preference Dataset.
16

---

Published as a conference paper at ICLR 2026
We use state-of-the arr open-source video generation models, Wan2.1-14B (Wan et al., 2025) and Hunyuan- (Kong et al., 2024). to generate 2 videos for each prompt. To annotate the preference order of these videos with finer granularity, dimensions: Visual Quality (VQ), Motion Consistency (MC), and Text Alignment (TA), with human annotators scoring across these three dimensions. The annotation page can be seen in Fig. 13.
follow Flow-DPO and establish three annotation
B.3 DETAILS OF STOCHASTIC DIFFERENTIAL EQUATIONS SAMPLING
TAPO a online training method that requires stochastic sampling. However, the diffusion model that TAPO use is based on flow matching, which relies on a deterministic generative process based on ODEs. We follow (Liu et al., 2025b) converting the deterministic ODE sampling into SDE sampling and applying it the selected timesteps Wr. Here we further elaborate on this sampling process in detail.
Specifically, for deterministic probability flow ODE of the reverse process (Song et al., 2020), takes the following form:
dæ. = [ - log (c(x))dd
(14)
where f(æt, t) denotes the drift coefficient while the gtt) denotes diffusion coefficient. The pr (2x) represents the distribution of z, in reverse process. According the Fokker-Plaan Planck equation (Risken, 1989), the aforementioned ODE and this probability flow SDE have the same marginal probability density:
dæ; - [f(,, -) ( (o:)(())))de g())dw,
(15)
In the above equation, g(t) be expressed the standard deviation dee And according to the definition of the standard Wiener process, dw vdle, where NoO,, I). Note that flow models define a continuous-time normalizing flow through an ODE:
dæ, = v.dl
(16)
Based on this special case of Eq. 14, we have:
1 (t) a. log))(xx))
f(x,,t) 2
(17)
Substituting into Eq. 15, obtain:
of da,, [v Vn, log log o.vdle,
(18)
The key the equation is to establish the relationship between the score function Va, log ge (w+) and the velocity field v,, Following (Liu et al., 2025b), by leveraging the linear interpolation pat- hand conditional expectation of [x we derive their connection through the marginal score computation. Therefore, the score function is represented as:
St. t
Voga,, Pr(x,) =
(19)
Substituting into Eq. 18, we have the final SDE:
o²
dæ, =
(x, + (1 - t));) dt + o et
(20)
2t
Applying the Fuler-Maruyama discretization for SDE and the prediction velocity vo( f) for v. can yields our final SDE sampling scheme:
of (x: + (1 (2, b)) At AtA
E++ = 2z + vo ( t) +
(21)
22
In implementation, we use a specific noise scheduler with timestep / distributed according to a logit- normal distribution (Esser et al., 2024) sampling over [0, T), resulting in the following sampling scheme:
of
(x141 + (1 t) t) ()) +)) ((t) + orto)tte
2t E++1
vo(
(22)
2t
where the (1)) denotes the At determined by the noise scheduler.
17

---

Published as a conference paper at ICLR 2026
B.4 TRAINING ALGORITHM OF TAPO
The algorithmic procedure of TAPO is presented in Algorithm 1.
Algorithm Trajectory Advantagess Preference Optimization
Require:
Initial diffusion model va; Socre-based Latent Reward model 8(:); prompt dataset 3 total sampling steps T; SDE sampling steps Wr = {T1, T2,...; :n.
1: for training iteration & = 1 to K do
2: Sample batch prompts y/ ~ y
3: for each prompt y yb do
4: Init the same noise 11 ~ (0, 0 to do
5: for sampling timestep
6: if E WT then
7:
Use SDE Sampling in Eq. 11 get win candidates X7" and lose candidates X/. 1P
8:
Calculate Reward (6) (e (
1 and select the best and worst samples
"/-1:-/-1 in Eq. 12
9:
else
10:
Use ODE Sampling to get x;" and x of tow branches. end if
11:
12:
end for
Obtain win-lose trajectory latents { a' a aty.... .............................. I F. Computing Loss APO in Eq. 13
14:
Update diffusion model via gradient ascent: 0 + VooTTArO
16:
end for
end for
C EXTENDED EXPERIMENTAL RESULTS
C.I USER STUDY
We provide more details on our user study implementation. Besides qualitative and quantitative comparisons, we also conduct a user study determine whether our method preferred by hu- mans. We invite 13 participants from different social backgrounds and each test session lasts about 30 minutes. During the investigation, we conducted a pairwise comparison between our method and competitors across three key dimensions: 1) Visual Quality, 2) Text Alignment, 3) Aesthetic Prefer- ence. For "Visual Quality", users were asked to select which of the two images better fine-grained details and layout quality. For "Text Alignment", users evaluated which image more accurately flected the target text description. For "Aesthetic Preference", users judged which image aligned better with their aesthetic preferences, considering factors such as visual quality and the absence of artifacts or distortions. This comprehensive evaluation framework ensures a thorough and objective assessment of our method's performance relative existing approaches.
The results are as shown in Fig. 9, our method defeats all competitors in all dimensions, especially in Aesthetic Preference. This highlights the powerful ability of our framework in improving more aspects beyond text-image alignment.
(a) Visual Quality
[b] Text Alignment
(c) Aesthetic Preferance
Our UPO
Dus 7s 3P0
60.F%
93.0%
$3.0%
46.2%
$4.0%
25.4%
Ous vs Dffsccooro
61.3%
Praferndd
Contoenturs
Prefereed
Congettarr Preferred
Competttor Preferred
Rate
Win Rate (%)
Figure 9: User study. The percentages indicate the proportion users who thinks our method wins the competitor.
18

---

Published as a conference paper at ICLR 2026
C.2 MORE BENCHMARK EVALUATION
TAPO T2ICompBench++ (T2I. Our method, TAPO, demonstrates ofate-of-the-art perfor- mance across most evaluation dimensions. Notably, TAPO achieves the highest scores in crucial areas such as Color (0.7837), Shape (0.5684), Texture (0. (0.7036), and shows a particularly significant improvement in 2D-Spatial understanding (0.2713). This consistently superior performance com- pared to existing alignment models highlights TAPO's effectiveness in enhancing the text-image alignment and overall generation quality, especially for complex visual attributes and spatial ar- rangements.
TAPO on Vench (T2V). To quantify the improvement of TAPO on text-to-video generation, we compare against two baseline methods Hunyuan-13B and Wan2.1-14B, as well as one prefer- ence alignment method VideoDPO (Liu et al., 2025d). From the experimental results in Tab.8, TAPO achieves competitive performance with an overall score of 84.87, marginally outperform- ing VideoDPO (84.70) and other methods. TAPO demonstrates clear advantages in the key dimen- sions: Subject Consistency (98.79), which is highly sensitive to minor degradations that significantly impact overall quality. Notably, TAPO shows substantial improvements in Spatial Relationship (+4.62), indicating better understanding of complex spatial semantics in video generation.
SLRM on GenAI-Rench(T2V), To evaluate the performance of SLRM on text-to-video and validate its noise compatibility across differ- ent timesteps, conduct experi- ments on GenAl-Bench and compare against existing video reward mod- els including LiFT and VisionRe- ward. From the experimental results, SLRM demonstrates superior perfor-
Table 6: Results of SLRM on GenAl-Bench (Text-to- Video). "w/ Ties" indicates that takes account of tied pairs when calculating accuracy.
Method
Clean Video
E-E [1,500]
[901, 1000
w/ Ties
who Ties
Ties
w/o
Tles
Ties
LIFT (Wang al., 2024b)
37.06
58.39
Visionkewrd (Nu et 2024b)
$1.38
71,04
43.09
54.81
42.51
54.07
SLRM (Ours)
64,44
mance in noisy latent evaluation. While VisionReward achieves the highest scores on clean videos (51.38 w/ Ties), SLRM consistently outperforms all baselines across intermediate timesteps, achiev- ing 53.38 vs. 43.09 in early timesteps [1, 500J and 49.28 vs. 42.51 in later timesteps t [501, 1000]. These consistent performance gains validate that score enhancement mechanism effectively pre- serves noise compatibility during preference learning, enabling reliable evaluation of intermediate latents throughout the diffusion process.
C.3 SENSITIVITY ANALYSIS OF REGULARIZATION HYPERPARAMETER OF 3
To investigate the impact of the regularization hyperparameter 3 in Eq. 13 method, conduct hyperparameter analysis with results shown in Fig. The results demonstrate that appropriate reg- ularization coefficients can prevent catastrophic forgetting and severe performance degradation. As illustrated in the figure, extremely small = 20) lead to suboptimal performance across all metrics, with PickScore of 21.070, GenEval of 60.210, and 1PSv2.1 of 27. .550. indicating insufficient regu- larization that may cause the model "Catastrophic forgetting" and degrade. Conversely, excessively large 8 5000) also result in performance drops, particularly evident in PickScore (21.810) and GenEval (63. 700), suggesting over-rogularization that constrains optimization effectiveness. The optimal performance achieved at moderate values, with 500 yielding the highest PickScore (23.210) and / 1000 achieving peak performance on GenEval (68.830) and HPSv2. (32.010). Therefore, we choose 8 1000 for other experiments.
C.4 ABLATION OF EVALUATION STEPS AND CANDIDATE LATENTS
In TAPO, the two hyperparameters governing stochastic exploration are the number of SDE sam- pling steps (i.e., the size of Wr), and the number of candidate latents, P. used to form win-lose pair after each SDE sampling step (i.e., the size of x = {<r_,c) ) and X = ) To investigate the influence these two hyperparameters on the quality of generated samples and com- putational efficiency, conduct detailed ablation study. Higher step numbers and larger candidate sizes mean that the alignment phase can obtain higher-quality training samples based on SLRM's prior.
19

---

Published as a conference paper at ICLR 2026
PickScore
Gentval
HPSV2.1
735
60,300
21.300
33.366
22.
Score
23.899
23.070
22.3
263
380
1030
Viluu
Value
Veluee
Figure 10: Sensitivity Analysis on the hyperparameter 3 in CTAPO of Eq. 13
Table Result of TAPO on T21-CompBench+4 (Huang et al., 2025).
Method
Color
Shape
Texture
2D-Spatial
3D-Spatial
Numeracy
Non-Spatial
Complex
SD-XL
0,5892
0.4230
0.5172
Base Model
SD3.5.M
0.6810
0.4921
0.6295
0.2293
0,3491
0.5137
0,3108
0.3513
FLUUX.1
0.6971
0.5130
0.6123
0.5246
0.3679
Diffusion-DDD DPO
0.6829
0,5190
0.6338
0.2322
0,3620
0.5295
0.3155
0.3663
Aligument Model
Inversion
0.6910
SPO
0.7296
0,5392
0.6762
0.2409
0,3703
0.5724
0,3127
0.3721
LPO
0.7460
0.5508
0.6793
0.2541
0.5835
0.3838
TAPO
0.7837
0.5684
0.7036
0.2713
0.4013
0.5794
0.3217
0.3961
As depicted in Fig. 11, for moderate values (e.g., P 2,3,4, increasing the SDE steps n (from 16) generally leads to improvements in visual quality and prompt alignment. However, our study reveals that this benefit does not extend indefinitely. When these hyperparameters becomes excessively large (e.g., P=5), this may lead reward hackingmeaning the model overfits the reward signal by generating 'win' samples that are superficially preferred but lack genuine quality. This suggests that while increased stochastic exploration can enhance sample quality, an overemphasis on it can cause the reward model (SLRM) to exploit spurious patterns or artifacts in its reward landscape, resulting in visually unappealing outputs that paradoxically achieve high reward scores. More results can be seen in Fig. 16 and Fig. 17
C.5 ANALYSIS REWARD INCONSISTENCY DURING SAMPLING
As discussed in the introduction, different timesteps in diffusion models emphasize distinct aspects, resulting in inconsistent reward signals. To intuitively demonstrate this problem, we present two cases from the trained model's sampling trajectories to illustrate this phenomenon. As shown in Fig 12, the first case, despite overall higher quality, the 'win' sample's composition and layout are less coherent than the 'lose' sample in early stages (f 850, 750), leading to lower reward
Table 8: Result of TAPO on VBench (Huang et al., 2024). We apply our TAPO on Text-to- Video (T2V). "3" indicates the VideoDPO (Liu al., 2025d) conduct Dataset (Appendix B.2). VBench consists 16 dimensions, and we present several key dimensions that measure video quality and semanties, along with the overall score of other dimensions. "SC"; Subject Consistency, "AQ": Aesthetic Quality, "MS": Motion Smoothness, "OC": Object Class, "Human Action", "SR": Spatial Relationship.
Preference Video
Quality Score
Semantic Score
Overall Score
Method
SC
AQ
MS
OC
HA
SR
Quality
Semantic
Total
Hunyuan- 13B (Kong el al., 2024)
97.37
60.36
98.90
86.
94.40
68.68
85.09
75.82
83.24
Wan2.1-14B (Wan 2025)
97.52
66.07
98.30
86.28
95.40
75.39
85,59
76.11
83.69
Wan2.1-1.35 (haseline)
96.34
62.43
97.44
88.81
98.20
76.46
85.30
80.09
84.26
VideoDPO eLiu al., 2025d)
96.6 68
64.80
98.10
90.26
96.64
80.25
85.00
80.95
84,70
TAPO
98.79
67.27
98.12
89.62
98.00
81.08
85.21
82,49
84,87
20

---

Published as a conference paper at ICLR 2026
SDE Stops
SOE Sleps
n-
n=2
n
n
621
n=
n2
6 16
Candi
ab
Hacken
cute young child with brown hair playing chess, cortoor llustration, sof: bockground, warm lighting, children's book illustration style.
group tiry hamaters having giont strawborr.. whims.co
gigh
Figure 11: Results under Different evaluation Steps n and Latent Candidates Size P. "Reward hacking" is observed with excessive number of evaluation steps and large latent candidate sizes.
"A floating castle above the clouds, waterfalls falling the sky, fantasy landscape, cinematic lighting"
Reward
3.1562
3.0762
2.7500
3.323]
3,6925
3.7344
3.3438
3 3.6252
Win Latents
Reward
2.9531
3.6000
3.0862
3.3206
3.8169
2,9300
2.5312
2.7156
Lose Latents
t- 950
1-
850
1- 750
t- 650
t- 550
450
t- 350
250
anowy forest, one foreground, three deer grazing the midground, endless pine trees fading into the background, cold blue lighting. volumetric tog"
Reward
2.7969
2.6875
2.6719
2.8672
2.9688
2.8750
2.6953
2.8125
Win Latents
Reward
2.3750
2.5625
2.6641
2.7969
2,8203
2.8516
2.7422
2,8672
OVALE
Lose Latents
t-
950
850
750
t-
650
t-
550
450
350
250
Figure 12: Inconsistent Reward Across Denoising Timesteps. The critical challenge of inconsis- tent reward signals for noisy latents across different timesteps, a core motivation for our work.
score. In contrast, in later steps (t 250), it achieves higher reward score due to its refined details. similar inconsistency is observed in Case 2. For instance, at an early timestep (t = 650), the 'win' sample better captures the concept of "three deer," thus getting higher score.
21

---

Published as a conference paper at ICLR 2026
treme Rana: 1:46
2. corning reppi turkey mp4
00:00:0000::000 0p.00
Jana
30 30
(Anneration Vicil
Al Ki
Scaring Prume:
Figure 13: Video Preference Dataset Annotation Interface.
RLD,
SEn0 2011
win boundary Lowes comeroverttt overaal score
To 200
2005
Deweed Soure
Doward Seere
Figure 14: Visualization about the Influence of Score Enhancement on Reward Distributions. 'w/o Score Enhancement' exhibits lower overall reward scores, and the separation between win and lose latents becomes significantly blurred when t + 0, indicating reduced discriminability under noisy latent conditions.
C.6 ANALYSIS OF THE SCORE ENHANCEMENT
To more clearly demonstrate the influence of the score enhancement mechanism the reward score ((1,,)) produced by SLRM, visualize the reward distributions of 1,000 sample pairs from the Pick-a-Pic vl test set. For each pair, we plot the distributions of S(x"", , c) and S(x). c) across different diffusion timesteps. As shown in Fig. 14, with score enhancement, the win-lose reward distributions remain well separated across all timesteps, and the overall reward magnitude stays high. This indicates that SLRM can consistently discriminate between high-quality and low-quality latents, even under noisy conditions (i.e., (=0), In contrast, without score enhancement, the reward distributions shift toward lower values, and the separation between win and lose latents becomes significantly blurred as the timestep increases. This suggests that the model struggles to maintain reliable reward predictions when operating noisy latents. Overall, these results show that score enhancement substantially improves the stability and discriminability of SLRM under noisy latents, validating its necessity for robust reward modeling throughout the diffusion trajectory.
22

---

Published as a conference paper at ICLR 2026
Dore HTHERRI
Pia Au
Reward
RRWArE
Raward
Figure 15: Visualization about the Influence of "Training on = 0 Only" on Reward Distribu- tions.
D MORE VISUALIZATION
We present additional experimental visualization of our TAPO, including text-to-image in Fig. 18 and text-to-video in Fig. 19, which demonstrate that our method outperforms existing approaches in human aesthetic preference, text alignment, and other aspects.
23

---

Published as a conference paper at ICLR 2026
SDE Steps
n=1
2
n
n= 16
P--
3:33
Latege Candidates
P-3
P=d
P=3 5
"A phoenix rising from lake of molten gold, sparks and embers, high detail, dramatic lighting, fantasy realism'
Figure 16: Results under Different SDE Steps and Latent Candidates Size
SDE Steps
n = 2
n 4
n
Pa
Latent Candidates
y P.
p 5
"A fluffy kitten wearing tiny wizard hat, sitting inside teacup, pastel colors, soft lighting. kawaii style, ultra detailed"
Figure 17: Results under Different SDE Steps and Latent Candidates Size P.
24

---

Published as a conference paper at ICLR 2026
SD 3.5-M (Base Model)
DiffusionDPO
InversionDPO
SPO
LPO
AAPO (Ours)
STOP
S TOP
STOP
STOP
STOP
SCHOP
"Red Stop sign, school bus in the background, 4k, realistic."
"An abstract painting of a dragon flying over a grass field."
B
B
B
"Cinematic still of highly reflective stainless steel Bitcoin logo in the desert, at sunset"
"Charcoal drawing of majestic black pegasus flying in a mountain landscape by jmw turner and bob ross and monet, charcoal."
"A hero battling monsters, pixel art with retro 8-bit graphics, limited color palette, nostalgic video game vibe, iniptune-inspired environment."
"Dark gothic style with moody atmosphere, candle-lit interiors, ornate architecture, deep shadows."
Figure 18: Qualitative Comparison Preference Pptimization Methods. Rows show alignment of the subjects, and rows 4-6 show the alignment of style.
25

---

Published as a conference paper at ICLR 2026
Wand 1-1.3b (Dase Model)
AAPO (Ours)
Flowing shanding
glowing stolsctites, flasting fragments, magical lighting
Kitchen with werm lightingg, wooden counterisps cluffered with Fresh Ingredients, lively thef chopping cerret rapidly, Simmering pots releosing steam, soft background chatter and clining uhensis
swirling glais, gracnful movements, subile rePactiona gloss surfook gentle Rickering in of condleiiht"
Two people kissing under a colorful umbrella, cesual raincoats, wet hair gistening with raiddoops standing neon-lit urbon street with reflections puddics, steady rainfall creating répies. febric awaying in the cool breeze, medium shat from low angle focusing their cleseness, cinematic realsm, meody streesperee
"Two Stlers in colorful winter gear gliding along snowy mountain traiis lined with evergreen frees, ski tracks the foroground. snowflakes swirling in the wind, wide-sngle low shot following their mation, photorealstic style with crisp lighting and vibront colors"
Figure 19: Results of TAPO in T2V.
26


Reviews:
[{"review_id": "YatKJdZenf", "reviewer": "Reviewer_PpWQ", "summary": "This paper presents a framework for aligning diffusion models with human preference, addressing two challenges: unreliable reward scoring on noisy latents and inconsistent preference ranking from single-timestep evaluations. Specifically, the former is addressed by a Score-based Latent Reward Model (SLRM) that uses a diffusion backbone and a score enhancement mechanism to maintain noise compatibility. The latter is addressed using Trajectory Advantages Preference Optimization (TAPO), a trajectory-aware sampling strategy that evaluates rewards at multiple timesteps. Experiments show strong alignment performance on T2I and T2V tasks.", "questions": "1. Have the authors tried to combine both pixel-level reward models (like HPSv3, for high-frequency details at $t \\approx 0$) and the latent-level SLRM (for noise compatibility at $t > 0$)? A hybrid reward based on the timestep seems necessary to capture all quality aspects.\n2. Instead of augmenting the score logit with the multiplicative score distance (Eq. 8), is it possible to directly combine the denoising score objective (Eq. 6) as an additive regularization loss? How does that compare with the proposed enhancement approach?\n3. Currently, TAPO samples $P=4$ candidates for both win and lose samples, and only utilizes the best and worst among the four 15. Have the authors considered random selection to avoid reward hacking? What if the algorithm is run on all possible pairwise combinations with a smaller $P$, such as $P=2$ or $P=3$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed reward model, *i.e.,* SLRM, is innovative and efficient. It directly inherits the noise compatibility from the pretrained diffusion models while providing a more comprehensive evaluation of the text-image alignment, thanks to the self-attention mechanism at multiple semantic levels. Additionally, the solution to degraded noise compatibility is insightful: It is necessary to retain the noisy compatibility by incorporating denoising score matching objective into the reward model training.\n2. The proposed TAPO strategy provides a practical way to account for the entire sampling trajectory of diffusion models.", "weaknesses": "1. The proposed latent-level reward model one inherent drawback. Unlike the pixel-level reward models that have access to high-frequency information, latent reward models may be less sensitive to finer details like textures. This also partial explains why SLRM underperform pixel-level score models, such as HPSv3 and PickScore, as the later stage of the sampling process (where $t\\approx 0$) tends to generate finer details.\n\n2. The paper fails to mention a highly-relevant prior work that addresses a similar core problem regarding sequential sampling trajectory. While this paper introduces \"trajectory-aware optimization\" as a new solution to \"single-timestep preference evaluation,\" it omits the existing work by Yang et al. (2024) [1], which shares the same insight that \"diffusion models focus on different dimensions at different timesteps\" and introduces a dense reward perspective into DPO-style objectives.\n\n3. Lack of computational cost comparison. The proposed TAPO method introduces significant computational overhead, requiring $n=8$ SDE steps and $P=4$ candidates, resulting in 64 reward evaluations per iteration. However, the training time or iteration cost for the baselines is not included in the paper for fair comparison.\n\n4. Missing general quality metrics results. While the authors explicitly state that excessive exploration ($P=5$) leads to \"reward hacking,\" there is no quantitative evidence against it for the default setting $P=4$. To see if preference scores improve without sacrificing underlying image fidelity, standard metrics such as FID would be helpful.\n\n[1] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. \"A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.\" Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWAgbWNKc8", "reviewer": "Reviewer_yDbo", "summary": "This paper introduces a novel framework to improve human preference alignment in diffusion models. It identifies two critical issues in existing reward-model-based alignment methods: (1) unreliable reward estimation on noisy latents, and (2) inconsistent preference evaluation across sampling trajectories. To address these, the authors propose:\n\n1. SLRM (Score-based Latent Reward Model): integrates the diffusion model’s score function into the reward estimation process for noise-compatible evaluation.\n2. TAPO (Trajectory Advantages Preference Optimization): performs multi-timestep evaluation to ensure consistent and trajectory-aware optimization.\n\nExperiments on text-to-image (SD3.5) and text-to-video (Wan2.1) tasks show that this method outperforms prior works (LPO, SPO, Diffusion-DPO) on both alignment and preference metrics.", "questions": "1. How do the authors ensure that the SLRM trained on final-image-based preference labels can generalize to intermediate noisy latents, given that the ground-truth supervision does not explicitly cover those states?\n2. Can the authors clarify under which conditions TAPO provides larger qualitative gains over LPO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation: The paper identifies a genuine gap in current diffusion preference optimization(noise robustness and trajectory consistency) and addresses both with well-justified solutions.\n\n2. Technical novelty: The integration of the diffusion score function into reward estimation (SLRM) is original and theoretically grounded. TAPO’s multi-timestep evaluation design is practical and effective.\n\n3. Strong empirical results: Extensive experiments on both T2I and T2V tasks, with detailed ablation studies and visual comparisons, show consistent superiority.", "weaknesses": "1. The diffusion-based step-wise reward model is interesting and makes sense to me; however, since the preference ground truth in the training data is still based on the final images, it may limit the generalization ability of the SLRM.\n2. I didn’t observe a significant improvement of TAPO over LPO in the qualitative comparison presented in Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p1r41l5Ogt", "reviewer": "Reviewer_eNSr", "summary": "The paper tackles two challenges in aligning diffusion models with human preferences using reward models: \n(1) Unreliable reward estimation on noisy latent representations, and (2) Inconsistent single-timestep preference evaluations across trajectories. \nTo address these, the authors propose Score-based Latent Reward Model (SLRM), which incorporates task tokens and denoising score enhancement for noise-robust reward modeling, and Trajectory Advantages Preference Optimization (TAPO), which evaluates preferences across multiple timesteps to ensure consistent trajectory-level preference optimization. \nExperiments on text-to-image and text-to-video tasks demonstrate improved noisy latent evaluation, alignment, and generation quality compared to state-of-the-art baselines.", "questions": "1. Efficiency Trade-off. How does TAPO’s multi-timestep evaluation scale with longer trajectories or larger models?\n\n2. Interpretability of Reward Scores. Can the authors provide more insights or visualizations into how denoising score enhancement affects reward distributions? For instance, does it correlate more strongly with human ratings than baseline models?\n\n3. Baselines. More baselines are encouraged to compare with, such as SPPO [1] and RainbowPA [2].\n\n[1] Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization. arXiv:2410.05255, 2025.\n\n[2] Diffusion-RainbowPA: Improvements Integrated Preference Alignment for Diffusion-based Text-to-Image Generation. Transactions on Machine Learning Research, 2025.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Clarity:** \n\nThe paper is clearly written, with motivating examples (Fig. 1) and architectural comparisons (Fig. 3). The separation between SLRM and TAPO contributions is explicit, making it easy to follow the technical flow.\n\n\n**Quality:**\n\nThe improvements over baselines (e.g., LPO, Diffusion-DPO) are consistent across multiple preference and alignment metrics.", "weaknesses": "**W1. Score Enhancement Interpretability.** \n\nThe denoising score augmentation mechanism is motivated theoretically, but its interpretability for human preference signals could be further explored. Does it merely stabilize noise handling, or does it bias preferences toward easier-to-denoise samples?\n\n**W2. Lack of Baselines.** \n\nSee Q3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)