Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization
Authors: Tao Zhang
Cheng Da
Kun Ding
Huan Yang
kun jin
Yan Li
Tingting Gao
Di ZHANG
Shiming Xiang
Chunhong Pan
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Model
Reward Model
Preference Optimization
Image Generation
Primary area: applications
TL;DR: Leveraging the pre-trained diffusion model as a powerful and cost-effective step-level reward model to optimize the diffusion model itself directly in the noisy latent space.

Abstract: Preference optimization for diffusion models aims to align them with human preferences for images. Previous methods typically use Vision-Language Models (VLMs) as pixel-level reward models to approximate human preferences. However, when used for step-level preference optimization, these models face challenges in handling noisy images of different timesteps and require complex transformations into pixel space. In this work, we show that pre-trained diffusion models are naturally suited for step-level reward modeling in the noisy latent space, as they are explicitly designed to process latent images at various noise levels. Accordingly, we propose the **Latent Reward Model (LRM)**, which repurposes components of the diffusion model to predict preferences of latent images at arbitrary timesteps. Building on LRM, we introduce **Latent Preference Optimization (LPO)**, a step-level preference optimization method conducted directly in the noisy latent space. Experimental results indicate that LPO significantly improves the model's alignment with general, aesthetic, and text-image alignment preferences, while achieving a 2.5-28x training speedup over existing preference optimization methods.

Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization
Tao Zhang!.
Cheng Da²
Kun Ding'
Huan Yang²
Kun Jin Yan Li²
Tingting Gao Di Zhang
Shiming Xiang Chunhong Pan
'MAIS, Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, UCAS
Kuaishou Technology
Abstract
Preference optimization for diffusion models aims align them with human pref- erences for images. Previous methods typically use Vision-Language Models (VLMs) as pixel-level reward models to approximate human preferences. However, when used for step-level preference optimization, these models face challenges handling noisy images of different timesteps and require complex transformations into pixel space. this work, we show that pre- tre-rraiedd diffusion models are naturally suited for step-level reward modeling in the noisy latent space, they explicitly designed to process latent images at various noise levels. Accordingly, we propose the Latent Reward Model (LRM), which repurposes components of the diffusion model to predict preferences of latent images at arbitrary timesteps. Build- ing LRM, introduce Latent Preference Optimization (LPO), a step-level preference optimization method conducted directly in the noisy latent space. Ex- perimental results indicate that LPO significantly improves the model's alignment with general, aesthetic, and text-imags alignment preferences, while achieving a 2.5-28x training speedup over existing preference optimization methods. Our code and models are available at //tttt:://ggihhiie com/ Kwwii/Kwiiiaa Kolors/LPO
Introduction
Diffusion models 1 have achieved significant success in the domain of text-to-image generation.
Inspired by advancements in preference optimization [2] 3 for Large Language Models (LLMs), several methods are proposed to align the diffusion model with human preferences. Diffusion-DPO [4] extends Direct Preference Optimization (DPO) [5] to diffusion models, leveraging human-annotated data for training without requiring reward model. However, the reliance offline sampling introduces a distribution discrepancy between the preference data and the model, resulting in reduced optimization effectiveness. In contrast, DDPO [6] and D3PO [7] employ online sampling and reward models for preference assessment. Furthermore, to address the issue of preference order inconsistency different timesteps, Step-by-step Preference Optimization (SPO) introduces the Step-aware Preference Model (SPM) to predict step-wise preferences. Consequently. SPO can align models with aesthetic preference through step-level online sampling and preference assignment.
Reward models are critical for the aforementioned methods and are mainly implemented by fine- tuning Vision-Language Models (VLMs) like CLIP (9] preference datasets. We designate these models as Pixel-level Reward Models (PRMs) since they exclusively accept pixel-level image inputs. When applied to step-level preference optimization, PRMs encounter several common challenges. (1) Complex Transformation: At each timestep t, PRMs necessitate additional processes of diffusion denoising and VAE [10] decoding transform noisy latent images I; into clean ones á., and
xs Corresponninng author: Kun Ding <kun.ding@ia.ac.m.>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Aesthetic
Alignment
General
Original SDXL
Difrusion DPO
SPO
LPO
Figure 1: Qualitative comparison among different preference optimization methods based on SDXL
11. LPO excels in both aesthetics and text-image alignment, resulting improved overall quality. Larger versions with specific prompts are provided in Fig. 12 and Fig. 13
pixel-level images 4. This results in an overly lengthy inference process, illustrated in Fig. 2 (a) and Fig. (a). (2) High-Noise Incompatibility: At large timesteps characterized by high-intensity noise, the predicted images 1 are significantly blurred (as illustrated in Fig. 9.. leading to severe distribution shift from the training data of VLMs, i.e., clear images, thus making the predictions of PRMs large timesteps unreliable. (3) Timestep Insensitivity: Since PRMs typically do not incorporate timesteps as input, it is challenging to understand the impact of different timesteps on image assessment. These issues hinder the effectiveness of PRMs for step-level reward modeling.
Is there a model that can naturally cap- ture human preferences directly in the latent space while being aware of timesteps and compatible with sigh-intensity noise? We find that the pre-trained diffusion model for text-to-image generation is ideal choice because it exhibits several favor- able characteristics. as listed in Fig. (b).
Laberent Text-lmage Alignmeet
Inherest Text-Image Alignment
Complex Transformaionn
No Transfomaation
High-Noise Incomparibility
High-Noise Compatibility
Timestep Insensitivity
Timestop Sensitivity
Visias Language Modele
Preference
Haa-LLvel Rewarl Model
of Difisiin Musel
Lateni Reward
Midel
(1) It possesses inherent text-image align- ment capabilities, owing to the pre-training on large-scale text-image pairs. (2) It can directly process noisy latent images 2e with- out requiring additional diffusion forward- ing and VAE decoding. (3) It is high-noise- compatible since it can extract features from Ie with various noise intensities, as is done during pre- pre-training. (4) It naturally exhibits strong sensitivity to the denoising timestep, enabling it to effectively grasp the model's
CLI BLIP
Preference Optiazztion
lntti tzation
Initalizatinn
Optinization
Preforasce
Other
Diflusin
DMO
Models
Same VAR Parrwdar..
Uniieation
Utilextion
Ray
PMM
LRM
(a) Pixel-LLvel Reward Model
(b) Latent Reward Model (Ouns)
Figure 2: The comparison between the pixel-level re- ward model (a) and the latent reward model (b). DMO denotes the diffusion model to be optimized.
attention at different timesteps. These pre-trained abilities make the diffusion model particularly suitable for step-level reward modeling in the noisy latent space.
Based on the above insights, we propose using the diffusion model as noise-aware Latent Reward Model (LRM).. Given noisy latent images 2t, LRM utilizes visual features from U-Net [12) or DiT 13 and textual features from the text encoder to predict step-level preference labels. A Visual Feature Enhancement (VFE) module is employed to enhance LRM's focus on the text-image alignment. To address the inconsistent preference issue in LRM's 's training data, we propose the Multi- Preferereee Consistent Filtering (MPCF) strategy to ensure that winning images consistently outperform losing ones across multiple dimensions. Finally, we employ LRM for step-level preference optimization, leading to a simple yet effective method termed Latent Preference Optimization (LPO), where all steps are conducted within the noisy latent space of diffusion models. Extensive experiments on SD1.5 [4] and SDXL [ demonstrate that LPO substantially improves the quality of generated

---

images and consistently outperforms existing DPO and SPO methods across the general, aesthetic, and alignment preferences, as indicated in Fig. D Meanwhile, LPO exhibits remarkable training efficiency, achieving speedup of 10-28x over Diffuison-DPO [ and 2 over SPO [ Furthermore, we explore a step-wise variant of GRPO [55] based on LRR and apply L.OO to the DiT-based SD3 [ model, demonstrating the generalization ability of LRM and LPO.
The core contributions of this paper are summarized as follows: (1) noise-aware Latent Reward Model is introduced, which repurposes the pre-trained diffusion model for step-level reward modeling in the noisy latent space. (2) A Multi-Preferenn Consistent Filtering strategy is proposed to refine the public preference dataset, enabling LRM to better align with human preferences. (3) A Latent Preference Optimization method based on LRM is introduced perform step-level preference optimization directly within the noisy latent space of diffusion models. (4) Extensive experimental results demonstrate the effectiveness, efficiency, and generalization ability of the proposed methods.
Related Work
Reward Models for Image Generation. Evaluating text-to-imase tome generative models is challenging problem. Several methods leverage VLMs to assess the alignment of images with human preferences. PickScore [6]. HPSv2 [77. and ImageReward [18] aim predict general preference by fine-tuning CLIP 9 or BLIP 19 on preference datasets. MPS [20] is proposed to capture multiple preference dimensions. Based on PickScore, SPM 8 is introduced predict step-level preference labels during the denoising process. Recently, some works [21, 22 23 employ more powerful VLMs to learn human preferences. However, these reward models, limited to accepting pixel-level images, often face problems like distribution shift and cumbersome inference when used in step-level preference optimization. In contrast, our proposed LRM can effectively mitigate these issues. To the best of our knowledge, we are the first to employ diffusion models themselves for reward modeling.
Preference Optimization for Diffussin Models. Motivated by improvements in Reinforcement Learning from Human Feedback (RLHF) in LLMs 2 3.. several optimization approaches for diffusion models have been proposed. Differentiable reward fine-tuning methods 24 25 26 (7]
directly adjust diffusion models to maximize the reward of generated images. However, they are susceptible reward hacking issues and require gradient backpropagation through multiple denoising steps. DPOK (28) and DDPO 6 formulate the denoising process as a Markov decision process and employ Reinforcement Learning (RL) techniques for preference alignment, but exhibit inferior performance on open vocabulary sets. Many works 4 7
apply DPO [5] in LLMs to diffusion models, yielding better performance than the aforementioned RI-bbased methods. To mitigate the issue of inconsistent preference order across different timesteps. SPO (8] proposes step-level preference optimization method. In addition, some works (4) also explore training-free preference optimization methods. These methods typically perform optimization in pixel space, necessitating complex transformations and encountering distribution shift at large timesteps. On the contrary, LPO optimizes diffusion models directly in the noisy latent space using the LRM as powerful and cost-effective reward model, demonstrating significant effectiveness and efficiency.
29
30
31 32
3 Preliminaries
3.1 Latent Diffusion Models
The forward process of diffusion models gradually adds random Gaussian noise to clean latent images To to obtain noisy latent images x, at timestep i, in the manner of Markov chain:
(( = NN (1 - o)) at = 1 - Bt,
where Ni, Σ) denotes the Gaussian distribution and B3 is a pre-defined dependenn variance schedule. I denotes the identity matrix. The backward process aims to denoise 2₁, which can be formulated as follows according to DDIM
355:
po po( NH,, e (O.I),
(2)
#1 = V 14--1000 + V1 5 - of c0,t, fo, = Ft T00,60
(3)
Vai
3

---

o = 1 01 - a.)11 0z, ā II
(4)
where 1j E [0. is hyperparameter to adjust the standard deviation ds- Co., and Co. denote the noise and clean latent images predicted by diffusion models with parameters at timestep t.
3.2 Preference Optimization for Diffuusio Models
Given winning latent images z losing ones z/- and condition c, Diffusion-DPO propagates the preference order (xt æd) to all denoising steps, thereby freely generating intermediate preference pairs (x),,))
2z). It encourage models pe to generate zi rather than x4 by minimizing
Po(a;" 11 Blog Pref |
Pres (ai (
po(z)] e)
LDro = - ER per (o) se) logo # ppp (1 (=)0))
(Blgg
where 3 is a regularization hyperparameter and Pref denotes the reference model, fixed to the initial value of po- However, the preference order along the denoising process is not always identical. Therefore, SPO
[8] proposes SPM to predict preferences for intermediate steps, and sample the win-lose pairs (x+]") ar() from the same 2 1 to make the paths comparable. Accordingly, the optimization objective of SPO is reformulated as minimizing
po (xt [T+1. 3log po(a)lee+1, Pref (x) (141,
Pres (af (afI ) (6)
LSPO = -Egg ploo log Bllgg
As a Pixel-level Reward Model (PRM), SPM faces some common issues discussed in Sec. | First, as illustrated in Fig. (a) and (c), it requires additional procedures including Lo, prediction and VAE decoding. Second, it struggles to process highly blurred I when / is large (Fig. D Third, although the AdaLN block [13 introduced in SPM improve its sensitivity to timesteps, insufficient pre-training makes it challenging to understand the focus of different timesteps on image generation. These issues jointly diminish the effectiveness of SPM in step-level preference optimization.
Method
To address the limitations PRMs, we propose the Latent Reward Model (LRM) Sec. 4.1 lever- aging diffusion models for reward modeling. To facilitate its training, we propose Multi-Preferenee Preference Consistent Filtering (MPCF) strategy in Sec. 4.2 Finally, the Latent Preference Optimization (LPO) is introduced in Sec.
4.3 to optimize diffusion models directly in the noisy latent space.
4.1 Latent Reward Model
Architecture of LRM. LRM leverages fea- tures of the U-Net and text encoder in the dif- fusion model for preference prediction, as de- picted in Fig. 3 Specifically, the textual features fp E IT X2p are extracted from the prompt by the text encoder, where nı and ni denote the number of tokens and the dimension of textual features, respectively. Following CLIP [9] we use the last token feas E I 11xmp to represent the entire prompt and incorporate a text pro- jection layer to obtain the final textual features T R Ixns, where na represents the final di- mension of textual and visual features. Each noisy latent image v, passed through the U- Net to interact with textual features Jy- The vi- sual features of U-Net are average pooled along the spatial dimension, resulting in the multiscale
S(oa2) -
TER
Toaxi
Pro
Aumago Railing
Vena gi
free # a
Vewm
VFE
Token
Diliaaion Model
Down
Dowa
4 e Rion
f R0
Trainin Only
Teli
VAE
Test Enaader
Ercosay
Figure 3: The architecture of LRM. The VAE en- coder is only used during training.

---

(a) PRM (SPM) Training
CE (50,00. 700750)
(b) LRM Training
(50,100, 900,950)
Prediction
Decoding
VAE Encoder
Difiain Model
VAE Decoder
PRM
Loss
VAEE Encoder
LRM
Loss
e!
promyu
prompt
(c) SPO Sampling
(d) LPO Sampling
(50.100 o0. 900,968
of
Selection
Xe+1
Xy Ny
Xo
Rey Prodiction
VAE Decoding
De > 0p-1
Rde
Cunsart
L.KM
LRM
Diffesion Model
VAE Decoder
Theestold
(f xx)
Cigy
Dynume Threshold
the > Che-1
(Aous. a5)
Figure 4: The training pipelines of PRM (a) and LRM (b), and sampling pipelines of SPO (c) and LPO (d). Compared to PRM. LRM can directly process latent images I; at various noise levels without requiring & prediction and VAE decoding. Therefore, LPO can perform sampling entirely within the noisy latent space throughout the full denoising process, covering t [0, 950).
down-block features Vdown and middle--bockk features Vinid follows
Vdown, Vmni = AU-
Vdown = {Va, i L,,,, L},
(7)
where L is the number of down blocks and Va, represents features of the i-th down blocks. However, it is observed that these features lack sufficient correlations with the textual features. Inspired by the Classifier-Free Guidance [36], we propose the Visual Feature Enhancement (VFE) module to enhance correlations between visual and textual features. It first extracts middle-block features Vmidd wcond without textual information, following a process similar to extracting Vmida but with a null prompt. The enhanced visual features Venh E R 1xxnm are then computed as follows:
Venh Vmii + (gs - 1)
ucond);
(8)
where gs 2 is hyperparameter and num is the dimension of Viiid larger gs value injects more text-related features into Vmid, thus enhancing its focus on text-image alignment. When gs = 1, the VFE module is disabled. Next, Venh is concatenated with down-block features Vdown along the channel dimension and projected into the final visual features V R Ilxaa via visual projection layer. Finally, the preference score nr and p is the dot product between textual and visual features:
x1)
(12(V) (2()));
where 7 is temperature coefficient following CLIP 9 and l2 denotes the L2 Norm.
Training Loss. LRM is trained on the public preference dataset Pick-a-Pic v1 [16], denoted as D, which consists of triples, each containing a preferred image I" less preferred image 1', and the corresponding prompt p. As depicted in Fig. 4 (b), given uniformly sampled timestep L~ (1, T) where T denotes the total denoising steps, the pixel-level image / is encoded into the latent image To using the VAE encoder. Noise e (O I) is then added to simulate x, in the backward denoising process. Following the Gradley-Terry (BT) model (37]. the training loss of LRM is formulated as:
exp(S(p, ))")
LLRM =
log exp(S(p, + exp(S(p,x)))
(10)
Var VA (T") + v1 ai * e
(w.1).
(11)
During training, the parameters of VAE are frozen to ensure that the latent space of the LRM remains stable, while the text encoder and U-Net are trainable.
4.2 Multi-Preefrrnce Consistent Filtering
Issue of Inconsistent Preferences. The training loss of LRM in Eqn. (10) involves the assumption that if æg. is preferred over a/. then after adding noise of equal intensity, the preference order remains unchanged, meaning x." continues to be preferred over a/f. However, this assumption breaks down
5

---

when the winning image excels in one as- pect but is inferior in other aspects. For example, if the winning image zi has bet- ter details but exhibits weaker text-image alignment compared to the losing image zo- the advantage in detail may be diminished after introducing significant noise, mak- ing the noisy losing image o potentially more preferred than z²". Such situations frequently occur in the Pick-a-Pic v1 [16] dataset. To demonstrate this, we employ the Aesthetic Score SA (38] to evaluate the
52.3%
Aesthetic-
GĄ 2
G4 <
40.5%
CLiPScore
Gr <c<00 0
61.7%
38.3%
VQAScore-
Gy 2
Gyst
27.9%
24.4%
31.6%
16.1%
Combined
G,
G, 0
6 0
20
60
80
100
Ratio preference pairs that meet the criteria
Figure 5: The distribution of preference consistency between human labels and reward scores in Pick-a-Pii
aesthetic quality while using CLIP Score
Sc [9] and VQAScore Sy (39 to assess the text-image alignment. Let G. = -. - (T') {A, C. V denote the preference score gaps between winning and losing images. As illustrated in Fig. 5) nearly half of the winning images have Aesthetic Scores lower than those of losing images. Similarly, around 40% of the winning images exhibit lower CLIP Scores and VQAScores. When considering both Aesthetic and CLIP scores, over 70% of the winning images score lower than the losing images in least one aspect.
Peati--Preeeeecce Consistent Filtering. We hypothesize that if the winning image outperforms the losing image across vari- ous aspects, the preference order is more likely to remain consistent after introduc- ing similar noise. Based on this hypoth- esis, we propose the Multi-Preference Consistent Filtering (MPCF) strategy, which aims approximate the condition of the above hypothe- sis by filtering data using various preference score gaps. Pick-a-Pic v1 contains 511.8 1,84 win-lose pairs and 71,907 tie pairs. Three strategies are explored for win-lose pairs, as detailed in Tab. [ The first strategy is the most strict, but it is observed that it causes the LRM to overfit to the aesthetic aspect while neglecting the text-image alignment. regarding aesthetics, resulting in the L.R being unable to perceive the aesthetics of the images. It is found that the second strategy effectively balances both aspects, leading us choose ultimately. Further investigations and experiments are detailed in Sec. 5.3
Table 1: Data filter strategies for Pick-a-Pic.
Part
Strategy
Filter Rule
Num
GA 2 0. 2 0, Gy 20
101,573
Win-Lose
GA -2.50.5
0,
168,539
-1, 2 Gv 2
201,885
Tee
[GAl & 0.2, (Gel & 0.03, [Gv & 0.07 8,537
contrast, the third strategy the most lenient
4.3 Latent Preference Optimization
To utilize LRM for step-level preference optimization, we propose the Latent Preference Optimization (LPO) method. We also explore the application of LRM on GRPO [15] in Section B
Sampling and Training. As illustrated in Fig. 4 (d), each timestep I, LPO samples a set of of, 1..... K from the same Tt+1- These noisy latent images are then directly fed into LRM to predict preference scores S i 1, ...,, K. The highest and lowest scores are normalized by the SoftMax function. If their score gap exceeds predefined threshold the, the Ie with the highest score selected x²" while the one with the lowest score designated as z/,, forming qualified training sample 1, x!). Finally, these samples are used to optimize the diffusion model using the loss in Eqn. As a result, the entire process of LPO is conducted within the noisy latent space, eliminating the need for d prediction and VAE decoding, which significantly enhances the training effectiveness and efficiency of LPO.
Optimization Timesteps. Due to the inaccuracy of SPM high-noise levels, SPO samples training data only at low to medium noise levels, specifically when t E 750). as shown in Fig. 4 (c). In contrast, owing to LRM's noise-aware ability, LPO can perform sampling and training through- out the entire denoising process, covering t E [0, 950]. Experiments in Sec. 5.3 and Section B demonstrate that optimization at high noise levels plays crucial role in preference optimization.
Dynamic Threshold. The sampling threshold th is crucial hyperparameter in LPO. A higher threshold tends to sample win-lose pairs with more pronounced differences, improving the quality of the training samples but reducing their overall quantity. Conversely, a lower threshold increases the number of training samples but may introduce certain amount of noisy samples. Given that the

---

standard deviation oe in Eqn. 2 decreases with t, using a constant threshold across all timesteps can hinder the training effectiveness. To handle this, we implement a dynamic threshold strategy, which sets lower thresholds for smaller timesteps, as follows:
Ge mmnn
the
(thmaa - thmmn) + themsns
(12)
mman
Min
where moa and mmin are the maximum and minimum values of or. The hyperparameters thtma and thmmn are used adjust the range of thresholds.
Selection of LRM in LPO. To avoid confusion, we refer to the diffusion model optimized in LPO as DMO. Since LPO is performed within the latent space, which is determined by the VAE encoder, the VAE of LRM should be identical to that of DMO. Therefore, the specific architecture LRM can be initialized from DMO any other diffusion models that share the same VAE encoder with DMO, as depicted in Fig. (b). The former is termed homogeneous optimization because LRM and DMO share the same architecture, whereas the latter is called heterogeneous optimization.
5 Experiments
5.1 Experimental Setup
The experiments are mainly conducted on SD1.5 [4 and SDXL [T] without refiner. The LRM is first trained on Pick-a-Pic and then used to fine-tune diffusion models through LPO. If not specified, we employ homogeneous optimization. Ablation experiments are conducted SD1.5.
LRM and LPO Training. We denote the LRM based on SD and SDXL. as LRM-1.5 and LRM-- XL, respectively. They are trained on the filtered Pick-a-Pic v1 [16 as clarified in Sec. 4.2 The gs in the VFE module set to 7.5. The same 4k prompts in SPO are used for the LPO training, randomly sampled from the training set of Pick-a-Pic v1. The DDIM scheduler (35 with 20 inference steps is employed. We use all steps for sampling and training, i.e. t E [0. 50. .... 900, 950]. The dynamic threshold range [thmin, thmmaa] is set [0.35, 0.5] for SD1.5 and [0.45, 0.6] for SDXL. The / in Eqn. (6) is set to 500 and the K in the sampling process is set to 4. More details are in Section A
Baseline Methods. We compare LPO with DDPO 6). D3PO 77. Diffusion-DPO
MaPO 29
SPO [8) and SePPO (30]. These methods are trained on Pick-a-Pic, which ensures relatively fair comparison. We also include InterComp [31 for reference, which uses a higher-quality internal
dataset. More details are provided in Section A
We evaluate various diffusion models across three dimensions: general preference, aesthetic preference, and text-image alignment. The PickScore [16), HPSv2 (17), 1 [17]. and ImageReward [18] are utilized to assess the general preference. The aesthetic preference evaluated using the Aesthetic Score (38 Following []. both general and aesthetic preferences are assessed on the validation unique split of Pick-a-Pic vl, which has 500 different prompts. For text-image alignment, we employ the GenEval
Evaluation Protocol.
40 and 121-CompRenche+ [41 metrics. For detailed evaluation practice, please refer to Section A Additionally, we propose two metrics to assess LRM's correlations with aesthetics and text-image alignment. Specifically, we calculate the score gaps G,,* E {A, C, L} between winning and losing images, where A, C. L represent Aesthetic, CLIP, and LRM. For LRM, the score is taken at 1 = O. Then the Pearson Correlation Coefficient [42 between Gr. and G is referred to as Aes-Corr while that between G1, and Ge is termed CLIP-Corr. They are evaluated on the validation unique and test unique splits of Pick-a-Pic vl.
5.2 Main Results
Quantitative Comparison. As indicated in Tab. and Tab. 3 LPO outperforms other methods across various dimensions by a large margin on SD1.5, especially in general preference and text-image alignment. Based on SDXL, LPO even slightly surpasses InterComp, although the latter utilizes a superior dataset comprising images from SD3 and FLUX 43). The user study results in Section B further confirm the superiority of LPO. We also validate the effectiveness of heterogeneous optimiza- tion in Tab. 4 Remarkably, fine-tuning SD2. using LRM-1.5 yields significant improvements across various aspects, demonstrating that an inferior diffusion model can effectively fine-tune an advanced model as long as they share the same VAE encoder. In contrast, applying LRM-1.5 for the LPO of SDXL is ineffective due distribution mismatch between the latent spaces of their VAE encoders.
7

---

Table 2: General and aesthetic preference scores on Pick-a-Pic validation unique set, along with GenEval [40] scores. denotes the metrics are copied from [8] Others are evaluated using the official
model. If not specified, we use 20 inference steps. PaP denotes the Pick-a-Pic [16] dataset. The
complete results on GenEval are provided in Tab. 25 and Tab. 26
Model
Method
Dataset PickScore
ImageReward HPSv2
HPSv2.1
Acathetic GenEval (20 Stop)
GenEval (50 Stop)
Original
20.56
0.0076
26.46
24.05
5.468
42.56
42.53
DDPO
PaP v2
21.06
0.0817
24.91
5.591
D3PO
PaP
20.76
.01235
23.97
5.527
SD1.5 14
Diff-DPO
+
PaP v2
20.99
0,3020
27.03
25.54
5.595
43.79
45.13
SPO B
PaP v1
21.43
0.1712
26.45
5.887
SPO
g
PaP v1
21.22
0.1678
26.73
25.83
5,927
40.46
41.53
ScPPO [3
PaP
21.25
0.5077
27.56
27.34
5.766
44.97
44.46
LPO (ours)
Pal
21.69
0,6588
27.64
27.86
5,945
48.39
48.77
Original
21.65
0,4780
27.06
26.05
5,920
49.40
$2.29
DiIL-DPO
1
Pal v2
22:22
0.8527
28.10
28.47
5,939
57.78
58.91
MaPO
PaP
21.89
0.7660
27.61
27.44
6.095
51,59
52.80
SDXL 1
SPO
PaP
23.06
1.0803
31.80
6.364
55.20
SPO
PaP
22.70
0.9951
28.42
31.15
6.343
50.52
52.75
InterComp
H1
Internal
22.63
1.2728
29.08
31.52
6.016
59.24
59.65
LPO (ours)
PaP vl
22.86
1.2166
28.96
31.89
6,360
59.27
59.85
Table 3: Quantitative results on 121-CompBench++ + 41 with 20 inference steps.
Model
Method
Color
Shape
Texture
2D 2D.Sparial
IDaStali
Numeracy
Spatial
Complex
Original M
0,3783
0.3616
0.4172
0.1230
0.2967
0,4485
0.3104
0.2999
DifT.DPO
0.4090
0,3664
0.4253
0.1336
0.3124
0.4543
0.3115
0.3042
SD1.5 M
SPO g
0.4112
0.4019
0.4044
0.1301
0.2909
0.4372
0,3008
0.2988
SePPO
0.4265
0,4170
0.3285
0.3109
0.3076
LPO (ours)
0,5042
0.4522
0,5259
0.1928
0.3562
0.4845
0,3110
0.3308
Original [1]
0.5833
0,4782
0.5211
0.1936
0.3319
0.4874
0.3137
0.3327
Dit 4
0.6941
0.5311
0.6127
0.2153
0.3686
0,5304
0.3178
0.3525
SDXL. [I]
MaPO
SPO
S
0.6410
0.4999
0,5043
0.5485
0,5551
0.2096
0,3629
0.4931
0.3098
0.3154
0.3229
0.3467
InterComp
0.7218
0.5335
0.6290
0.2406
0,3929
0,5395
0.3212
0,3659
LPO (ours)
0.7351
0.5463
0.6606
0.2414
0.4075
0.5493
0,3152
0.3801
Qualitative Comparison. The qualitative comparisons of various methods are illustrated in Fig.
and Fig. [ig. [5] Images generated by Diffusion-DPO exhibit deficiencies in color and detail, whereas those from SPO demonstrate reduced semantic relevance and excessive details in some images, resulting in cluttered visuals. In contrast, the images produced by LPO achieve a strong balance between text-image alignment and aesthetic quality, delivering higher overall image quality.
Training Efficiency Comparison. LPO enables significantly faster training. As shown in Tab. 5 by performing reward modeling and preference optimization directly in the noisy latent space, LRM bypasses both &. prediction and VAE decoding during sampling. When K 4. its sampling time is only 1/6 that of SPM in SPO, yielding substantial time savings-ard and the same efficiency gain applies to LRM training. Consequently, as illustrated in Tab. [6] I.PO requires only 23 A100 hours for SD115-11 training time is reduced to 1/28 and 1/2.5 of that for Diffusion- -PO and SPO, respectively.
1/10 the training time of Diffusion-DPe and T/3.5 that of SPO. On SDXL., LPO's
Further Explorations. In Section we present additional exploration experiments on LRM and LPO. First, we validate the effectiveness of LRM on step-wise variant of GRPO [15]. We then assess the applicability of LPO on DiT-Based SD3 [ and evaluate its generalization on other datasets. Additionally, we report the performance of LRM when used as a pixel-wise reward model.
5.3 Ablation Studies
MPCF. As shown in Tab. 7 MPCF plays a vital role in LRM training. The first filtering strategy enforces that winning images must outperform losing images across all aspects. However, since the diffusion model lacks explicit text-image alignment pre-training like CLIP, it prone to overfitting to visual features, indicated by a higher Aes-Corr. This overfitting results in reduced attention to alignment, as reflected by lower CLIP-Corr and GenEval scores. The second and third strategies relax the aesthetic constraints to varying degrees. While the third, most lenient strategy can cause LRM to focus solely on alignment and neglect image quality, as evidenced by a negative Aes-Cor value, the second strategy achieves better balance, yielding the highest general preference scores.
8

---

Table 4: Heterogeneous optimization based LRM-SD1.5.
Table 6: Comparisons of training
Model
Method
Aesibetic
Gentival
I-R
HPSV2.1
speed between different methods.
SD2.1 [4
Original
5.673
48.59
20.92
0.3063
27.05
25.49
Reward
Preference
Total 1
(Same VAF)
IPPO (ours)
3.969
56.01
21.76
0.7978
28.05
28.61
Method
Modeling
Optimization (A100 h)
SDXL
Original
5,920
49.40
21.65
0,4780
27.06
26.05
SDL.5
(DE.. VAE)
LPO (ours)
5,953
40.85
20.82
0,3919
27.10
26.69
Dif
SPO
48
80
Table 5: Time of each sampling step on single A100.
LPO (ours)
8
Reward
Denoising Step fo.: Prediction
VAE Decoding
Reward
Total (K=4)
SDXL
Model
(#x+1
(Ze
Bo,s)
(2a.e
1e)
Prediction
DFO
2,560
SPM (PRM)
Kx 0.019s
Kx 0.031s
Kx 0.006s
0.243s
SPO
116
118
234
LPO (ours)
40
LRM (ours)
0.01%
0
0
XX 0.005s
0.039%
Table 7: Ablation results on LRM's training data.
Table 8: Ablation results on the VFE module.
LRM
LPO
LRM
LPO
Strategy
VFE
ge
Acs-Corr CLIP-Corr
Aesthctic
GenEval
PickScore
Aes-Corr CLIP-Con
Aesthetic
GonEval
PickScore
MPCF
0.1342
0.2274
5.772
45.66
21.49
x
1.0
0.1712
0.3211
6.053
46.60
21.51
0.4860
0.1011
6.390
45.77
21.61
0.1233
0.3441
47.35
21.53
0.1136
0.3588
5.945
48.39
21.69
7.5
0,1136
0.3588
5,945
48.39
21.69
10.1152
0.4480
5.750
48.62
21.47
10.0
0.1063
0.3592
48.13
21.56
Notably, even without MPCF and using the same training data, LPO still outperforms SPO in both general and text-image alignment preferences, as detailed in Section C
Structure LRM. As illustrated in Tab. S the introduction of VFE (gs > 1) leads to lower Aes-Corr values but higher CLIP-Corr values, indicating an enhanced emphasis on text-image alignment. This results in improvements both the GenEval and PickScore, with only a minor decline in Aesthetic Score. As gs increases, the LRM's correlation with alignment steadily improves, while its correlation with aesthetics decreases. When gs is set to 7.5, the model achieves the best overall performance.
Optimization Timesteps. Tab. 9 ablates different optimization timestep ranges, indicating that broader ranges lead to better performance. Notably, LRM can effectively predict preferences even at very large timesteps, e.g., (750, 950). achieving results comparable to those obtained across the entire denoising process, i.e. [o, 950). This highlights the critical role of large timesteps in step-level preference optimization. In contrast, SPO performs poorly in large timesteps, as indicated in Tab. 21 in Section C This comparison underscores the superiority of using diffusion models for step-level reward modeling at large timesteps, since LRM can directly process noisy latent images without suffering from distribution shift. A qualitative comparison different ranges is provided in Fig. I0
Dynamic Sampling Threshold. The standard deviation of samples at smaller timesteps is relatively small according to the DDPM scheduling 44. making the constant threshold insufficient
to accommodate all timesteps. As indicated in Tab. 10 the dynamic threshold strategy generally outperforms the constant threshold across different intervals, effectively alleviating this problem. We further explore other dynamic strategies in Tab. 24 in Section
6 Discussion and Conclusion
Conclusion. In this paper, we propose LRM, a method that utilizes diffusion models for step-level reward modeling directly in the noisy latent space, based on the insights that diffusion models inherently possess text-image alignment abilities and can effectively perceive noisy latent images across different timesteps. To facilitate LRM training, introduce the MPCF strategy mitigate the inconsistent preference issue in training data. Building on LRM, we further propose LPO, a step-level preference optimization that operates entirely within the noisy latent space. LPO not only achieves substantial training speedups but also delivers remarkable performance improvements across various evaluation metrics, highlighting the effectiveness of employing the diffusion model itself to guide its preference optimization. We hope our findings can open new avenues for research in preference optimization for diffusion models and contribute to advancing the field of visual generation.
Limitations. (1) The Pick-a-Pic dataset mainly contains images generated by SD1.5 and SDXL, which generally exhibit low image quality. Introducing higher-quality images is expected to enhance

---

Table 9: Ablation results on timestep ranges.
Table 10: Ablation results on threshold strategies.
Range of (
P-S
I-R
HP5v2
HPSv2.1
Aesthetic
GenEval
Threshold
P-S
I.R
HPSv2
HPSv2.1
Aesthetic
GenEival
-0.0987
26.25
23.61
5.434
0.3
21.22
0.5122
27.30
27.12
5.853
[250, 450]
20.76 0,1430
26.90
25.37
5.527
43.00
0.4
21.32
0,4789
27.08
26.37
5.832
48.32
[500, 700)
20.95 0.1591
26.71
25.16
5.742
44.44
0.5
21.57
0,5888
27.54
27.42
5.900
48.39
[750, 950]
21.54
0.6337
27.47
27.64
5.853
48.28
0.6
21.35
0.5510
27.25
26.73
5.877
47.97
0.0204
5.573
[0.3,
0.6405
27.55
5.916
[0. 700]
21.02
0.3087
27 10
26.25
5.765
44.93
[0.35, 0.5]
21,69 0,6588
27.64
27.86
5.945
48.39
[0. 950) 21.69 0.6588
27.64
27.86
5.945
48.39
[0.4, 0.55]
21.48 0,4791
27.30
27.13
5.882
48.77
the generalization of the LRM. (2) MPCF relies on three reward models to approximate human preferences for automatic data filtering. However, the filtered data may still inherit biases or limitations shared by these reward models. (3) Since LPO is performed within the latent space, which is determined by the VAE encoder, the VAE of LRM should be identical that of DMO.
Future Work. (1) As a step-level reward model, the LRM can be easily applied to reward fine-tuning methods 45 24 26], avoiding lengthy inference chain backpropagation and significantly accelerating the training speed. (2) The LRM can also extend the best-of-N approach to a step-level version, enabling exploration and selection at each step of image generation, thereby achieving inference-time optimization similar to GPT-ol [66]
Acknowledgments and Disclosure of Funding
This research was supported by the National Natural Science Foundation of China under Grants 62433003 and 62306310.
References
[1] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, and R. Rombach, "Scaling rectified flow transformers for high-resolution image synthesis," in ICML, vol. 235, 2024, pp. 12 106-126333
[2] H. Touvron, Martin, K. Stone, Albert, A. Almahairi, Y. Babaci, N. Bashlykov. S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288 2023.
[3] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten,
A. Yang, A. Fan et al., "The Ilama 3 herd models," arXiv preprint arXiv:4407.21783, 2024.
[4] B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and Naik, "Diffusion model alignment using direct preference optimization," in CVPR, 2024, pp.
[5] R. Rafailov, A. Sharma, E. Mitchell, C. Manning. S. Ermon, and C. Finn, "Direct preference optimization: Your language model is secretly reward model," in NeurIPS, vol. 36, 2023, pp. 53 7387-33741
[6] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine, "Training diffusion models with reinforcement learning," in ICLR, 2024.
[7] K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, W. Shen, X. Zhu, and X. Li, "Using human feedback to fine-tune diffusion models without any reward model," in CVPR, 2024, pp. 8941-8951.
[8] Z. Liang, Y. Yuan, S. Gu, B. Chen, T. Hang, M. Cheng, J. Li, and L. Zheng, "Aesthetic post- training diffusion models from generic preferences with step-by-step preference optimization." arXiv preprint axXiv:2406.043/4 2024.
[9] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision." in ICML, vol. 139, 2021, pp. 8748-8763.
10

---

[10] D. P. Kingma, "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114 2013.
[11] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rom- bach, "SDXL: Improving latent diffusion models for high-resolution image synthesis," arXiv preprint ar arXiv:2307.01992, 2023.
[12] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," " in MICCAI, 2015, pp. 241.
[13] W. Peebles and S. Xie, "Scalable diffusion models with transformers," in ICCV, 2023, pp. 4195-4205.
[14] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in CVPR, 2022, pp. 18684-10695.
[15] Z. Shao, Wang. Q. Zhu, R. Xu, Song, Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu al., "Deepseekmath: Pushing the limits of mathematical reasoning in open language models," arXiv preprint arXiv:2402.00 03300, 2024.
[16] Y. Kirstain, A. Polyak. U. Singer, S. Matiana, J. Penna, and O. Levy, "Pick-a-Pic: An open dataset of user preferences for text-to-image generation," in NeurlPS, vol. 36, 2023, pp. 36 652- 36 663.
[17] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li, "Human preference score v2: solid benchmark for evaluating human preferences of text-to-image synthesis," arXiv preprint arXiv:2306. 09341, 2023.
[18] Xu, X. Liu, Y. Wu, Tong. Q. Li, M. Ding, Tang, and Y. Dong, "ImageReward: Learning and evaluating human preferences for rext-to-image generation," in NeurlPS, vol. 36, 2023, pp. 15 903005 935.
[19] Li, D. Li, C. Xiong, and S. Hoi, "BLIP: Bootstrapping language-image pre-training for unified vision-languag understanding and generation," in ICML, vol. 162, 2022, pp. 12 22888-11 12 900.
[20] S. Zhang, B. Wang, J. Wu, Y. Li, T. Gao, D. Zhang, and Z. Wang, "Learring multi-dimennional human preference for sext-to-image generation," " CVPR, 2024, pp. 8018-8027.
[21] Xu, Y. Huang, Cheng. Yang. Xu, Y. Wang. W. Duan, S. Yang. Q. Jin. S. Li al., "Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation," arXiv preprint 22..2:2412.21053 2024.
[22] Y. Wang, Y. Zang, H. Li, C. Jin, and J. Wang, "Unified reward model for multimodal under- standing and generation," arXiv preprint arXiv:2503.05236, 2025.
[23] Y. Wang, Z. Li, Y. Zang, C. Wang. Q. Lu, C. Jin, and J. Wang. "Unified multimodal chain-of- thought reward model through reinforcement fine-tuning," arXiv preprint arXiv:2505.03336 2025.
[24] K. Clark, P. Vicol, K. Swersky, and D. J. Fleet, "Directly fine-tuning diffusion models on differentiable rewards," ICLR, 2024.
[25] X. Wu, Y. Hao, M. Zhang, K. Sun, Z. Huang, G. Song. Y. Liu, and H. Li, "Deep reward supervisions for tuning text-to-image diffusion models," in ECCV, 2025, pp. 108-124.
[26] D. Jiang, G. Song, X. WI, R. Zhang, D. Shen, Z. Zong, Y. Liu, and H. Li, "CoMat: Align- ing text-to-image diffusion model with image-to-text concept matching," arXiv preprint arXiv:2404 arXiv:2404.03653 2024.
[27] J. Zhang, J. Wu, Y. Ren, X. Xia, H. Kuang, P. Xie, J. Li, X. Xiao, W. Huang, S. Wen, L. Fu, and G. Li, "Unifl: Improve latent diffusion model via unified feedback learning," in NeurlPS, vol. 37, 2024, pp. 67 355-67 382
[28] Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee, "DPOK: Reinforcement learning for fine-tuning text-to-image diffusion models," in NewrIPS, vol. 36, 2023, pp. 79 885.

---

[29] J. Hong, S. Paul, N. Lee, K. Rasul, J. Thome, and J. Jeong, "Margin aware preference opti- mization for aligning diffusion models without reference," arXiv preprint Aklv:24006064244, 2024.
[30] D. Zhang, G. Lan, D.-J. Han, W. Yao, X. Pan, H. Zhang, M. Li, P. Chen, Y. Dong, C. Brinton et al., "Seppo: Semi-policy preference optimization for diffusion alignment," arXiv preprint arXiv:2410.05255, 2024.
[31] X. Zhang, L. Yang, G. Li, Y. Cai, xie jiake, Y. Tang, Y. Yang. M. Wang, and B. CUI, "Iterompp: Iterative composition-aware feedback learning from model gallery for text-to-image generation," in ICLR, 2025.
[32] S. Karthik. H. Coskun, Z. Akata, S. Tulyakov, J. Ren, and A. Kag, "Scalable ranked preference optimization for text-to-image generation." " arXiv preprint arXiv:2410.18013. 2024.
[33] L. Byring, S. Karthik, K. Roth, A. Dosovitskiy, and Z. Akata, "Reno: Enhancing one-step text-to-image models through reward-based noise optimization," in NeurlPS, vol. 37, 2024, pp. 125 529
[34] S. Kim, M. Kim, and D. Park, "Test-time alignment of diffusion models without reward over- optimization." in ICIR, 2025.
[35] Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," in ICLR, 2021.
[36] J. Ho and T. Salimans, "Classifier-fres diffusion guidance," arXiv preprint 2rXiv:227..12555 2022.
[37] R. A. Bradley and M. E. Terry, "Rank analysis of incomplete block designs: the method of paired comparisons," Biometrika, vol. 39, pp. 324-345, 1952.
[38] C. Schuhmann, R. Beaumont, Vencu, Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and Jitsev, "LAION-5B: An open large-scale dataset for training next generation image-text models," in NeurIPS, vol. 35, 2022, pp. 25 278-25 294.
[39] Z. Lin, D. Pathak, B. Li, J. Li, Xia, G. Neubig, Zhang, and D. Ramanan, "Evaluating text-to-visual generation with image-to-tens generation," in ECCV, 2025, pp. 366-384.
[40] D. Ghosh, H. Hajishirzi, and L. Schmidt, "Geneval: An object-focused framework for evaluating text-to-image alignment," in NeurlPS, vol. 36, 2023, pp. 52 132-52 152.
[41] K. Huang, C. Duan, K. Sun, E. Xie, Z. Li, and X. Liu, "T2i-compbench+t: An enhanced and comprehensive benchmark for compositional text-to-image generation," " IEEE TPAMI, vol. 47, pp.
2025.
[42] K. Pearson, "Mathematical contributions to the theory of evolution. iii regression, heredity, and panmixia," Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, pp. 253-31 318, 1896.
[43] B. F. Labs, "Flux," httgss//gtthu..ccoomoaafffssssss
2024.
[44] Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," in NeurIPS, vol. 33, 2020, pp. 6840-6851.
[45] M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki, "Aligning text-to-image diffusion models with reward backpropagation." arXiv preprint arXiv:2310.037339 2023.
[46] "Learning reason with Ilms," OpenAl, Tech. Rep., 2024. [Online]. Available:
http://ppena..com/mmee///eeeeeme
(o-reason- with-llms/
[47] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, I.. Schmidt, and Jitsev, "Reproducible scaling laws for contrastive language-image learning."
in CVPR, 2023, pp. 2818-2823 2829.
12

---

[48] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "LoRA: Low-rank adaptation large language models." in ICLR, 2022.
[49] X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu, "Ella: Equip diffusion models with Ilm for enhanced semantic alignment," arXiv preprint arXiv:403.005135 2024.
[50] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, "Flow matching for generative modeling," in ICLR, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately summarize proposed methods and their performance, as well as their impact on the research field.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations and future work the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
This paper provides clear description of the LRM architecture and training pipeline, the sampling and training procedures LPO, the evaluation protocols, and the specific hyperparameter settings, in the method, experiment, and appendix sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper provides the complete code with a detailed README file to facilitate reproduction of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudoueo Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper provides detailed description of the training and details, as well as necessary ablation studies of important hyperparameters.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due the computational constraints, bars not included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper depicts the number and type of computing devices, as well as the time of execution in the experiment and appendix sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: respect.
We have reviewed the NeurlPS Code of Ethics and conform to it in every
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses both potential positive and negative societal impacts in the last section of the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: This paper discusses the risk of model misuse in the broader impact section and will require users to comply with usage guidelines and relevant laws.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper cites the original paper that produced code packages and datasets used in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details about the model training, the method limitation, and the licenses of this work and the corresponding code are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Experimantal Details
LRM Training. We train LRM-1.5 for 4,000 steps with 500 warmup steps and LRM-XL for 8,000
steps with 1000 warmup steps, both using the learning rate le-5. The batch size is 128 for LRM-1.5 and 32 for LRM-XL, respectively. Following CLIP [9], the initial value of r in Eqn. (9) is set to e2.6592, The training resolution is 512 for SD1.5 and 1024 for SDXL. For SDXL, which includes two text encoders, we utilize only the OpenCLIP ViT-bigG 47 as the text encoder of LRM-XI..
LPO Training. Both SD1.5 and SDXL are fine-tuned for 5 epochs. Following SPO 8 we employ LoRA 48 to fine-tune models. The LoRA rank is 4 for SD1.5 and 64 for SDXL. All experiments are conducted using A 100. Other hyperparameters are provided Tab. The experimental setting of SD2.1 1 is the same as SD1.5.
Table 11: Hyperparameters of LPO training.
SD1.5
14
SDXL
Learning Rate
5e-5
le-5
LoRA Rank
4
64
500
500
K Sampling Threshold Range
[0.35, 0.5]
[0.45, 0.6]
Sampling Batch Size Training Resolution
5
4
512x512
x024x 1024
Training Epoch
5
5
Training Batch Size
10
Baseline Methods. Specifically, SPO 8 and LPO are trained on Pick-a-Pic v1 [16] while Diffusion- DPO 4, ScPPO [30], and MaPO 29 are trained Pick-a-Pic v2 [16], extended version of Pick-a-Pic v1. Pick-a-Pic v1 consists of 583,737 preference pairs, whereas Pick-a-Pic v2 contains over 950,000 pairs. In the practice of SPO, DDPO (6) and D3PO (7) are reproduced on 4k prompts randomly sampled from Pick-a-Pic vI, which are the same as the training data used for SPO and LPO. Therefore, the comparison between these methods can be considered fair. Furthermore, for reference purposes, we also make a comparison with InterComp [31] Instead of Pick-a-Pic, InterComp utilizes an internally constructed preference dataset of higher quality, including images generated by more advanced models such as SD3 [ and FLUX [43]. In contrast, the data in Pick-a-Pic only consists of images from models like SDXL [1] and SD2.1 [4]. Despite the relatively low-quality training data, LPO still performs slightly better than InterComp, demonstrating the effectiveness of our method.
Evaluation Practice. For all baseline methods, we use the official models for evaluation without retraining. For the evaluation of text-image alignment, we utilize official codebases of GenEval 40 and T21-CompBench++ 44. Following SPO, we employ several public preference models to assess general and aesthetic preferences, including PickScore (16), ImageReward [18) HPSv2 [17. IPSv2. 177, and Aesthetic Predictor (38] Since SPO does not provide the evaluation code, implement our pipeline, adhering to SPO's setting: using the DDIM (35) scheduler with 20 inference steps and the same random seed ensure reproducibility. We apply the same code for all models to ensure fair comparison. Although there are some discrepancies with SPO's reported results, we believe the relative comparisons between different models are reliable and meaningful.
B Additional Exploration Experiments
User Study. We conduct human evaluation experiments for LPO, SPO, and Diffusion-DPO, The evaluation set consists of 200 prompts, with 100 randomly sampled from HPSv2 benchmark [17] and 100 randomly sampled from DPG-Bench 44). For each prompt, the model generates four images. For each image, five expert evaluators score it across three dimensions: general preference, visual appeal, and prompt alignment. Subsequently. votes are cast to determine the winning relationships among the di different models. The results are illustrated in Fig. 6 Similar to the results automatic metrics in the main paper, LPO outperforms both SPO and Diffusion-DPO across three dimensions. Compared to T2I-CompBench and GenEval results, the advantage of LPO on text-image alignment in the human evaluation results is not particularly pronounced. We believe this is because the prompts
21

---

General Preference
69.50
LPO Win
Visual Appeal
69.50
30.50
Diffusion-DPO Win
Prompt Alignment
54.50
45.50
20
40
60
80
100
Win
General Preference
58,00
42.00
Visual Appeal
56.00
44.00
SPO Win
Prompt Alignment
56.50
43.50
20
40
80
100
Win Rates
Figure 6: User study results on 200 prompts from HPSv2 benchmark 11 and DPG-Bench [49]
Table 12: The performance of different preference optimization methods based on LRM. The general and aesthetic scores calculated on the Pick-a-Pic validation unique set. We use 20 inference steps.
GenEval Training Time + (20 Step)
Model
Method
PickScore ImageReward HPSv2
HPSv2.1
Aesthetic
(A100) hour)
Original
20.56
0.0076
26.46
24.05
5.468
42.56
SD1.5
12
LPO
21.69
0.6588
27.64
27.86
5.945
48.39
Step-Wist Wise GRPO
21.52
0,5861
27.47
28.35
6.087
45.88
Table 13: General and aesthetic preference scores on Pick-a-Pic validation unique set, along with GenEval 40) scores. We 20 inference steps.
Model
Method
Dataset
PickScore
ImageReward
HPSv2
HPSv2.1
Aesthetic
Original
21.74
0.9527
28.27
28.74
5.741
SD3-Medium
LPO
Pick-a-Pic v1 [16
22.29
1.1306
28.64
30.44
5.993
Table 14: Qualitative results on GenEval 44
with 20 inference steps based on SD3-Me Mediim.
Model
Method
Single Object
Two Object
Counting Colors Position Color Attribution Overall
Original
98. 13
81.57
57.81 60.63
82.98
24.50 26.25
58.75 54.75
67.29
SD3-Medium
LPO
100.00
86.11
84.04
68.63
used for human evaluation not specifically designed to assess text-image alignment, and different models exhibit relatively good alignment.
Exploration of GRPO Based on LRM. Actually, LRM is not strictly bound to LPO, and it can also be applied other preference optimization methods. Inspired by DeepSeekMath [5]. we implement step-wise GRPO algorithm for preference optimization in the image domain and use LRM as the reward model. Specifically, at each denoising step from Tz+1 to It, we sample group of x i 1, ..., K according to Eqn. (2). The LRM is employed to provide rewards R. {r/. ..., K} for
these noisy latent images. We then normalize these rewards to calculate corresponding advantages by (B-mean(RR)) Finally, we optimize diffusion models by minimizing the following objective: sta
A
LGRPO = Terro==Blp B[ffi] ~ pe(
pe (x (1)1,c)
clip po(z)|
Poana (2) (1)), -,, e, +e)+ Ai Ai- BLD www.eellleee spellpers] } (13)
min
Palaa (1r (
conduct preliminary experiments of wiee GRPO on the SDI.5 model. We set K to 4, 3 to 0.1 and to 0.1. As shown in Tab. 12] based our LRM-1.5, step-wise GRPO consistently improves the model's performance across various dimensions. However, in the current setting, step-wise GRPO performs slightly worse than LPO and requires 8x the training time. We will further explore the application of L.RM-based GRPO for image-domain preference optimization in future work.
Exploration on DiT-Based Models with Flow Matching Methods (SD3). In this paper, we mainly conduct experiments on U-Ner-based models with the DDPM 44 scheduling method. Here we explore the effecttivnness of LRM and LPO on baTebased models with the Flow Matching method
22

---

Table 15: Performance of LPO on HPDv2. General and aesthetic preference scores on Pick-a-Pic validation unique set, along with GenEval 440 scores. not specified, we use 20 inference steps.
GenEval GenEval (20 Step) (50 Step)
Model
Method
Dataset
PickScore ImageReward HPSv2 HPSv2.1
Aesthetic
Original
20.56
0.0076
26.46
24.05
5.468
42.56 47.29
42.53
SDI.5 14
LPO
HPDv2 17
21.24
0.7248
28.13
28.93
5.917
48.27
Table 16: Performance of LPO on HPDv2. General and aesthetic preference scores on 400 randomly sampled HPDv2 test prompts with 20 inference steps.
Model
Method
Dataset
PickScore
ImageReward
HPSv2
HPSv2.
Aesthetic
Original
20.56
0.0076
26.46
24.05
5.468
SD1.5 14
LPO
HPDv2 17
21.61
0.7282
28.81
29.25
6.019
Table 17: Performance of LPO on HPDv2. Quantitative results on 121-CompBench++
with 20
inference steps. LPO is trained on the HPDv2 [17] dataset.
Model
Method
Color
Shape
Texture
2D-Spatial 3D-Spatial Numeracy Non-Spatial Complex
Original 0.3783 0.3616 0.4172
0.1230
0.2967
0.4485
0.3104
0.2999
SDI.5
14
LPO
0.5761
0.5067 0.6013
0.1671
0.3345
0.4671
0.3035
0.3332
Firstly, based on SD3-medium M. we train the LRM on Pick-a-Pic vl [16]. We extract visual features from the 18th layer of MMDiT [ and utilize text features from all text encoders. The encoders of the SD-medium are frozen during training. Secondly, to provide the exploration space for
50].
LPO, we transform Flow Matching from ODE (Ordinary Differential Equation) to SDE (Stochastic Differential Equation). Specifically, we replace predicted noise c0,r in Eqn. (3) with predicted velocity Uge using the relation and Eqn. (4) with af and of in the forward process of Flow Matching, ie., de = 0/20 + ofc. Then the backward process can be reformulated as:
= + a e. Additionally, we substitute vāz and vI - āg in Equ. (3)
Ft-1 d0 d(v.e) + V (0 of(a/vo, 21) + oien & N(O,),
(14)
(
2
0g 0
2
o "
1
(15)
Finally, we employ LPO SD3-medium. As shown in Tab. 13 and Tab. 14 even though the training data Pick-a-Pie only contains images from SDXL and SD2.T, LPO still demonstrates notable performance gains various metrics. We believe that if there is higher-quality training data, such as that in InterComp, the performance gain will be more pronounced.
Experimental Results on HPDv2. To validate the generalization of LRM and LPO, we conduct experiments on the HPDv2 [17 dataset without any hyperparameter optimization. Specifically, we train LRM on the entire HPDv2 training set and then randomly sample 4,000 prompts from this set for LPO training. In addition to the previous evaluation metrics, we also randomly sample 400 prompts from the HPDv2 test set and calculate the general and aesthetic scores for these prompts. As shown in Tab. 15 Tab. 16 and Tab. 17 LPO significantly outperforms the original models across various evaluation dimensions.
LRM as Pixel-Wiss Reward Model. The LRM can also serve as a pixel-wise reward model when combined with the corresponding VAE encoder. Tab, [8 shows the correlations of different reward models with aesthetics and text-image alignment. Aes-Corr is employed to assess the correlation with aesthetics, while the CLIP-Corr and VQA-Corr are utilized to measure alignment correlation. The calculation method for VQA-Corr similar that of CLIP-Corr, as described in Sec. 5.1 with the CLIP Score replaced by VQAScore. is observed that the LRMs, especially LRM-XL, exhibit aesthetic and alignment correlations comparable to those of VLM--bssed models, as indicated by similar Aes-Corr and VQA-Corr values. The CLIP-Corr of LRMs is slightly lower than the average value of VL.M-based reward models, likely because VL.M-baeed models are fine-tuned versions of CLIP 9 or BLIP [9). leading to higher similarity to CLIP. The accuracy on validation and test sets
23

---

Table 18: The correlations of reward models with aesthetics and text-image alignment, along with the preference prediction accuracy on the validation and test set of Pick-a-Pic vl.
Model
Aes-Corr CLIP-Corr VQA-Corr Val-Test Accuracy
Aesthetic
38
54.03
Specific Reward Model
CLIP Score (CLIP-H) 9
61.84
(VLM-Based)
VQAScore
39
59.16
ImwgeReward
18
0.108
0.425
0.417
62.66
HPSv2 17
0.007
0.602
0.406
64.76
General Reward Model
HPSv2.1
17
0,191
0.432
0.332
65,58
(VI.M-Based)
PickScore [6
0,066
0,490
0,402
71.93
Average
0.093
0.487
0.389
LRM-1.55 0)
0.115
0.359
0.339
65.46
General Reward Model
LRM-1.5 1.5 (L 200)
0.111
0.356
0.336
67.21
(Diffusion-Bare -Based)
LRM-XL (L = 0)
0.073
0,403
0.390
67.44
LRM-XL (L = 200)
0.089
0.401
0.385
69.31
Table 19: Comparison between SPO and LPO without MPCF. The general and aesthetic scores
calculated on the Pick-a-Pic validation unique set. If not specified, we use 20 inference steps.
GenEval (20 Step) (50 Step)
GenLival
Model
Method
PickScore
ImageReward HPSv2
HPSv2.1
Aesthetic
SPO
21.22 21.49
0.1678
26.73
25.83
5.927
40.46 45.66
41.53
SD1.5
14
LPO wo MPCF
0,4406
27.39
27.61
5.772
46.21
Table 20: Comparison results 1221-ComBeench++
41
with 20 inference steps between SPO and
LPO without MPCF.
Model
Method
Color
Shape
Texture
2D-Spatial 3D-Spatial Numeracy Non-Spatial Complex
SPO
0.4112
0.4019
0.4044
0.1301
0.2909
0.4372
0.3008
0.2988
SD1.5
14
LPO
0.4717
0.4002
0.4846
0.1793
0.3460
0.4596
0.3094
0.3171
of Pick-a-Pic is also present in Tab. 18 LRMs exhibit competitive performance, with larger variants (LRM-XL) achieving higher accuracy. Moreover, we observe that adding slight noise to the latent images (t 200) leads to improved accuracy, which may be attributed to the fact that slight noise can better activate the feature extraction capabilities and improve the generalization diffusion models.
C Additional Ablation Experiments
Comparison between SPO and LPO without MPCF. We conduct comparison between SPO and LPO on identical training data. Specifically, we compare SPO with LPO without the MPCF strategy. As illustrated in Tab. [99 and Tab, 20 LPO without MPCF still outperforms SPO, especially in general and text-image alignment preference, demonstrating the effectiveness of reward modeling and preference optimization the noisy latent space.
The Compatibility with High Noise at Large Timesteps. In Tab. 21 we analyze the compatibility of SPO and LPO with large training timesteps and high noises. Notably, for SPO, the training efficiency within the timestep range [0, 950] is significantly lower than that at [0, 750], and it is even inferior the original model. In contrast, LPO optimization with this range remarkably enhances the model's ss performance. Furthermore, training only with the range [750, 950] can even achieve comparable performance to that of the range [0, 950], illustrating the superior adaptability of LRM to high-intensity noise at large timesteps.
investigate the timestep sensitivity of LRM. When LRM is consistently provided with a fixed timestep during LPO sampling, the model's performance degrades significantly. This highlights the importance of timestep input and confirms that LRM effectively leverages timestep information to predict preferences for noisy latent images.
LRM's Sensitivity to Timesteps. In Tab.
Regularization HyperParameter B. 9 is regularization hyperparameter to control the deviation
of the optimized model (po in Eqn. (6) with the reference model Pref- As shown in Tab. 23 as increases, the regularization effect becomes stronger, which slows down the model's optimization

---

Table 21: Comparison between SPO and LPO with different training timestep ranges. We use 20 inference steps. denotes the results quoted from
Model
Method
Range of t
PickScore
ImageReward
HPSv2
IPPv2.1
Aesthetic
GenEval
original
20.56
0.0076
26.46
24.05
5,468
42.56
[0, 750]
21.43
0.1712
26.45
5,887
SPO'
SDL.5
14
[0, 950]
19.77
-0.4529
22.72
5.111
[0, 700]
21.02
0.3087
27.10
26.25
5.765
44.93
LPO
[750, 950]
21.54
0.6337
27.47
27.64
5.853
48.28
[0, 950]
21.69
0.6588
27.64
27.86
5.945
48.39
Table 22: Comparison of timestep inputs of LRM during LPO sampling. We use 20 inference steps.
Model
Input Timestep
PickScore
ImageReward
HPSv2
HPSv2.1
Aesthetic
GenEval
Real Timestep
21.69
0.6588 -0.5043
27.64
27.86
5.945
48,39
SD1.5
14
19.64
25.37
21.74
5.283
32.21
Table 23: Ablation results the hyperparamter B.
B
Aesthetic
GenEval
PickScore
ImageReward
HPSv2
HPSv2.
20
5.920
46.95
21.53
0.6407
27.54
28.13
100
6.031
48.23
21.68
0,6443
27.47
27.52
500
5.945
48.39
21.69
0.6588
27.64
27.86
1000
5.858
48.44
21.39
0.4785
27.22
26.47
5000
5.647
43.87
20.95
0,2970
27.00
25.68
speed and leads to poorer performance. Conversely, if 3 is too small (3 = 20), the regularization constraint becomes too weak, potentially harming the model's generalization and reducing image quality. The performance is optimal when 3 is within the range of 100 to 500.
Standard Deviation
Constant (0.5) Dynamic ((0.35-0.5))
0.500
Variance
4000
0.475
Timestep
Threshold
Sample Num
0.425
2000
0.403
0.375
1000
500
Timesteps
Figure 7: The thresholds of different strategies.
Figure 8: The number of valid samples of differ- timesteps.
Other Dynamic Threshold Strategy. In the main paper, we dynamically correlate the threshold in LPO sampling with the standard deviation a; in DDPM [4].. Here we explore alternative strategies. The first strategy involves replacing o, with variance of Consequently, the threshold can be formulated as:
the of o2 min
(thimar
thtmen) thmin-
(16)
? o2 mii
The second strategy is linearly correlating the threshold with the timestep t:
Emán
the Eyna tmin
(thmaz
thmin)
(17)
where fmin and tmar denote the minimum and maximum optimization timesteps. respectively.
The values different strategies are illustrated in Fig. 7 The threshold range [0.35, 0.5] and the timestep range is [0, 950). It can be observed that the standard deviation strategy sets higher thresholds for middle timesteps, while the variance strategy sets lower thresholds. shown in Tab. 24 the standard deviation strategy achieves the best performance across various metries. This
25

---

Table 24: Ablation results on the dynamic threshold strategies.
Strategy
Aesthetic
GenEval
PickScore
ImageReward
HPSv2
HPSv2.1
Standard Deviation
5,945
48.39
21.69
0.6588
27.64
27.86
Variance
5,921
47.78
21.57
0,5054
27.42
26.93
Timestep
5.929
48.22
21.65
0.6465
27.42
27.25
Table 25: Complete results on GenEval [40
with 20 inference steps.
Model
Method
Single Object
Two Object Counting Colors Position Color Attribution Overall
Original
97.50
37.12
34.69
75.53
3.75
6.75
42.56
Dift--DPO
98.44
38.38
36.25
77.93
4.50
7.25
43.79
SD1.5
14
SPO 8
95.00
33,84
32.50
69.95
4.25
7.25
40.46
SePPO
30
99.06
39.90
34.69
81.91
6.25
8.00
44.97
LPO
97.81
54.80
40.94
79.52
7.00
10.25
48.39
Original
93.75
63.38
30.94
80.05
9.25
19.00
49.40
Dift--DPO
99.06
76.52
45.00
88.83
11.50
25.75
57.78
SDXL
|||
MaPO 29
95.63
68.94
32.19
83.51
11.50
17.75
51.59
SPO
94.38
69.44
31,88
81.65
10.25
15.50
50.52
InterComp
99.06
82.83
43.75
84.04
14.00
31.75
59.24
I.PO
99.69
81.57
43.75
89.10
14.00
27.50
59.27
Table 26: Complete results on GenEval
440
with 50 inference steps.
Model
Method
Single Object
Two Object Counting Colors Position Color Attribution Overall
Original Diff.-DPO
98.13
37.88
33.13
76.33
3.75
6.00
42.53
98.13
41.16
37.81
81.91
4.50
7.25
45.13
SD1.5
SPO
95.63
36.62
34.83
72.34
3.75
6.50
41.53
SePPO
30
98.13
41.16
33.44
79.52
7.25
7.25
44.46
LPO
97.81
55.30
42.19
80.59
6.75
10.00
48.77
Original
94.38
67.68
41.56
81.65
10.50
18.00
52.29
Diff--DPO
99.06
77.78
49.69
86.17
13.25
27.50
58,91
SDXL 11
MaPO
29
96.56
66.41
40.00
84.31
10.75
18.75
52.80
SPO"
97.81
73.48
41.25
85.64
13.00
20,00
55.20
SPO
8
96.88
69.70
37.19
83.51
9.50
19.75
52.75
InterComp 99.06
85.10
41.25
86.97
13.50
32.00
$9.64
LPO
99.69
84.34
43.13
90.43
13.75
27.75
59.85
may because the standard deviation strategy better accommodates the variations between dy at different timesteps.
Fig. g illustrates the number valid samples of different timesteps for both constant and dynamic (standard deviation) threshold strategies. The dynamic threshold strategy significantly increases the number valid samples, especially intermediate timesteps.
Complete Results on GenEval. The complete GenEval results corresponding with that in Tab. are
presented in Tab. 25 and Tab. 26
D Additional Visualizations
Predicted Images J, at Different Timesteps. Fig. Gillustrates the predicted images T, generated through the diffusion forward and VAE decoding processes, as shown in Fig. 4 (a). As discussed in the main paper, predicted images at large timesteps tend be very blurred, causing significant distribution shift from the original images. This makes it challenging for PRMs to adapt to these blurred images without adequate pre-training and extensive datasets, resulting in unreliable predictions at large timesteps. In contrast, LRM can naturally perceive noisy latent images even at large timesteps, it does during pre-training, representing significant advantage over PRMs.
Optimization Timesteps. The generated images different optimization timestep ranges are illustrated in Fig. 10 Larger timestep ranges result in more pronounced improvements in the quality of the generated images. We think there are two main reasons. Firstly, small timesteps in the denoising process primarily focus high high-frequency details, which do not lead to significant changes
26

---

Original
CRTTA
COYRA
CLYA
COUI
Figure 9: Predicted images I in Fig. (a) at different timesteps. The original images come from the
Pick-a-Pic v1 dataset [16].
in the layout and style of the image. Secondly, indicated in Fig. 8 the smaller the timestep. the fewer the valid samples, resulting in less optimization at the corresponding timesteps. Furthermore, compared images in the range [750, 950), those in [0, 950] exhibit richer details, including both foreground and background, which also demonstrates that optimization at smaller timesteps aids in the enhancement of image details.
More Comparison. Fig. II presents the generated images of different methods based on SD1.5 [1].
Fig. 12 and Fig, 13 show the larger version of images in Fig. [ with corresponding prompts. 14 and Fig. [ provide more generated images various methods based on SDXL [1I]. Some keywords in prompts that other models fail to depict highlighted using bold formatting.
Fig.
E
Additional Discussions
During the research process, observed the following types of failure cases:
Reward hacking: When training steps excessive, the model exhibits reward hacking, where the reward metric continues to improve while the quality of generated images degrades. This stems from a certain misalignment between the reward model and human preferences, which remains common issue across most existing methods. Mitigating this problem requires a combination of strategies, including improvements in preference data and reward modeling approaches.
Decline in diversity: Extended training also leads reduction in generation diversity. This is caused by the reward model narrowing the output distribution of the diffusion model. A general solution is prompt engincering, such as rephrasing input prompts enhance the diversity of generated images via prompt variation.
Lack of fine details on SDXL: For some prompts, SDXL models optimized with LPO generate images with less detailed content compared those optimized with SPO. This phenomenon was not observed on SD1.5. We will investigate this in future work to identify the underlying causes.
27

---

Broader Impacts
This work introduces preference optimization method for text-to-image diffusion models. The method may have the following impacts:
The reward model plays crucial role in shaping the preferences of diffusion models. However, if the training data for the reward model contains biases, the optimized model may inherit or even amplify these biases, leading to the generation of stereotypical discriminatory content about certain groups. To mitigate this risk, is essential to ensure that the training data is diverse, representative, and fair.
This method offers way enhance the quality of generated images in terms of aesthetics, relevance, and other aspects by leveraging well-designed reward model. It can also be applied to safety domains, optimizing the model to prevent the generation of negative content.
The optimized model can generate highly realistic images, which may be used to create misinfor- mation misleading content, thereby impacting public opinion and social trust. Therefore, is essential to develop effective detection tools and mechanisms to identify synthetic content.
28

---

Original SDXL
[0, 200]
[250, 450]
[500, 700]
[750, 950]
[0, 950]
An some woman
attractve
giant bagie
man watching the sunset
monk robe by round spaceship in dhamatic ophoina
medium shot, wearing antraal hide sacket. The artsst hax masterfully rendered character digital capturing the intricate detais of for and clothing coxturo
Figure 10: Qualitative comparison of various optimization timestep ranges based SDXL.
29

---

Original SD1.5
Diffusion-DPO
SPO
LPO
brown monkey with playful expression pair of aviator glasses swings through lush lushjungle jungle canopy. its tail curis around vine, and the sunlight reflects off its sunglasses, capturing of adventure and
an all painting depicts starry night in the desert, with red astronaut staring Intently at blue astronaut who clutches a knife. the vast desert stretches out before them, dunes rippling moonlight, the sky above is filled with twinkling stars, while the distant horizon glows faintly from the setting sun, tension between two astronauts is palpable, as dramatic confrontation about to unfold.
Close-up portrait middie-aged man, illuminated by soft natural light. face is deeply lined and show emotion, expressing sadness and contemplation. Shot using 85mm lens. the focus his facial features while the background is softly blurred. Soft lighting creates subtle highlights and shadows accentuate the skin's texture.
A dolphin jumping rowboat
Figure 11: Qualitative comparison of various preference optimization methods based SD1.5.
30

---

Original SDXL
Diffusion- DPO
SPO
LPO
A house in the style of Escher
cinematic still of stainless steel robot swimming a pool
An anime color style
The green plant was on the left of the white lamp
Corgi with helmet bicycle
Figure 12: Qualitative comparison of various preference optimization methods based SDXL.
31

---

Original SDXL
Diffusion-DPO
SPO
LPO
Saturn rises on the horizon
A hot female Alex from Minecraft
Mushroom concept car, mushroom shape, alien technology, interstellar transportation, cutting- white color
edge technology, plasma
HPP
PPP
HAARD
Ha caleddar
2DY
happy
DOUP!Y
vibrant, smiling calendar mascot leaps air glee, holding pen marking days. mascot is surrounded by a confetti explosion of dates and months, alluding to the joy and excitement scheduling events. mascot, elegant logo reads happy calendar friendly, modern font. Figure 13: Qualitative comparison of various preference optimization methods based SDXL.
32

---

Original SDXL
Diffusion- DPO
SPO
a majestic dog-cal chimaera is perched on a moss-covered rock in an enchanted forest, with feline face and canine body seamlessly blended. it sports luxurious mane and bushy tail, while vibrant, ethereal lights dance around it, enhancing the digital fantasy atmosphere.
8000
sleek, silver casio watch with digital display sits atop polished wooden surface. its metallic band catches sunlight streaming through nearby window, casting a warm glow on the watch face. the time reads 12:34 pm, while the stopwatch feature ticks away the background.
Einstein's famous photo, Van Gogh's Starry Night style, the oil painting replaces the characters in Van Gogh's self-portrait with Einstein's s portrait, masterpiece
A smiling beautiful sorceress wearing high necked blue suit surrounded by swirling rainbow aurora, hyper-realistic, cinematic, post-production
Wool felt texture, at night, under starry sky, the forest, little girl wearing yellow clothes, with cute hairpins, squats on the ground, stroking deer, tiny particles glow, perfect composition.
Figure 14: Qualitative comparison of various preference optimization methods based on SDXI.
33

---

Original SDXL
Diffusion-DPO
SPO
LPO
sculpture sitting posture showing iconic bow, classic round face design including beard and round nose, finely carved fish patterns on the transparent geometric shape
Hello Kitty sculpture, crystal
image the beloved character. Stepped base, winter park environment background, ground covered with snow, cloudy day with diffuse light, figures front showing size proportions, transparent ice showing internal light refraction, professional photography perspective, ultra-high definition image quality, magnificent scale, Exquisite ice sculpture craftsmanship, winter festival atmosphere
A woman wearing an ultramarine jacket and yellow slacks, yellow socks and yellow shoes, wearing all leash
wavy hair without bangs, holding black dog with her hands, the dog in her Right, fashion model, red lipstick, on an asphait road with yellow line, single yellow background, oriental fusion clothing, angle, like fashion pictorial, concept art, decorative, hyper-detailed, realistic, bright colors, realistic, model posture
black sunglasses,
weary unicom grad student sits cluttered wooden desk, surrounded by stacks of books and papers. the light small lamp illuminates under the unlcorn's eyes, as it holds a cup of coffee in one hoof, trying to stay awake while studying. the moonlight from the window reflects off its horn, adding magical touch the
room, casting shadows
walls,
evident
detailed sketch his signature red cap and overalls, white gloves, and holds green turtle shell. the drawing captures his cheerful expression and mustache, fine pencii adding depth and dimension.
game is depicted textured parchment.
Figure 15: Qualitative comparison of various preference optimization methods based SDXL.
34


Reviews:
[{"review_id": "tJd2DSGriu", "reviewer": "Reviewer_aRG8", "summary": "The authors proposed LPO, an novel algorithm that leverages latent diffusion backbone as a powerful and efficient reward model that learns to rank noisy images in the latent space. Sampling inference strategies including multi-preference filtering, dynamic thresholding has been introduced for efficient online sampling. LPO achieved superior results comparing to multiple baselines in both general image quality and human preference alignment on SD1.5, SDXL. Further studies also demonstrated that LPO can be generalized to flow matching objective and DiT structure.", "questions": "* Do the authors decide MPCF strategies based on heuristics? Do you think it's possible to develop a principled learned MPCF filtering pipeline?\n\n* Pick a Pic have 5M image pairs, may I know how long does it take to filter the image pairs?\n\n* Table 19 and Table 20 seems a bit unfair comparison as pick score has a smaller size with CLIP + small MLP, while even smallest SD1.5 LRM is CLIP + Unet. It's hard to tell how much latent space reward modeling itself improves alignment performance.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "* Strengths:\n  * The authors conduct comprehensive experiments across multiple evaluation metrics of T2I image synthesis and verify the proposed method with base T2I models covering various architectures and training objectives\n * LPO demonstrated robust and superior results against baselines\n\n* Weaknesses:\n  * The novelty of the proposed method is limited, the proposed method essentially leverages the SD-VAE to allow the reward model training to latent space and employed stronger filtering mechanism to further improve alignment performance, the technical contribution is limited.\n * Poor generalizability: Even though LRM 1.5 was proved effective on SD2.1, the success can be attribute more to the similar capability of the two models as LRM 1.5 was ineffective for SDXL. Therefore, the proposed method needs to fine-tune diffusion backbone as reward model before alignment if the base model differs significantly from a pre-trained LRM, adding complexity in potential tuning to the post-training pipeline.", "was_revised": true, "final_justification": "See my comments", "rebuttal": "Thanks for your effort in reviewing our paper and giving appreciation regarding our methods and experiments. We hope the following responses solve your concerns.\n\n> Q1 (Weakness 1): The novelty of the proposed method is limited, the proposed method essentially leverages the SD-VAE to allow the reward model training to latent space and employed stronger filtering mechanism to further improve alignment performance, the technical contribution is limited.\n\nR1: Thanks for your question. There may be some misunderstanding of the novelty and contributions of our work. We respond from the following aspects:\n\n- First, the proposed LRM does more than shift reward model training from the pixel space to the clean latent space via the VAE. More importantly, **LRM leverages the noise-aware ability of pre-trained diffusion models to shift the reward model training from the clean latent space to the noisy latent space**. This enables LRM to understand noisy latent images $x_t$, resulting in significantly better noise compatibility (as shown in Table 21), and faster reward model training and preference optimization (Tables 5 and 6).\n\n- Second, while LLM-as-a-judge [1] has been widely adopted in text generation, in image generation, our work is the **first** to use the diffusion model itself for reward modeling and preference optimization. **It demonstrates superior effectiveness and efficiency compared to VLM-based methods, as appreciated by Reviewer Xmnx, rhE9, and 2Nf4**. Just as diffusion models have opened new directions in language modeling [2], our work pioneers a novel path in reward modeling for visual generation. We believe this area holds great potential and deserves further exploration.\n\n- Third, although MPCF is a filtering strategy, it reflects our in-depth thinking on step-level reward modeling. SPO [3] assumes that preferences vary across timesteps. However, the reward model training of SPO assumes preferences remain unchanged when noise is added to image pairs at various timesteps, creating an **evident conflict**. Based on this observation, we propose the MPCF strategy to filter the data and mitigate the conflict between the assumptions of step-wise preference optimization and reward model training.\n\nIn summary, we sincerely hope you may reconsider the novelty and contributions of this work to the research community.\n\n\n\n> Q2 (Weakness 2): Poor generalizability: Even though LRM 1.5 was proved effective on SD2.1, the success can be attribute more to the similar capability of the two models as LRM 1.5 was ineffective for SDXL. Therefore, the proposed method needs to fine-tune diffusion backbone as reward model before alignment if the base model differs significantly from a pre-trained LRM, adding complexity in potential tuning to the post-training pipeline.\n\nR2: Thanks for your question. As stated in Lines 238–239, the VAE of LRM must be identical to that of the diffusion model to be optimized. Therefore, LRM-1.5 is effective on SD2.1 (same VAE), but ineffective on SDXL (different VAE). Regarding the need to fine-tune a reward model for each diffusion model, we respond from the following two perspectives:\n\n- First, for step-level reward models, fine-tuning is necessary when the base diffusion models differ significantly. Even at the same timestep *t*, the distribution of $x_t / \\hat{x}\\_{0,t} / I_t$ for the same prompt differs across models. Therefore, the reward model must be aligned with the $x_t / \\hat{x}\\_{0,t} / I_t$ distribution of each diffusion model. **For example, SPO [3] also trains separate SPMs for SD1.5 and SDXL**.\n\n- Second, even for reward models that only take $I_0$ as input, the image distributions generated by different diffusion models can vary considerably. To our knowledge, in current industrial practice [4,5], **it is common to separately fine-tune reward models using images sampled from the current diffusion model to be optimized**. This helps adapt the reward model to the image distribution of the current diffusion model and enables mitigation of model-specific failure cases. It is uncommon to use the same reward model to optimize significantly different diffusion models simultaneously.\n\nConsidering the above, **we believe that LRM does not introduce significant additional training complexity compared to other reward models.** In addition, extensive experiments in the paper demonstrate that **our method generalizes well across different model architectures (SD1.5, SDXL, SD3), datasets (Pick-a-Pic, HPDv2), and optimization methods (LPO, GRPO).**\n\n\n\n> Q3 (Question 1): Do the authors decide MPCF strategies based on heuristics? Do you think it's possible to develop a principled learned MPCF filtering pipeline?\n\nR3: Thanks for your valuable question. MPCF employs three pre-trained public reward models to filter data, with the threshold for the score gap $ G_* $ chosen according to experimental results. In this work, we select $ G_* $ by evaluating the performance of LRM trained on the filtered data. Our goal is to encourage LRM to attend to both image aesthetics and image-text alignment, as indicated by Aes-Corr, CLIP-Corr, and the performance of models optimized via LRM and LPO. We would greatly appreciate it if you could provide more details about \"a principled learned MPCF filtering pipeline,\" and we are willing to follow up on your further suggestions.\n\n\n\n> Q4 (Question 2): Pick a Pic have 5M image pairs, may I know how long does it take to filter the image pairs?\n\nR4: Thanks for your question. In fact, the Pick-a-Pic v1 dataset [6] contains 0.58M image pairs, not 5M, as stated in Lines 204–205. For MPCF filtering, we used 8 A100 GPUs: the Aesthetic Predictor took 0.7 hours, CLIPScore 0.9 hours, and VQAScore 5 hours, resulting in a total of 52.8  ($(0.7+0.9+5)\\times8$) A100 GPU hours. Notably, this is a one-time cost, and the filtered data can be reused across different model architectures and hyperparameter settings, including SD1.5, SDXL, and SD3. **Considering the speed comparison in Table 6, even after including this one-time computational cost, the overall training time of our method remains lower than that of Diffusion-DPO and SPO**.\n\n\n\n> Q5 (Question 3): Table 19 and Table 20 seems a bit unfair comparison as pick score has a smaller size with CLIP + small MLP, while even smallest SD1.5 LRM is CLIP + Unet. It's hard to tell how much latent space reward modeling itself improves alignment performance.\n\nR5: Thanks for your question. **In fact, the parameter count of LRM-1.5 is less than half that of PickScore**. We respond from the following two perspectives. \n\n- PickScore [6] and SPM in SPO [3] are both finetuned from CLIP-H, whose vision and text encoders together contain over **980M** parameters. In contrast, LRM-1.5 uses the CLIP-L text encoder (123M parameters) and only the down and mid blocks of the U-Net (346M parameters), resulting in a total of **~470M** parameters, **which is less than half the size of PickScore**.\n\n|Model|Number of Parameters|\n|-|-|\n|PickScore (SPM)|630M (CLIP-H Vision Encoder) + 353M (CLIP-H Text Encoder) + Small MLP|\n|LRM-1.5|123M (CLIP-L Text Encoder) + 346M (U-Net Down & Mid Block) + Linear Projection Layer|\n\n- We also conduct experiments on two internal DiT‑based models. We compare two different reward models: one is LRM, which utilizes parts of the parameters of the DiT, and the other is a VLM with more than 25B parameters. They are trained on internal image‑text alignment datasets and then employed for the preference optimization of the DiT models. For the same diffusion model, we train two reward models using identical data. The human‑evaluation results for image‑text alignment of optimized DiT models are as follows:\n\n| Diffusion Model | Reward Model | Alignment |\n| --------------- | ------------ | --------- |\n| DiT-S      | LRM (9.5B)   | **72.5**  |\n| DiT-S      | VLM (>25B)   | 71.2      |\n| DiT-L         | LRM (15B)    | **82.8**  |\n| DiT-L        | VLM (>25B)   | 80.8      |\n\nThese results consistently demonstrate that reward modeling in the noisy latent space, as done by LRM, can significantly enhance alignment performance **while being more parameter-efficient**.\n\n\n\n[1] Gu J, Jiang X, Shi Z, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024.\n\n[2] Nie S, Zhu F, You Z, et al. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.\n\n[3] Liang Z, Yuan Y, Gu S, et al. Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. CVPR 2025.\n\n[4] Gong L, Hou X, Li F, et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model. arXiv preprint arXiv:2503.07703, 2025.\n\n[5] Gao Y, Gong L, Guo Q, et al. Seedream 3.0 technical report. arXiv preprint arXiv:2504.11346, 2025.\n\n[6] Kirstain Y, Polyak A, Singer U, et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS, 2023."}, {"review_id": "woHouTcFkZ", "reviewer": "Reviewer_Xmnx", "summary": "The paper proposes LPO (Latent Preference Optimization), a follow-up to the SPO approach for preference optimization and text-to-image alignment. Both methods focus on training a model to optimize during the denoising steps of the diffusion process. However, compared to SPO, which needs a pixel-level evaluator, LPO uses a Latent Reward Model (LRM) trained for alignment directly in the latent space without needing a decoder to extract the pixel representations. \n\n- Training purely in latent space means no need for the decoding step and access to full noise range.\n- They propose Visual Feature Enhancement (VFE) which is based on Classifier-Free Guidance to amplify text conditioning\n- Multi-Preference Consistent Filtering (MPCF), which is used to counter many issues in the Pick-a-Pic dataset that is widely used but noisy and low quality.\n- These changes result in faster training and SOTA results for alignment and aesthetics.", "questions": "- As stated in the weaknesses, the paper should introduce failure cases and specific examples where the current method struggles (even with the ideal hyperparmeters) and provide some insight as to why this happens. This is vital for future work and understanding the limitations.\n\n- I have a few concerns with the speed comparison, which I think are very important to address. Looking at the SPO paper, they claim \"We use 4× A100 GPUs, which take ∼12 and ∼29.5 hours to train SD-1.5 and SDXL, respectively.\" but also \"We also spend 8 and 29 hours training SPM for SD-1.5 and SDXL, respectively.\"  Looking at the comparison SRM and LRM require extra training time so the comparison with DPO is not fair and extra training should be accounted for. \n\n- I was also not able to find the DPO training time figures so I think a specific source should be included or if the authors reproduced it themselves they should provide details for this. \n\n- While the ablation studies are extensive and well executed, they are conducted only on SD1.5.  However SDXL and SD1.5 can be very different. It will be beneficial to perform threshold, VFE and MPCF experiments on SDXL to show that hyper-parameters generalize across model scales.", "limitations": "- Should add \"Since LPO is performed within the latent space, which is determined by the VAE encoder, the VAE of LRM should be identical to that of DMO.\"\n\n- The training process uses MPCF, which filters data based on scores like Aesthetic Score, CLIP Score and VQAScore. This process can inherit any issues that are shared across these metrics.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- Model is trained purely in the latent space, which makes training faster and allows optimization across the full noise range.\n- Additional novel components are employed such as VFE and MPCF which assist training and improve results.\n- Extensive experimental results across multiple datasets, models (SD 1.5 and SDXL) and human evaluation.\n- Detailed ablation studies and hyper-parameter turning, where results such as the full noise range having the best score makes a strong case in favor of LPO.\n\nWeaknesses\n- The method involves extensive hyperparameter tuning, such as threshold schedules and filtering strategies. This could limit robustness when transferred to different architectures or datasets.\n- There are no specific failure cases or qualitative examples demonstrating where LPO underperforms when using the recommended hyperparameters. This limits insight into potential failure modes and areas for future improvement.\n- MPCF is a data filtering strategy and while it is shown to be effective, it is influenced by the similar scores it is evaluated against (Aesthetic is also in eval). This can be a concern if there are similar biases between the two groups (filtering and eval metrics).", "was_revised": true, "final_justification": "The authors have addressed my concerns thus I am willing to maintain my score.", "rebuttal": "Thanks for your effort in reviewing our paper and giving appreciation regarding our novelty, methods, and experiments. We hope the following responses address your concerns.\n\n> Q1 (Weakness 1): The method involves extensive hyperparameter tuning, such as threshold schedules and filtering strategies. This could limit robustness when transferred to different architectures or datasets.\n\nR1: Thanks for your valuable question. Your concern is very reasonable, as an excessive number of hyperparameters can indeed affect the robustness of a method. In this work, we conducted experiments to verify the generalization of the proposed method and hyperparameters across datasets and model architectures.\n\n- For **datasets**, we used exactly the same hyperparameters as those in the experiments on Pick-a-Pic, including MPCF, VFE, and Dynamic Threshold, and performed experiments on HPDv2 (Tables 15, 16, and 17 in the appendix), confirming the effectiveness of both the method and the hyperparameters.\n- For **model architectures**, we applied the same MPCF strategy (with the same hyperparameters) on SD1.5, SDXL, and SD3, and again verified the method's effectiveness. However, for different models, we used different Dynamic Threshold hyperparameters, as this parameter is influenced by the distribution of $x_t$ in each model. **We also complement extra hyperparameter experiments on SDXL to validate their generalization**, as detailed in **R6**.\n\nOverall, these experiments demonstrate that, with no adjustment or minor adjustment of a small subset of hyperparameters, our methods generalize well across datasets and model architectures.\n\n\n\n> Q2 (Weakness 2 & Question 1): No specific failure cases and their corresponding analysis.\n\nR2: Thanks for your constructive suggestion. We regret that the current paper lacks the visualization and analysis of failure cases. During our research, we observed the following types of failure cases:\n\n- **Reward hacking**: When training steps are excessive, the model exhibits reward hacking, where the reward metric continues to improve while the quality of generated images degrades. This stems from a certain misalignment between the reward model and human preferences, which remains a common issue across most existing methods. Mitigating this problem requires a combination of strategies, including improvements in preference data and reward modeling approaches.\n- **Decline in diversity**: Extended training also leads to a reduction in generation diversity. This is caused by the reward model narrowing the output distribution of the diffusion model. A general solution is prompt engineering, such as rephrasing input prompts to enhance the diversity of generated images via prompt variation [1].\n- **Lack of fine details on SDXL**: For some prompts, SDXL models optimized with LPO generate images with less detailed content compared to those optimized with SPO. This phenomenon was not observed on SD1.5. We will investigate this in future work to identify the underlying causes.\n\nDue to the rebuttal policy, we are unable to include images, external links, or upload new PDFs in our response. **We will include the above analysis and corresponding image examples in the revised version.**\n\n\n\n> Q3 (Weakness 3): MPCF is a data filtering strategy and while it is shown to be effective, it is influenced by the similar scores it is evaluated against (Aesthetic is also in eval). This can be a concern if there are similar biases between the two groups (filtering and eval metrics).\n\nR3: Thanks for your insightful question. This is a very reasonable concern. In our experiments, we also observed a certain correlation between the aesthetic scores used in filtering strategies and in evaluation metrics. For example, strictly enforcing aesthetic constraints (the first strategy in Table 1) leads to aesthetic hacking (Lines 206–207), and as a result, the aesthetic score in the final evaluation metric is also higher (the first strategy in Table 7). \n\nHowever, since we have not yet found a better open-source reward model for the aesthetic dimension, we still used the Aesthetic Predictor for automatic evaluation. **To provide a more objective assessment, we also conducted multi-dimensional (inlcuding aesthetic) human evaluations for the final model (Figure 6 in the appendix). The results of the user study are consistent with those from the automatic evaluation.** We will also consider using different aesthetic models for filtering and evaluation in future work.\n\n\n\n> Q4 (Question 2): Concerns with the speed comparison.\n\nR4: Thanks for your detailed question. In Table 6, we have listed the time for reward modeling and preference optimization separately, as well as their total time. For Diffusion-DPO, we set the reward modeling time to 0. In fact, in the paper, **the training time we refer to (Line 15, Line 74, Lines 290–292) always includes both reward modeling and preference optimization**, so the comparison is fair.\n\n\n\n> Q5 (Question 3): I was also not able to find the DPO training time figures so I think a specific source should be included or if the authors reproduced it themselves they should provide details for this.\n\nR5: Thanks for your question. The training time of Diffusion-DPO is provided in Table 6. We reproduced the results using the open-source repository (you can search DiffusionDPO on GitHub; we cannot post the direct URL due to the rebuttal policy) with 16 A100 GPUs, a local batch size of 1, and a gradient accumulation step of 128. Therefore, the effective batch size is 2048, consistent with that claimed in the original paper [2]. The number of training steps is 2000.\n\n\n\n> Q6 (Question 4): While the ablation studies are extensive and well executed, they are conducted only on SD1.5. However SDXL and SD1.5 can be very different. It will be beneficial to perform threshold, VFE and MPCF experiments on SDXL to show that hyper-parameters generalize across model scales.\n\nR6: Thanks for your valuable suggetion. We supplement ablation experiments on SDXL involving MPCF, VFE, and Dynamic Threshold. The results are provided as follows, **exhibiting similar patterns and trends as those observed on SD1.5 (Tables 7, 8, and 10)**. This further demonstrates the generalizability of our methods and hyperparameter settings across models of different scales and architectures.\n\n- MPCF\n\n|Strategy|Aes-Corr|CLIP-Corr|Aesthetic|GenEval|PickScore|\n|-|-|-|-|-|-|\n|wo MPCF|0.1355|0.1052|6.134|56.33|22.63|\n|1|**0.4528**|0.1716|**6.629**|57.56|22.72|\n|2|0.0729|0.4032|6.360|**59.27**|**22.86**|\n|3|-0.0698|**0.4656**|6.137|59.00|22.68|\n\n- VFE\n\n|VFE|gs|Aes-Corr|CLIP-Corr|Aesthetic|GenEval|PickScore|\n|-|-|-|-|-|-|-|\n|$ \\times $|1.0|**0.1534**|0.3314|**6.381**|58.33|22.73|\n|$ \\checkmark $|3.0|0.0989|0.3912|6.369|58.76|22.81|\n|$ \\checkmark $|7.5|0.0729|0.4032|6.360|**59.27**|**22.86**|\n|$ \\checkmark $|10.0|0.0632|**0.4147**|6.322|59.03|22.77|\n\n- Dynamic Threshold\n\n|Threshold|PickScore|ImageReward|HPSv2|HPSv2.1|Aesthetic|GenEval|\n|-|-|-|-|-|-|-|\n|0.5|22.77|1.1838|28.73|31.46|6.327|58.48|\n|0.6|22.79|1.1535|28.65|31.64|6.323|57.19|\n|[0.4, 0.55]|22.81|**1.2387** |28.86|31.52|6.354|58.72|\n|[0.45, 0.6]|**22.86**|1.2166|**28.96**|**31.89**|**6.360**|**59.27**|\n|[0.5, 0.65]|22.79|1.2091|28.95|31.74|6.277|58.68|\n\n\n\n> Q7 (Limitation 1): Should add \"Since LPO is performed within the latent space, which is determined by the VAE encoder, the VAE of LRM should be identical to that of DMO.\"\n\nR7: Thanks for your valuable suggestion. We will include this statement in the \"Limitation\" section in the revised version.\n\n\n\n> Q8 (Limitation 2): The training process uses MPCF, which filters data based on scores like Aesthetic Score, CLIP Score, and VQAScore. This process can inherit any issues that are shared across these metrics.\n\nR8: Thanks for your insightful suggestion. As you pointed out, the data filtered by MPCF is influenced by the filtering metrics used (Aesthetic, CLIP Score, and VQA Score). The combined use of multiple metrics helps reduce this risk, but it still persists. We will include a discussion of this limitation in the revised version.\n\n\n\n> Q9: Typos\n\nR9: Thanks very much for your detailed review. We will correct the following typos in the revised version.\n\n- Table 23: \"Ablataion\" -> \"Ablation\"\n- Line 150: \"averagely\" -> \"average pooled\"\n\n\n\n[1] Gong L, Hou X, Li F, et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model. arXiv preprint arXiv:2503.07703, 2025.\n\n[2] Wallace B, Dang M, Rafailov R, et al. Diffusion model alignment using direct preference optimization. CVPR, 2024."}, {"review_id": "Wxcmgw1z9C", "reviewer": "Reviewer_rhE9", "summary": "The paper proposes a novel way to train latent reward models for improving diffusion models. It shows that the pretrained diffusion models are naturally suited for step-level reward modelling. The architecture design for the reward model is natural and novel. Ablations on the architectures also present the robustness of the design choice. Also, the final results on benchmarks on Pick Score, HPS, ImageReward, GenEval, etc., demonstrate the excellence of the LRM. Due to its latent nature, the inference efficiency is higher than in previous works.", "questions": "1. The reward modeling is compatible with the U-Net architecture, which is widely used. Future exploration could focus on transformer-based architectures. What is the main architectural difference between SD3 in the appendix and SDXL in the main paper?\n\n2. How sensitive is LRM to the sampling method? There are many sampling strategies, and the proposed LPO is only suitable for some slower stochastic samplers. It is beneficial to examine the compatibility of LPO with different samplers. \n\n3. The proposed LRM significantly reduces the computational cost of image evaluation. However, during inference, the diffusion model still needs to generate more samples in the latent space. A fair efficiency comparison with baseline methods would be helpful—for example, measuring the time required to obtain 100 good images after selection by the reward model.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is easy to read and well-arranged. \n2. The architecture design together with the ablation studies are insightful and clean.\n3. The empirical results show the excellence of the proposed LRM and LPO. \n\n\nWeaknesses:\n\n1. The current reward model is restricted to U-Net architecture, which may not be compatible with SOTA diffusion transformer architecture.", "was_revised": true, "final_justification": "the rebuttal has addressed some of my concerns. I thus maintain my score.", "rebuttal": "Thanks for your effort in reviewing our paper and giving appreciation for our writing, novelty, and methods. We hope the following responses solve your concerns.\n\n> Q1 (Weakness 1): The current reward model is restricted to U-Net architecture, which may not be compatible with SOTA diffusion transformer architecture.\n\nR1: Thanks for your concern. In the appendix (Lines 521–535, Tables 13 and 14), **we have verified the effectiveness of LRM and LPO on SD3‑Medium, which is a model based on the diffusion transformer (DiT) architecture.**\n\n**In addition, we also conduct experiments on two first‑tier, internal DiT‑based models.** We compare two different reward models: one is LRM, which utilizes part of the parameters of the DiT, and the other is a VLM with more than 25B parameters. They are trained on internal image‑text alignment datasets and then employed for the preference optimization of the DiT models. For the same diffusion model, we train two reward models using identical data. The human‑evaluation results for image‑text alignment of optimized DiT models are as follows:\n\n| Diffusion Model | Reward Model | Alignment |\n| --------------- | ------------ | --------- |\n| DiT-S       | LRM (9.5B)   | **72.5**  |\n| DiT-S       | VLM (>25B)   | 71.2      |\n| DiT-L         | LRM (15B)    | **82.8**  |\n| DiT-L        | VLM (>25B)   | 80.8      |\n\nAll the above experimental results demonstrate that our methods can be effectively applied to the DiT architecture and outperform larger VLMs.\n\n\n\n> Q2 (Question 1): The reward modeling is compatible with the U-Net architecture, which is widely used. Future exploration could focus on transformer-based architectures. What is the main architectural difference between SD3 in the appendix and SDXL in the main paper?\n\nR2: Thanks for your question and constructive suggestion. We will continue to explore transformer‑based architectures. The SD3 model [1] in the appendix is based on the multimodal diffusion transformer (MMDiT) architecture, where text and image features interact through a self‑attention mechanism. In contrast, the SDXL model [2]  in the main paper is based on the U‑Net architecture, where text and visual features interact through cross‑attention.\n\n\n\n> Q3 (Question 2): How sensitive is LRM to the sampling method? There are many sampling strategies, and the proposed LPO is only suitable for some slower stochastic samplers. It is beneficial to examine the compatibility of LPO with different samplers.\n\nR3: Thanks for your detailed question. We believe there may be some misunderstanding here. LRM and LPO are not only suitable for slower stochastic samplers. We address this from the following two perspectives:\n\n- **Training of diffusion models**: LPO is an online-sampling-based method that directly optimizes the parameters of the diffusion model, rather than a training-free test-time scaling (TTS) approach. As such, LPO requires an SDE-form stochastic sampler for exploration and sampling, **but it is not limited to slow stochastic samplers.** In fact, in our experiments on SD1.5 and SDXL, we used a **20-step DDIM sampler** (Equations 2, 3, and 4) during LPO sampling (Figure 4 (d)), with $\\eta$ set to 1 to convert it into the SDE form. In the SD3 experiments in the appendix, we switched Flow Matching from ODE to SDE (Equations 14 and 15) and also used **20 steps** for LPO sampling. **In topics that do not focus on sampling acceleration, we believe that a 20-step stochastic sampler is not slow.**\n- **Inference of diffusion models**: During inference evaluation of diffusion models optimized by LPO, we use deterministic samplers in ODE form. Specifically, for SD1.5 and SDXL, we use a **20-step DDIM sampler** with $\\eta$ set to 0 (Tables 2 and 3), and for SD3, we use a **20-step Euler sampler** (Tables 13 and 14).\n\nThese experimental settings effectively demonstrate that both the LRM and LPO methods, as well as diffusion models optimized by LPO, can consistently improve performance across a variety of samplers. We believe the misunderstanding may be caused by the lack of detailed descriptions of the samplers used in the paper. We will provide a more detailed explanation of the sampling strategies in the revised version.\n\n\n\n> Q4 (Question 3): The proposed LRM significantly reduces the computational cost of image evaluation. However, during inference, the diffusion model still needs to generate more samples in the latent space. A fair efficiency comparison with baseline methods would be helpful—for example, measuring the time required to obtain 100 good images after selection by the reward model.\n\nR4: Thank you for recognizing the efficiency of LRM and for your valuable suggestion. Regarding inference, there may be some misunderstanding. **LPO is an online-sampling-based method that directly optimizes the parameters of the diffusion model, rather than a training-free test-time scaling (TTS) approach.**\n\n- If \"inference\" refers to the sampling procedure during LPO, it is indeed the case that multiple $x_t$ samples are drawn at each step and scored by the reward model to construct win–lose pairs for LPO training. This exploration process is similar to other online optimization methods such as SPO [3]. Since each $x_t$ is sampled from a Gaussian distribution $p(x_t|x_{t+1})$, the sampling itself requires negligible cost. The main computational overhead lies in scoring multiple $x_t$ samples with the reward model. **In this regard, LRM has a significant advantage over VLMs, as it avoids both $\\hat{x}_{0,t}$ prediction and VAE decoding (Figure 4 (a) and (b)). We provide a fair efficiency comparison in Table 5 of the paper.**\n\n- If \"inference\" refers to the inference of the optimized model after LPO training, **we do not generate multiple samples in latent space. Instead, we use a deterministic ODE-based sampler directly for inference**.\n\nRegarding your suggestion—\"measuring the time required to obtain 100 good images after selection by the reward model\"—we are somewhat confused, as this seems to be an evaluation strategy typically used in training-free methods. If possible, we would appreciate it if you could provide more details on how such a comparison could be conducted in our setting. We would be glad to perform this experiment if applicable.\n\n\n\n[1] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis. ICML, 2024.\n\n[2] Podell D, English Z, Lacey K, et al. Sdxl: Improving latent diffusion models for high-resolution image synthesis. ICLR, 2024.\n\n[3] Liang Z, Yuan Y, Gu S, et al. Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. CVPR 2025."}, {"review_id": "12TTbFn3es", "reviewer": "Reviewer_2Nf4", "summary": "This paper proposes LRM (Latent Reward Modeling) and LPO (Latent Preference optimization) as ways to improve the aesthetic and text alignment of the model. LRM allows for step-level preference optimization directly in the latent space. This is done by using a pretrained diffusion model as it already provides the characteristics for noise-aware, step-dependend modeling. Further the paper proposes VFE and MPCF to improve image-text correlation in UNet-Features and avoid the sometimes contradictory labels in the Pick-a-Pic dataset. The method shows promising results while being cheaper to use than their competitors.", "questions": "- Did you observe any failure cases like reward hacking?\n- Could you provide more reasoning for MPCF (regarding W4) and motivation for it? \n\nI am willing to increase my score if my concerns (esp. W1 / Clarity + W3 + W4) are well addressed by the authors.", "limitations": "The Limitations & Future Work section concentrates more on the latter and the only limitation discussed is the dataset. There are no limitations discussed that are relevant to the core method.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Weaknesses:\n- The paper misses structure and especially a list of core contributions as the authors introduce / propose LRM, MPCV, LPO and VFE which I would consider core contributions. \n- The VFE experiment (Table 8), shows that without VFE Aesthetic score and AesCorr is higher than with any VFE configuration which makes me doubt the general usefullness of VFE.\n- The authors reason in l.181 - l. 188 that the winning and loosing $x_t$ are dependend on $t$ and the respective amount of noise added which is understandable. However, a supporting experiment that provides intuition is missing. This also comes back to W1, where MPCV is motivated using this argument of noise dependend $x^w_t$ and $x^l_t$ but there is no proof of this motivation in terms of references or own experiments. \n- Extending the previous point, it is pointed out that int he Pick-a-Pic v1 dataset $x^w_t$ does not clearly perform better on eval benchmarks (Clip Score Aesthetic Score and VQAScore) than $x^l_t$. In the following paragraph (l. 196 onward), MPCF is explained which just says that instead of relying on a single of the benchmarks one takes multiple. This is a weak argument especially as the authors previously criticize the correlation between $x^w_t$ and these benchmarks one a few lines above.  \n- Following this reasoning, it would be vital to include the user study in the main paper. \n- Some minor spelling mistakes (l. 150, \"averagely\"-> \"average pooled\")\n\nStrengths:\n- The core method (LRM) makes sense and is intuitive.\n- It provides good result on the T2I-CompBecnh ++ which shows that this method can provide meaningful improvements compared to the baseline diffusion model and their competitors while requiring less training time. \n- There are numerous ablations that show good settings which give more intuition of the method.", "was_revised": true, "final_justification": "The main concern for me was clarity. In the rebuttal the authors provided good explanations and promised a revision of certain aspects that will help make the work overall more structured. Therefore I raised my score.", "rebuttal": "Thanks for your effort in reviewing our paper and giving appreciation for our novelty, methods, and experiments. We hope the following responses solve your concerns.\n\n> Q1 (Weakness 1): The paper misses structure and especially a list of core contributions as the authors introduce / propose LRM, MPCV, LPO and VFE which I would consider core contributions.\n\nR1: Thanks for your constructive question. In the Introduction section, we did not explicitly list the core contributions of this work. We will add the following content in the revised version:\n\nThe core contributions of this paper are summarized as follows:\n\n- A noise-aware Latent Reward Model (LRM) is introduced, which repurposes the pre-trained diffusion model for step-level reward modeling in the noisy latent space, with a Visual Feature Enhancement (VFE) module being used to enhance LRM's focus on the text-image alignment.\n- A Multi-Preference Consistent Filtering (MPCF) strategy is proposed to refine the public preference dataset, enabling LRM to better align with human preferences. \n- A Latent Preference Optimization (LPO) method based on LRM is introduced to perform step-level preference optimization directly within the noisy latent space of diffusion models.\n- Extensive experimental results demonstrate the effectiveness, efficiency, and generalization ability of the proposed methods.\n\nIn the Method section, we introduce LRM, MPCF, and LPO in three separate subsections, following the order of \"reward model architecture\" → \"reward model data filtering and training\" → \"optimization method\". VFE is included in the LRM subsection. We provided a summary of this structure in Lines 130–133. We believe this ordering is reasonable, and we hope the revised version will better highlight our contributions and present a more coherent overall structure.\n\n> Q2 (Weakness 2): The VFE experiment (Table 8), shows that without VFE Aesthetic score and AesCorr is higher than with any VFE configuration which makes me doubt the general usefullness of VFE.\n\nR2: Thanks for your detailed question. **VFE serves as a trade-off module between alignment and aesthetics.** The VFE module is inspired by the Classifier-Free Guidance (CFG) [1]. Its purpose is to enhance the weight of text-related visual features within the overall visual representation (Equation 8), thereby increasing LRM’s focus on text-image alignment. However, this comes at the cost of reducing the contribution of visual features unrelated to the text, which leads to a decrease in the correlation between LRM and aesthetics (Aes-Corr).  Although a slight decline in the Aesthetic score, it significantly improves text-image alignment (GenEval) and the overall preference score (PickScore), as discussed in Lines 308–312.\n\n> Q3 (Weakness 3): The authors reason in l.181 - l. 188 that the winning and loosing $ x_t $ are dependend on t and the respective amount of noise added which is understandable. However, a supporting experiment that provides intuition is missing. This also comes back to W1, where MPCV is motivated using this argument of noise dependend $x_t^w$ and $x_t^l$ but there is no proof of this motivation in terms of references or own experiments.\n\nR3: Thanks for your constructive suggestion. We regret the limited explanation on this point. We respond in the following aspects:\n\n- **Analysis of this motivation**: What we intended to convey in the paper is that if $x_0^w$ outperforms $x_0^l$ only in certain aspects, then after adding noise of equal intensity during the forward process, $x_t^w$ may become worse than $x_t^l$. The dual problem in the reverse denoising process is: **for the same prompt, two generated images $I_a$ and $I_b$ may have inconsistent preferences at intermediate steps versus the final output—$x_t^{I_a}$ being better than $x_t^{I_b}$ does not necessarily imply $x_0^{I_a}$ is better than $x_0^{I_b}$.**\n\n- **Proof of this motivation**: The above dual problem has been discussed in prior work. For example, in ImageReward [2] (Figure 4), the authors run inference with different seeds for the same prompt and evaluate the ImageReward scores of intermediate steps $x_t$ ($\\hat{x}\\_{0,t}$). The curves corresponding to different seeds show many intersections, each indicating a change in preference order. **For instance, between the brown and pink lines, at step 25, the score of $x_{25}^{brown}$ is higher than $x_{25}^{pink}$, but at step 40, $x_{40}^{pink}$ surpasses $x_{40}^{brown}$, illustrating that preferences in the intermediate steps of diffusion models do not necessarily align with final-step preferences.** Similarly, Figure 1 in SPO [3] also visualizes this kind of inconsistency. \n\nWe will incorporate these prior discussions in the revised version to provide a more comprehensive motivation for MPCF.\n\n> Q4 (Weakness 4): Extending the previous point, it is pointed out that in the Pick-a-Pic v1 dataset $x_t^w$ does not clearly perform better on eval benchmarks (Clip Score Aesthetic Score and VQAScore) than $x_t^l$. In the following paragraph (l. 196 onward), MPCF is explained which just says that instead of relying on a single of the benchmarks one takes multiple. This is a weak argument especially as the authors previously criticize the correlation between $x_t^w$ and these benchmarks one a few lines above.\n\nR4: Thanks for your detailed question. We'd like to emphasize that **we are not criticizing the reward models (benchmarks)  themselves—in fact, we rely on their robustness—but rather questioning the label quality of the Pick-a-Pic dataset**. As some of its original labels may not be trustworthy, we use the more reliable reward models to filter the data. We believe there may be some misunderstanding. To clarify, we outline the overall logic behind MPCF as follows:\n\n- First, we argue that if the training data for LRM is of low quality—specifically, if the winning image is only marginally better than the losing image in one particular aspect—then after injecting the same level of noise during the forward process, the resulting noisy version of the winning ($x_t^w$) may actually become worse than that of the losing ($x_t^l$). This motivates a stricter assumption: if the winning image is superior to the losing image in **multiple aspects**, then the relative preference between the two images is more likely to remain consistent after noise is added.\n\n- Consequently, during LRM training, we aim to use only image pairs where the winning is clearly better than the losing across multiple dimensions. However, this is not always the case with the Pick-a-Pic dataset [4]. Since it was constructed via crowdsourcing, some image pairs have low-quality labels, which means that the winning image may not be better in all aspects, as illustrated in Figure 5.\n\n- To filter out such unreliable training pairs in Pick-a-Pic, we leverage three open-source reward models: the Aesthetic Predictor for aesthetic judgment, and CLIPScore and VQAScore for text-image alignment evaluation. If the original label in Pick-a-Pic ($I^w$ vs. $I^l$) is consistent with all three reward models ($G=S(I^w)-S(I^l)>0$), we regard the winning image as being superior in multiple aspects and retain that pair for LRM training.\n\nWe hope the above explanation helps clarify the motivation and rationality of MPCF. We will revise the manuscript accordingly to make it easier to follow.\n\n> Q5 (Weakness 5): Following this reasoning, it would be vital to include the user study in the main paper.\n\nR5: Thanks for your suggestion. If the user study refers to evaluating the LPO-optimized model, the results are shown in Figure 6 in the appendix. If it refers to the Pick-a-Pic dataset, conducting a user study for such a large dataset is costly, which is why MPCF uses three reliable reward models for automatic filtering. These models serve as proxies to approximate human judgments in a more scalable way. We’re willing to follow further suggestions if you have specific ideas.\n\n> Q6 (Weakness 6): Typos.\n\nR6: Thanks for your detailed review. We will correct the following typos in the revised version.\n\n- Line 150: \"averagely\" -> \"average pooled\"\n- Table 23: \"Ablataion\" -> \"Ablation\"\n\n\n\n> Q7 (Question 1): Failure cases like reward hacking?\n\nR7: Yes. When training steps are excessive, the model exhibits reward hacking, where the reward metric continues to improve while the quality of generated images degrades. This stems from a certain misalignment between the reward model and human preferences, which remains a common issue across most existing methods. Mitigating this problem requires improvements in training data and reward modeling approaches. We will include the discussion about failure cases in the revised version.\n\n> Q8 (Question 2): More reasoning for MPCF (regarding W4) and motivation for it?\n\nR8: Thanks for your constructive suggestions. We provide more details in **R3** and **R4**.\n\n> Q9 (Limitation 1): There are no limitations discussed that are relevant to the core method.\n\nR9: Thanks for your valuable suggestion. We will include the following limitations in the revised version:\n\n- Since LPO is performed in the latent space defined by the VAE encoder, the VAE used in the LRM must be identical to that in the diffusion model to be optimized.\n\n- MPCF relies on three reward models to approximate human preferences for automatic data filtering. However, the filtered data may still inherit biases or limitations shared by these reward models.\n\n[1] Ho J, Salimans T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.\n\n[2] Xu J, Liu X, Wu Y, et al. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS, 2023.\n\n[3] Liang Z, Yuan Y, Gu S, et al. Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. CVPR 2025.\n\n[4] Kirstain Y, Polyak A, Singer U, et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS, 2023."}]

Author rebuttal:

Meta-review: This paper introduces a reward model for Diffusion preference optimization that is defined on the latent space of the diffusion model itself, and uses elements of it, as opposed to the traditional use of VLMs. They additionally refine the preference learning algorithm with ideas for preference pair filtering. The resulting algorithm leads to improvements in visual quality and alignment, along with training time speedups.

Strengths: Use of the latent space  to predict rewards in this way is novel. The key insight that preferences over final images may not be consistent with preferences of the noised images at each time-step is important. Reviewers appreciated the contributions in the algorithm design as well. Overall, small but consistent gains were achieved. 

There were no serious weaknesses raised by reviewers that were not satisfied by the end of the discussion. One downside of the method is that the RM has to be re-trained to align with each generative model separately.

Overall this seems like a refinement of existing approaches that is well justified and useful for practitioners in this popular field.
Decision: Accept (poster)