Title: Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Model

Reward Model

Preference Optimization

Image Generation
Abstract: Preference optimization for diffusion models aims to align them with human preferences for images. Previous methods typically use Vision-Language Models (VLMs) as pixel-level reward models to approximate human preferences. However, when used for step-level preference optimization, these models face challenges in handling noisy images of different timesteps and require complex transformations into pixel space. In this work, we show that pre-trained diffusion models are naturally suited for step-level reward modeling in the noisy latent space, as they are explicitly designed to process latent images at various noise levels. Accordingly, we propose the **Latent Reward Model (LRM)**, which repurposes components of the diffusion model to predict preferences of latent images at arbitrary timesteps. Building on LRM, we introduce **Latent Preference Optimization (LPO)**, a step-level preference optimization method conducted directly in the noisy latent space. Experimental results indicate that LPO significantly improves the model's alignment with general, aesthetic, and text-image alignment preferences, while achieving a 2.5-28x training speedup over existing preference optimization methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LRM (Latent Reward Modeling) and LPO (Latent Preference optimization) as ways to improve the aesthetic and text alignment of the model. LRM allows for step-level preference optimization directly in the latent space. This is done by using a pretrained diffusion model as it already provides the characteristics for noise-aware, step-dependend modeling. Further the paper proposes VFE and MPCF to improve image-text correlation in UNet-Features and avoid the sometimes contradictory labels in the Pick-a-Pic dataset. The method shows promising results while being cheaper to use than their competitors.\n\nSTRENGTHS AND WEAKNESSES:\nWeaknesses:\n- The paper misses structure and especially a list of core contributions as the authors introduce / propose LRM, MPCV, LPO and VFE which I would consider core contributions. \n- The VFE experiment (Table 8), shows that without VFE Aesthetic score and AesCorr is higher than with any VFE configuration which makes me doubt the general usefullness of VFE.\n- The authors reason in l.181 - l. 188 that the winning and loosing $x_t$ are dependend on $t$ and the respective amount of noise added which is understandable. However, a supporting experiment that provides intuition is missing. This also comes back to W1, where MPCV is motivated using this argument of noise dependend $x^w_t$ and $x^l_t$ but there is no proof of this motivation in terms of references or own experiments. \n- Extending the previous point, it is pointed out that int he Pick-a-Pic v1 dataset $x^w_t$ does not clearly perform better on eval benchmarks (Clip Score Aesthetic Score and VQAScore) than $x^l_t$. In the following paragraph (l. 196 onward), MPCF is explained which just says that instead of relying on a single of the benchmarks one takes multiple. This is a weak argument especially as the authors previously criticize the correlation between $x^w_t$ and these benchmarks one a few lines above.  \n- Following this reasoning, it would be vital to include the user study in the main paper. \n- Some minor spelling mistakes (l. 150, \"averagely\"-> \"average pooled\")\n\nStrengths:\n- The core method (LRM) makes sense and is intuitive.\n- It provides good result on the T2I-CompBecnh ++ which shows that this method can provide meaningful improvements compared to the baseline diffusion model and their competitors while requiring less training time. \n- There are numerous ablations that show good settings which give more intuition of the method.\n\nQUESTIONS:\n- Did you observe any failure cases like reward hacking?\n- Could you provide more reasoning for MPCF (regarding W4) and motivation for it? \n\nI am willing to increase my score if my concerns (esp. W1 / Clarity + W3 + W4) are well addressed by the authors.\n\nLIMITATIONS:\nThe Limitations & Future Work section concentrates more on the latter and the only limitation discussed is the dataset. There are no limitations discussed that are relevant to the core method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel way to train latent reward models for improving diffusion models. It shows that the pretrained diffusion models are naturally suited for step-level reward modelling. The architecture design for the reward model is natural and novel. Ablations on the architectures also present the robustness of the design choice. Also, the final results on benchmarks on Pick Score, HPS, ImageReward, GenEval, etc., demonstrate the excellence of the LRM. Due to its latent nature, the inference efficiency is higher than in previous works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is easy to read and well-arranged. \n2. The architecture design together with the ablation studies are insightful and clean.\n3. The empirical results show the excellence of the proposed LRM and LPO. \n\n\nWeaknesses:\n\n1. The current reward model is restricted to U-Net architecture, which may not be compatible with SOTA diffusion transformer architecture.\n\nQUESTIONS:\n1. The reward modeling is compatible with the U-Net architecture, which is widely used. Future exploration could focus on transformer-based architectures. What is the main architectural difference between SD3 in the appendix and SDXL in the main paper?\n\n2. How sensitive is LRM to the sampling method? There are many sampling strategies, and the proposed LPO is only suitable for some slower stochastic samplers. It is beneficial to examine the compatibility of LPO with different samplers. \n\n3. The proposed LRM significantly reduces the computational cost of image evaluation. However, during inference, the diffusion model still needs to generate more samples in the latent space. A fair efficiency comparison with baseline methods would be helpful—for example, measuring the time required to obtain 100 good images after selection by the reward model.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes LPO (Latent Preference Optimization), a follow-up to the SPO approach for preference optimization and text-to-image alignment. Both methods focus on training a model to optimize during the denoising steps of the diffusion process. However, compared to SPO, which needs a pixel-level evaluator, LPO uses a Latent Reward Model (LRM) trained for alignment directly in the latent space without needing a decoder to extract the pixel representations. \n\n- Training purely in latent space means no need for the decoding step and access to full noise range.\n- They propose Visual Feature Enhancement (VFE) which is based on Classifier-Free Guidance to amplify text conditioning\n- Multi-Preference Consistent Filtering (MPCF), which is used to counter many issues in the Pick-a-Pic dataset that is widely used but noisy and low quality.\n- These changes result in faster training and SOTA results for alignment and aesthetics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Model is trained purely in the latent space, which makes training faster and allows optimization across the full noise range.\n- Additional novel components are employed such as VFE and MPCF which assist training and improve results.\n- Extensive experimental results across multiple datasets, models (SD 1.5 and SDXL) and human evaluation.\n- Detailed ablation studies and hyper-parameter turning, where results such as the full noise range having the best score makes a strong case in favor of LPO.\n\nWeaknesses\n- The method involves extensive hyperparameter tuning, such as threshold schedules and filtering strategies. This could limit robustness when transferred to different architectures or datasets.\n- There are no specific failure cases or qualitative examples demonstrating where LPO underperforms when using the recommended hyperparameters. This limits insight into potential failure modes and areas for future improvement.\n- MPCF is a data filtering strategy and while it is shown to be effective, it is influenced by the similar scores it is evaluated against (Aesthetic is also in eval). This can be a concern if there are similar biases between the two groups (filtering and eval metrics).\n\nQUESTIONS:\n- As stated in the weaknesses, the paper should introduce failure cases and specific examples where the current method struggles (even with the ideal hyperparmeters) and provide some insight as to why this happens. This is vital for future work and understanding the limitations.\n\n- I have a few concerns with the speed comparison, which I think are very important to address. Looking at the SPO paper, they claim \"We use 4× A100 GPUs, which take ∼12 and ∼29.5 hours to train SD-1.5 and SDXL, respectively.\" but also \"We also spend 8 and 29 hours training SPM for SD-1.5 and SDXL, respectively.\"  Looking at the comparison SRM and LRM require extra training time so the comparison with DPO is not fair and extra training should be accounted for. \n\n- I was also not able to find the DPO training time figures so I think a specific source should be included or if the authors reproduced it themselves they should provide details for this. \n\n- While the ablation studies are extensive and well executed, they are conducted only on SD1.5.  However SDXL and SD1.5 can be very different. It will be beneficial to perform threshold, VFE and MPCF experiments on SDXL to show that hyper-parameters generalize across model scales.\n\nLIMITATIONS:\n- Should add \"Since LPO is performed within the latent space, which is determined by the VAE encoder, the VAE of LRM should be identical to that of DMO.\"\n\n- The training process uses MPCF, which filters data based on scores like Aesthetic Score, CLIP Score and VQAScore. This process can inherit any issues that are shared across these metrics.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed LPO, an novel algorithm that leverages latent diffusion backbone as a powerful and efficient reward model that learns to rank noisy images in the latent space. Sampling inference strategies including multi-preference filtering, dynamic thresholding has been introduced for efficient online sampling. LPO achieved superior results comparing to multiple baselines in both general image quality and human preference alignment on SD1.5, SDXL. Further studies also demonstrated that LPO can be generalized to flow matching objective and DiT structure.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths:\n  * The authors conduct comprehensive experiments across multiple evaluation metrics of T2I image synthesis and verify the proposed method with base T2I models covering various architectures and training objectives\n * LPO demonstrated robust and superior results against baselines\n\n* Weaknesses:\n  * The novelty of the proposed method is limited, the proposed method essentially leverages the SD-VAE to allow the reward model training to latent space and employed stronger filtering mechanism to further improve alignment performance, the technical contribution is limited.\n * Poor generalizability: Even though LRM 1.5 was proved effective on SD2.1, the success can be attribute more to the similar capability of the two models as LRM 1.5 was ineffective for SDXL. Therefore, the proposed method needs to fine-tune diffusion backbone as reward model before alignment if the base model differs significantly from a pre-trained LRM, adding complexity in potential tuning to the post-training pipeline.\n\nQUESTIONS:\n* Do the authors decide MPCF strategies based on heuristics? Do you think it's possible to develop a principled learned MPCF filtering pipeline?\n\n* Pick a Pic have 5M image pairs, may I know how long does it take to filter the image pairs?\n\n* Table 19 and Table 20 seems a bit unfair comparison as pick score has a smaller size with CLIP + small MLP, while even smallest SD1.5 LRM is CLIP + Unet. It's hard to tell how much latent space reward modeling itself improves alignment performance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}