Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion
  Models
Authors: name: Shuchen Xue; name: Chongjian Ge; name: Shilong Zhang; name: Yichen Li; name: Zhi-Ming Ma
Published: 2025-09-29 17:02:20

Abstract: Reinforcement Learning (RL) has emerged as a central paradigm for advancing
Large Language Models (LLMs), where pre-training and RL post-training share the
same log-likelihood formulation. In contrast, recent RL approaches for
diffusion models, most notably Denoising Diffusion Policy Optimization (DDPO),
optimize an objective different from the pretraining objectives--score/flow
matching loss. In this work, we establish a novel theoretical analysis: DDPO is
an implicit form of score/flow matching with noisy targets, which increases
variance and slows convergence. Building on this analysis, we introduce
Advantage Weighted Matching (AWM), a policy-gradient method for
diffusion. It uses the same score/flow-matching loss as pretraining to obtain a
lower-variance objective and reweights each sample by its advantage. In effect,
AWM raises the influence of high-reward samples and suppresses low-reward ones
while keeping the modeling objective identical to pretraining. This unifies
pretraining and RL conceptually and practically, is consistent with
policy-gradient theory, reduces variance, and yields faster convergence. This
simple yet effective design yields substantial benefits: on GenEval, OCR, and
PickScore benchmarks, AWM delivers up to a 24times speedup over Flow-GRPO
(which builds on DDPO), when applied to Stable Diffusion 3.5 Medium and FLUX,
without compromising generation quality. Code is available at
https://github.com/scxue/advantage_weighted_matching.