Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human
  Preference
Authors: name: Xiangwei Shen, name: Zhimin Li, name: Zhantao Yang, name: Shiyi Zhang, name: Yingfang Zhang, name: Donghao Li, name: Chunyu Wang, name: Qinglin Lu, name: Yansong Tang

Recent studies have demonstrated the effectiveness of directly aligning
diffusion models with human preferences using differentiable reward. However,
they exhibit two primary challenges: (1) they rely on multistep denoising with
gradient computation for reward scoring, which is computationally expensive,
thus restricting optimization to only a few diffusion steps; (2) they often
need continuous offline adaptation of reward models in order to achieve desired
aesthetic quality, such as photorealism or precise lighting effects. To address
the limitation of multistep denoising, we propose Direct-Align, a method that
predefines a noise prior to effectively recover original images from any time
steps via interpolation, leveraging the equation that diffusion states are
interpolations between noise and target images, which effectively avoids
over-optimization in late timesteps. Furthermore, we introduce Semantic
Relative Preference Optimization (SRPO), in which rewards are formulated as
text-conditioned signals. This approach enables online adjustment of rewards in
response to positive and negative prompt augmentation, thereby reducing the
reliance on offline reward fine-tuning. By fine-tuning the FLUX.1.dev model
with optimized denoising and online reward adjustment, we improve its
human-evaluated realism and aesthetic quality by over 3x.