VARD: Efficient and Dense Fine-Tuning for Diffusion Models with
  Value-based RL
Authors: name: Fengyuan Dai, name: Zifeng Zhuang, name: Yufei Huang, name: Siteng Huang, name: Bangyan Liao, name: Donglin Wang, name: Fajie Yuan

Diffusion models have emerged as powerful generative tools across various
domains, yet tailoring pre-trained models to exhibit specific desirable
properties remains challenging. While reinforcement learning (RL) offers a
promising solution,current methods struggle to simultaneously achieve stable,
efficient fine-tuning and support non-differentiable rewards. Furthermore,
their reliance on sparse rewards provides inadequate supervision during
intermediate steps, often resulting in suboptimal generation quality. To
address these limitations, dense and differentiable signals are required
throughout the diffusion process. Hence, we propose VAlue-based Reinforced
Diffusion (VARD): a novel approach that first learns a value function
predicting expection of rewards from intermediate states, and subsequently uses
this value function with KL regularization to provide dense supervision
throughout the generation process. Our method maintains proximity to the
pretrained model while enabling effective and stable training via
backpropagation. Experimental results demonstrate that our approach facilitates
better trajectory guidance, improves training efficiency and extends the
applicability of RL to diffusion models optimized for complex,
non-differentiable reward functions.