Weak-to-Strong Diffusion with Reflection
Authors: name: Lichen Bai; name: Masashi Sugiyama; name: Zeke Xie
Published: 2025-02-01 16:00:08

Abstract: The goal of diffusion generative models is to align the learned distribution
with the real data distribution through gradient score matching. However,
inherent limitations in training data quality, modeling strategies, and
architectural design lead to inevitable gap between generated outputs and real
data. To reduce this gap, we propose Weak-to-Strong Diffusion (W2SD), a novel
framework that utilizes the estimated difference between existing weak and
strong models (i.e., weak-to-strong difference) to approximate the gap between
an ideal model and a strong model. By employing a reflective operation that
alternates between denoising and inversion with weak-to-strong difference, we
theoretically understand that W2SD steers latent variables along sampling
trajectories toward regions of the real data distribution. W2SD is highly
flexible and broadly applicable, enabling diverse improvements through the
strategic selection of weak-to-strong model pairs (e.g., DreamShaper vs. SD1.5,
good experts vs. bad experts in MoE). Extensive experiments demonstrate that
W2SD significantly improves human preference, aesthetic quality, and prompt
adherence, achieving SOTA performance across various modalities (e.g., image,
video), architectures (e.g., UNet-based, DiT-based, MoE), and benchmarks. For
example, Juggernaut-XL with W2SD can improve with the HPSv2 winning rate up to
90% over the original results. Moreover, the performance gains achieved by W2SD
markedly outweigh its additional computational overhead, while the cumulative
improvements from different weak-to-strong difference further solidify its
practical utility and deployability.