Distribution Backtracking Builds A Faster Convergence Trajectory for
  One-step Diffusion Distillation
Authors: name: Shengyuan Zhang; name: Ling Yang; name: Zejian Li; name: An Zhao; name: Chenye Meng; name: Changyuan Yang; name: Guang Yang; name: Zhiyuan Yang; name: Lingyun Sun
Published: 2024-08-28 17:58:17

Abstract: Accelerating the sampling speed of diffusion models remains a significant
challenge. Recent score distillation methods distill a heavy teacher model into
an one-step student generator, which is optimized by calculating the difference
between the two score functions on the samples generated by the student model.
However, there is a score mismatch issue in the early stage of the distillation
process, because existing methods mainly focus on using the endpoint of
pre-trained diffusion models as teacher models, overlooking the importance of
the convergence trajectory between the student generator and the teacher model.
To address this issue, we extend the score distillation process by introducing
the entire convergence trajectory of teacher models and propose Distribution
Backtracking Distillation (DisBack) for distilling student generators. DisBask
is composed of two stages: Degradation Recording and Distribution Backtracking.
Degradation Recording is designed to obtain the convergence trajectory of
teacher models, which records the degradation path from the trained teacher
model to the untrained initial student generator. The degradation path
implicitly represents the intermediate distributions of teacher models. Then
Distribution Backtracking trains a student generator to backtrack the
intermediate distributions for approximating the convergence trajectory of
teacher models. Extensive experiments show that DisBack achieves faster and
better convergence than the existing distillation method and accomplishes
comparable generation performance. Notably, DisBack is easy to implement and
can be generalized to existing distillation methods to boost performance. Our
code is publicly available on https://github.com/SYZhang0805/DisBack.