Efficient Generative Model Training via Embedded Representation Warmup
Authors: name: Deyuan Liu; name: Peng Sun; name: Xufeng Li; name: Tao Lin
Published: 2025-04-14 12:43:17

Abstract: Diffusion models excel at generating high-dimensional data but fall short in
training efficiency and representation quality compared to self-supervised
methods. We identify a key bottleneck: the underutilization of high-quality,
semantically rich representations during training notably slows down
convergence. Our systematic analysis reveals a critical representation
processing region -- primarily in the early layers -- where semantic and
structural pattern learning takes place before generation can occur. To address
this, we propose Embedded Representation Warmup (ERW), a plug-and-play
framework where in the first stage we get the ERW module serves as a warmup
that initializes the early layers of the diffusion model with high-quality,
pretrained representations. This warmup minimizes the burden of learning
representations from scratch, thereby accelerating convergence and boosting
performance. Our theoretical analysis demonstrates that ERW's efficacy depends
on its precise integration into specific neural network layers -- termed the
representation processing region -- where the model primarily processes and
transforms feature representations for later generation. We further establish
that ERW not only accelerates training convergence but also enhances
representation quality: empirically, our method achieves a 40times
acceleration in training speed compared to REPA, the current state-of-the-art
methods. Code is available at https://github.com/LINs-lab/ERW.