DDT: Decoupled Diffusion Transformer
Authors: name: Shuai Wang; name: Zhi Tian; name: Weilin Huang; name: Limin Wang
Published: 2025-04-08 07:17:45

Abstract: Diffusion transformers have demonstrated remarkable generation quality,
albeit requiring longer training iterations and numerous inference steps. In
each denoising step, diffusion transformers encode the noisy inputs to extract
the lower-frequency semantic component and then decode the higher frequency
with identical modules. This scheme creates an inherent optimization dilemma:
encoding low-frequency semantics necessitates reducing high-frequency
components, creating tension between semantic encoding and high-frequency
decoding. To resolve this challenge, we propose a new
\color{ddtD}ecoupled \color{ddtD}iffusion
\color{ddtT}ransformer~(\color{ddtDDT}), with a decoupled
design of a dedicated condition encoder for semantic extraction alongside a
specialized velocity decoder. Our experiments reveal that a more substantial
encoder yields performance improvements as model size increases. For ImageNet
256times256, Our DDT-XL/2 achieves a new state-of-the-art performance of
{1.31 FID}~(nearly 4times faster training convergence compared to previous
diffusion transformers). For ImageNet 512times512, Our DDT-XL/2 achieves a
new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our
decoupled architecture enhances inference speed by enabling the sharing
self-condition between adjacent denoising steps. To minimize performance
degradation, we propose a novel statistical dynamic programming approach to
identify optimal sharing strategies.