RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers
Authors: name: Ke Cao; name: Jing Wang; name: Ao Ma; name: Jiasong Feng; name: Zhanjie Zhang; name: Xuanhua He; name: Shanyuan Liu; name: Bo Cheng; name: Dawei Leng; name: Yuhui Yin; name: Jie Zhang
Published: 2025-02-20 09:10:05

Abstract: The Diffusion Transformer plays a pivotal role in advancing text-to-image and
text-to-video generation, owing primarily to its inherent scalability. However,
existing controlled diffusion transformer methods incur significant parameter
and computational overheads and suffer from inefficient resource allocation due
to their failure to account for the varying relevance of control information
across different transformer layers. To address this, we propose the
Relevance-Guided Efficient Controllable Generation framework, RelaCtrl,
enabling efficient and resource-optimized integration of control signals into
the Diffusion Transformer. First, we evaluate the relevance of each layer in
the Diffusion Transformer to the control information by assessing the
"ControlNet Relevance Score"-i.e., the impact of skipping each control layer on
both the quality of generation and the control effectiveness during inference.
Based on the strength of the relevance, we then tailor the positioning,
parameter scale, and modeling capacity of the control layers to reduce
unnecessary parameters and redundant computations. Additionally, to further
improve efficiency, we replace the self-attention and FFN in the commonly used
copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM),
enabling efficient implementation of both the token mixer and channel mixer.
Both qualitative and quantitative experimental results demonstrate that our
approach achieves superior performance with only 15% of the parameters and
computational complexity compared to PixArt-delta. More examples are available
at https://relactrl.github.io/RelaCtrl/.