DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion
  Modeling
Authors: name: Yuang Ai, name: Qihang Fan, name: Xuefeng Hu, name: Zhenheng Yang, name: Ran He, name: Huaibo Huang

Diffusion Transformer (DiT), a promising diffusion model for visual
generation, demonstrates impressive performance but incurs significant
computational overhead. Intriguingly, analysis of pre-trained DiT models
reveals that global self-attention is often redundant, predominantly capturing
local patterns-highlighting the potential for more efficient alternatives. In
this paper, we revisit convolution as an alternative building block for
constructing efficient and expressive diffusion models. However, naively
replacing self-attention with convolution typically results in degraded
performance. Our investigations attribute this performance gap to the higher
channel redundancy in ConvNets compared to Transformers. To resolve this, we
introduce a compact channel attention mechanism that promotes the activation of
more diverse channels, thereby enhancing feature diversity. This leads to
Diffusion ConvNet (DiCo), a family of diffusion models built entirely from
standard ConvNet modules, offering strong generative performance with
significant efficiency gains. On class-conditional ImageNet benchmarks, DiCo
outperforms previous diffusion models in both image quality and generation
speed. Notably, DiCo-XL achieves an FID of 2.05 at 256x256 resolution and 2.53
at 512x512, with a 2.7x and 3.1x speedup over DiT-XL/2, respectively.
Furthermore, our largest model, DiCo-H, scaled to 1B parameters, reaches an FID
of 1.90 on ImageNet 256x256-without any additional supervision during training.
Code: https://github.com/shallowdream204/DiCo.