Title: DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Generation

Diffusion Models

ConvNets
Abstract: Diffusion Transformer (DiT), a promising diffusion model for visual generation, demonstrates impressive performance but incurs significant computational overhead. Intriguingly, analysis of pre-trained DiT models reveals that global self-attention is often redundant, predominantly capturing local patterns—highlighting the potential for more efficient alternatives. In this paper, we revisit convolution as an alternative building block for constructing efficient and expressive diffusion models. However, naively replacing self-attention with convolution typically results in degraded performance. Our investigations attribute this performance gap to the higher channel redundancy in ConvNets compared to Transformers. To resolve this, we introduce a compact channel attention mechanism that promotes the activation of more diverse channels, thereby enhancing feature diversity. This leads to Diffusion ConvNet (DiCo), a family of diffusion models built entirely from standard ConvNet modules, offering strong generative performance with significant efficiency gains. On class-conditional ImageNet generation benchmarks, DiCo-XL achieves an FID of 2.05 at 256$\times$256 resolution and 2.53 at 512$\times$512, with a **2.7$\times$** and **3.1$\times$** speedup over DiT-XL/2, respectively. Furthermore, experimental results on MS-COCO demonstrate that the purely convolutional DiCo exhibits strong potential for text-to-image generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDiffusion Transformer (DiT), a promising diffusion model for visual generation, demonstrates impressive performance but incurs significant computational overhead. Intriguingly, analysis of pre-trained DiT models reveals that global self-attention is often redundant, predominantly capturing local patterns. This paper proposes a pure convolution-based diffusion model, deemed as DiCo, to explore the cnn-like diffusion variant.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The comparison is fair and reasonable.\n2. The writing is clear and easy to understand.\n3. The acceleration of DiCo is significant and obvious.\n\nWeaknesses\n1. Lack of text-to-image results. As convolution is naturally incompatible with attention, I am doubtful about the multimodal performance, like text-to-image tasks.\n2. The necessity of the long residuals. UNet often converges faster with an inferior final performance.  As pointed out by DiT[1] and SD3[2], ***the U-Net inductive bias is not crucial(see the introduction section in DiT), and the U-shape limits further scaling (see Fig.4 in SD3)***.  Thus, DiCo looks completely like a technological regression.\n3. The claim of being state-of-the-art (SoTA) is unjustified, as DiCo's performance significantly lags behind modern diffusion transformers such as lightning-dit[3].\n4. There is no comparison with FlowDCN[4]. Deformable convolution is one of the most powerful convolution variants. It is appropriate to provide some discussions and comparisons.\n\n[1] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 4195-4205.\n\n[2] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]//Forty-first international conference on machine learning. 2024.\n\n[3]Yao J, Yang B, Wang X. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 15703-15712.\n\n[4]Wang S, Li Z, Song T, et al. FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolution[J]. arXiv preprint arXiv:2410.22655, 2024.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 1, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new backbone for diffusion models, Diffusion ConvNet (DiCo), as an alternative to DiT. It replaces self-attention with convolutions and proposes the CCA module to reduce channel redundancy. On the ImageNet benchmarks, DiCo surpasses state-of-the-art diffusion models while achieving significant efficiency gains.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n1. DiCo models demonstrate substantial computational efficiency gains compared with DiTs, concurrently achieving SOTA performance on ImageNet benchmarks. \n2. This work identifies the channel redundancy issue when introducing convolutions to DiTs and addresses it by introducing CCA.  The proposed solution appears to be reasonable.\n3. Comprehensive ablation studies are provided, detailing both architectural design and the contribution of individual components.\n\n- Weaknesses:\nThe experimental evaluation is primarily restricted to class-conditional image generation. While the authors acknowledge limitations for not extending evaluations to other prominent generative tasks (e.g., text-to-image synthesis), this circumscribes a comprehensive assessment of DiCo's broader applicability and generalizability across the diverse landscape of diffusion model applications.\n\nQUESTIONS:\n1. Regarding the CCA Module: Given the functional overlap between CCA and existing channel attention mechanisms (e.g., channel self-attention) in modulating channel features and the seemingly minor performance differences in the ablation study, it would be beneficial to elucidate the specific innovations within CCA that are critical for its effectiveness.\n\n2. Addressing Long-Range Dependencies and Generalizability. Given ConvNets' inherent locality, it is unclear how DiCo effectively captures the crucial long-range dependencies essential for generating high-quality images, especially in complex scenarios such as high-resolution image synthesis, multi-object scenes, or intricate image editing tasks. It would benefit from a discussion on how DiCo addresses this challenge.\n\n3. Questions about the ablation experiment. Table 4 consistently shows FID values below 54 across various configurations, whereas baseline models (DiT-S/2, DiC-S, DiG-S/2) in Table 1 report FID scores exceeding 58. This substantial performance improvement observed in DiCo warrants further explanation.\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

comments: SUMMARY:
In this paper,, the authors observe that self-attention in diffusion transformers seems to mostly capture local patterns, motivating going back to a convolutional architecture that could be more efficient. However, naively replacing the self attention layers with convolutional layers degrades performance, and the authors observe many channels remaining inactive during generation. Therefore, the authors introduce a compact channel attention (CCA) mechanism, which dynamically activates informative channels with lightweight linear projection. This enables this convolutional architecture achieve SOTA results on variety of benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
* well-structured and well-motivated
* compelling results on benchmarks
* unique architectural modifications

Weaknesses:
* no text-to-image results, even small-scale
* only 256x256 and 512x512 results on ImageNet

QUESTIONS:
My understanding is all the experiments shown here are for latent diffusion. do the observations about local attention patterns and improvements with CCA also hold true for pixel-level diffusion?

are the baselines in table 1 with CFG or not?

a big advantage of using a convolutional arch is that the scaling of computation with resolution will not be quadratic like with self-attention. Could a comparison at a larger scale like 1024x1024 be done to demonstrate this?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper replace self-attention layer in DiT with a novel CNN block which is enhanced by proposed compact channel attention (CAA) mechanism. By this means, the proposed model DiCo outperforms DiT on higher performance and efficiency at class-conditional image generation task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Considerable implemention details and experimental setting;\n2. Reader-friendly good  and clear writing;\n3. Visual insights, especially finding of localized representations in Figure4.\n\nWeakness:\n1. Insufficient innovation in the model architecture. \n2. Lacking comparison with U-net based diffusion models.  \n3. Inappropriate abbreviation.\n\nQUESTIONS:\n1. Insufficient innovation in the model architecture. The proposed DiCo seems to be the fusion of Stable Diffusion U-shape architecture and CNN-version DiT block. The highlighed CCA mechanism is quite similar to SE block. The biggest difference is removing the channel compression FC layer? Although combining different methods to achieve new functions can be considered as an innovation, it is undoubtedly that the degree of this mode of innovation is not sufficient enough for NeurIPS. \n2. Lacking comparison with U-net based diffusion models.  Since DiCo owns the U-shape architecture and the DiCo blocks are actually based on convolution module, why not conduct comparison with early U-net based diffusion, such as classic Stable Diffusion v1.5 and xl? Since authors have compared DiCo with DiT, why not compare to DiT based model such as Stable Diffusion 3 and FLUX? Though these models are original for text-to-image generation, I think simply replacing text prompt encoder with class embedder of DiT can enable the fair comparison. \n3. Inappropriate abbreviation. Authors abbreviated convolutional neural networks as ConvNets, which easily lead readers to mistake it for ConvNeXt. I believe it is a commen sense in the deep learning community that the abbreviation of convolutional neural networks is \"CNN\". If the authors insist on using the abbreviation \"ConvNets\", please provide a convincing reason for it.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}