DC-DPM: A Divide-and-Conquer Approach for Diffusion Reverse Process

Authors: Yue-Jiang Dong, Hubery Yin, Fangyikang Wang, Yizhe Zhao, Chao Zhang, Chen Li, Song-Hai Zhang
Primary area: generative models
Keywords: Diffusion Model; Reverse Process Transition Kernel; Divide-and-Conquer

Abstract:
Diffusion models have achieved great success in generative tasks\textblue{, with the quality of generated samples guaranteed by their convergence properties, typically derived within the context of stochastic differential equations(SDE) and often involving Kolmogorov equations for proofs. This paper introduces a novel method for proving the convergence of diffusion models, which relies on direct estimation of distributions without the need for SDE tools. This approach inspires a \textbf{D}ivide-and-\textbf{C}onquer strategy for approximating the reversed transition kernel of \textbf{D}iffusion \textbf{P}robabilistic \textbf{M}odels (DC-DPM), which is not derived from SDEs, making previous convergence methods inapplicable. However, our method can be easily extended to accommodate this. As} our DC-DPM learns specific kernels for each partition \textblue{, these kernels require merging. According to the proof of convergence, we} design two merging strategies for these cluster-specific kernels along with corresponding training and sampling methods.
Experimental results demonstrate the superior generation quality of our method compared to the traditional single Gaussian kernel. Furthermore, our DC-DPM can synergize with previous kernel optimization methods, enhancing their generation quality, especially with a small number of timesteps.

Decision: reject
Reviewer scores: 6, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730337705653

thread_type:
official_review

<thread_object>
creator: reviewer_skma

<content>
summary:
This paper explores accelerating diffusion model sampling by improving the transition kernel of the reverse process. The authors aim to narrow the gap between the ground-truth reverse kernel and the conventional unimodal Gaussian kernel by employing a divide-and-conquer approach—using distinct kernels tailored to clustered data. Leveraging the proposed proof technique, the authors demonstrate that the improved kernel can guarantee convergence to the ground-truth reverse process. Experimental results indicate that this approach may effectively speed up diffusion models while maintaining performance.

strengths:
1. This paper addresses a crucial problem in accelerating diffusion model sampling from a relatively under-explored perspective, i.e., improving the reverse process kernel.
2. The paper provides a substantial amount of theoretical analysis, along with a novel proof technique that can support convergence analysis.
3. The presentation is clear and easy to follow.

weaknesses:
1. The motivation of the proposed approach comes from the inherent discrepancy between the ground truth reverse process, expressible as a mixture of Gaussians (in Eq. 5), and the traditional reverse kernel based on uni-modal Gaussian (i.e., Eq. 6). While this is a natural approach, a limitation remains as the authors still rely on the unimodal Gaussian kernel to approximate the MoG kernel for clustered data, as noted in L244-246. The convergence result to the ground-truth reverse process with this uni-modal approximation seems also missing.
2. Although the proposed approach offers advantages in accelerating diffusion models, it necessitates specialized training, making integration into conventional diffusion frameworks—proven to be highly effective—challenging.
3. Empirical validation on challenging, large-scale benchmarks (such as ImageNet) is lacking. The paper primarily focuses on relatively small, simpler datasets, such as CIFAR-10 and CelebA-HQ-256, raising questions about the scalability of the proposed approach when applied to more complex datasets.
4. A broader comparative study would strengthen the paper. Acceleration of diffusion models is a highly active research area, with numerous approaches offering speed improvements from different perspectives. However, the study covers only two baselines, both of which follow a similar methodological approach, i.e., improving the reverse kernel.
5. An analysis of computational costs would provide a more comprehensive comparison with baseline methods, especially since the proposed method involves the learning of multiple kernels, as well as additional neural network training (for DC-LD).

questions:
See the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621967225

thread_type:
official_review

<thread_object>
creator: reviewer_p77b

<content>
summary:
This paper focuses on the challenge of approximating the reversed transition kernel of diffusion models with a Gaussian distribution. To address this problem, they propose a divide-and-conquer strategy and introduce DC-DPM. This approach clusters the data and learns transition kernels tailored to each cluster. The training and sampling processes are facilitated by two proposed approaches: label diffusion approximation and fixed class approximation. Additionally, the paper provides a theoretical convergence analysis. They empirically demonstrate that the proposed method yields improved data quality on benchmark dataset.

strengths:
* The proposed method is both intuitive and reasonable. Since the characteristics of the transition kernel may vary depending on the data, adding it as a conditioning factor is likely to be effective. This is further supported by many previous experimental results showing that conditional datasets tend to achieve better FID scores than unconditional ones.
* The theoretical background supporting the proposed method is well-founded and lends credibility to the approach.

weaknesses:
* Several assumptions are introduced to support the theoretical background, but it would be beneficial to explain whether these assumptions align with practical applications or under what conditions they hold in practical scenarios.

* Although the paper states that convergence is guaranteed with arbitrary data partitioning, there may be issues related to convergence speed, so it would be beneficial to further explain for the partitioning.

* The arbitrary data partitioning may also pose a challenge for training $a_\phi$ network, as the learning difficulty may vary depending on the partitioning.

* In the explanation of Corollary 3, it is mentioned that $\epsilon_{al}$ is expected to have a smaller influence than $\epsilon_{yl}$. However, this influence could vary depending on the scales of the constants $C_2$ and $C_3$, so further analysis on this point would be valuable.

* As the current experiments focus mainly on benchmark datasets with limited datasets and network structures, further experimental validation would strengthen the credibility of the method.

  * As data partitioning is an important part of the proposed method, it would be helpful to evaluate the effectiveness of the method on datasets with diverse semantic features, such as ImageNet.

  * It would be beneficial to test the proposed method on more advanced diffusion models, such as EDM [1].

  * Demonstrating its applicability to common applications of diffusion models, such as text-to-image generation, would further strengthen the approach.

* The LD variants sometimes underperform compared to the baseline models, so analyzing these cases would provide insight into potential limitations.

[1] Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022

questions:
* Could the authors provide their thoughts on the points mentioned in Weaknesses, particularly regarding the experimental point?
* In the convergence analysis, what distinguishes the proposed method from previous works in terms of achieving the tighter bound?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695340225

thread_type:
official_review

<thread_object>
creator: reviewer_nruj

<content>
summary:
This paper proposes a Divide-and-Conquer strategy to improve the traditional single Gaussian transition kernel representation in each denoising step of Diffusion Probabilistic Models (DC-DPM), enhancing generation quality particularly over a limited number of timesteps. By dividing the data into clusters, our DC-DPM learns specific kernels for each partition.

strengths:
A Divide-and-Conquer strategy to improve the traditional single Gaussian transition kernel representation in each denoising step of Diffusion Probabilistic Models (DC-DPM), which was shown to improve the sample generation quality in the numerical experiments.

weaknesses:
The model assumptions seem confusing and need more clarification; there is limited comparison with existing theory on diffusion models; see below for more detailed comments.

questions:
It would be beneficial if the authors could discuss its potential connection with conditional sampling. For example, in the CIFAR example, the clusters are based on the image labels, and thus the $y_\theta$ becomes $y_\theta(x,t,c)$ where c is the data label, this resembles the conditional sampling in certain extent.

My major concern and question are as follows. The data distribution is assumed as eq (1) which is essentially a uniform distribution supported on training data. This is a very confusing assumption to me, and since this is the starting point of all the theoretical analysis, I believe the authors needs to elaborate more on why this assumption is reasonable. Does this imply that the learned diffusion model will simply "memorize" data and cannot generate distribution outside of the empirical support well? Furthermore, I believe that in most theoretical analyses for diffusion models, the ground truth data distribution is assumed as a non-parametric distribution (continuous distribution), rather than the eq(1) assumed here. Additionally, as a paper studying the theoretical perspectives of diffusion models, some representative papers in this field are not cited or discussed, e.g.,  “Oko, Kazusato, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. ICML 2023” and many others.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731046323404

thread_type:
official_review

<thread_object>
creator: reviewer_agy8

<content>
summary:
The paper introduces a strategy called Divide-and-Conquer to accelerate the convergence of Diffusion Probabilistic Models. This approach begins by partitioning the data, allowing for separate learning of transition kernels for each partition. These transition kernels are then combined during both the training and sampling phases to form a unified model. The authors also provide a formal convergence proof for their proposed strategy, reinforcing its theoretical soundness. Experimental results on the CelebA and CIFAR-10 datasets demonstrate that the proposed method improves both the Fréchet Inception Distance (FID) and the visual quality of generated images, highlighting its effectiveness in practical applications.

strengths:
- The paper presents a straightforward yet effective strategy to enhance the learning performance of Diffusion Probabilistic Models (DPMs).

- The authors also develop a convergence proof for the proposed method, providing a solid theoretical foundation that supports its reliability.

- Additionally, the manuscript is well-organized and clearly presented.

weaknesses:
- The paper presents numerous theorems, but it is unclear which are original contributions. Proper citation of existing results is needed to clarify which findings are novel and which build on prior work.
- The evaluation is limited to two datasets, which may not comprehensively assess the proposed method’s effectiveness. Including additional datasets would improve the robustness of the benchmarking.
- Only one baseline method is compared, despite significant related work in kernel representation that could serve as relevant baselines.
- The qualitative results in Figure 3 provide generated images with minimal discussion and analysis, limiting insights into the model's qualitative performance.

questions:
What is the difference between DC-DPM and GMS, which use GMM as kernels?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732442305922

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all Reviewers, AC and PC: Contribution and summary of modifications during discussion

comment:
We deeply thank the reviewers for their appreciation of this work and insightful feedbacks. Your suggestions greatly help us to improve and enrich our work. In the following discussions, we individually address your comments.

**Upon reviewing the abstract and the introduction, we realized that the main contribution of this paper wasn't sufficiently emphasized, which could have led the reviewers to overlook it. This paper goes beyond a technical improvement by proposing a divide-and-conquer strategy for the diffusion reverse transition kernel. More importantly, we introduce a novel method to prove the convergence of diffusion models. Unlike previous methods that require the reverse process to be derived from a Stochastic Differential Equation (SDE), our approach demonstrates that diffusion models with non-SDE reverse processes can also converge. Consequently, we have revised our abstract and introduction to better highlight these points. The revised sections are indicated in blue.**

Our method thus extends the framework of converged diffusion models and introduces a more powerful representation of the reverse process, namely DC-DPM, which employs a divide-and-conquer approach for the reversed transition kernel. According to our experimental results, DC-DPM exhibits superior generation capabilities compared to conventional diffusion models. It is important to note that the convergence of our DC-DPM cannot be proven using any of the existing SDE-based diffusion model convergence proof methods.

Additionally, DC-DPM enables a new way to integrate auto-regressive models with diffusion models. Specifically, the label diffusion model can be viewed as a single-token-predicting auto-regressive model and can be extended to generate more complex modes like text and audio. 
In our approach, diffusion models and auto-regressive models interact at each diffusion denoising step, unlike previous methods such as MAR[1] and Transfusion[2], where auto-regressive models merely provide conditions for diffusion models.

We have updated our paper for adding highlights on the modifications we made to our paper. we mainly modified our manuscript as listed below:
+ The abstract and introduction are modified to demonstrate our contribution more clearly and emphasize our theoretical contribution.
+ More detailed explanation of our proposed theory are included in the method section for better clarity. 
+ Experiments on a more challenging, larger-scale dataset, ImageNet64x64[3], are provided in appendix, showcasing our method can still outperform previous methods on datasets with diverse semantic features. 
+ Experiments applying our DC-DPM to other representative diffusion acceleration methods, including DPM-Solver[4], DPM-Solver++[5], and quality improvement method EDM[6], are provided in the appendix to demonstrate the further improvements achieved when combined with our method.

Due to the page limit of the main paper, we move the ablation study result in the paper before to the appendix.

We hope our revised manuscript and additional experimental results can help address our reviewers' concerns and doubts. We look forward to your further replies. Thanks.

References:

[1] Li T, Tian Y, Li H, et al. Autoregressive Image Generation without Vector Quantization. ArXiv 2024. 

[2] Zhou C, Yu L, Babu A, et al. Transfusion: Predict the next token and diffuse images with one multi-modal model. ArXiv 2024.

[3] Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical image database. CVPR 2009.

[4] Lu C, Zhou Y, Bao F, et al. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS 2022.

[5] Lu C, Zhou Y, Bao F, et al. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. ArXiv 2022.

[6] Karras T, Aittala M, Aila T, et al. Elucidating the design space of diffusion-based generative models. NeurIPS 2022.
</content>
</thread_object>

</thread>


Metareview: