Semi-Implicit Denoising Diffusion Models (SIDDMs)
Authors: yanwu xu
Mingming Gong
Shaoan Xie
Wei Wei
Matthias Grundmann
kayhan Batmanghelich
Tingbo Hou
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model
GAN
Semi-implicit Modeling

Abstract: Despite the proliferation of generative models, achieving fast sampling during inference without compromising sample diversity and quality remains challenging. Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. The Denoising Diffusion Generative Adversarial Networks (DDGAN) attempted to circumvent this limitation by integrating a GAN model for larger jumps in the diffusion process. However, DDGAN encountered scalability limitations when applied to large datasets. To address these limitations, we introduce a novel approach that tackles the problem by matching implicit and explicit factors. More specifically, our approach involves utilizing an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. This combination allows us to effectively match the joint denoising distributions. Unlike DDPM but similar to DDGAN, we do not enforce a parametric distribution for the reverse step, enabling us to take large steps during inference. Similar to the DDPM but unlike DDGAN, we take advantage of the exact form of the diffusion process. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.

Semi-Implicit Denoising Diffusion Models (SIDDMs)
Yanwu Xul Mingming Gong², Shaoan Xie Wei Wei', Matthias Grundmann's Kayhan Batmanghelich" Tingho Hou Electrical and Computer Engineering, Boston University, kayauuu
School of Mathematics Statistics, The University of Melbourne mingmiing.-onggouummilll edu.au "Carnegie Mellon University shaoanOcmu.ede
Abstract
Despite the proliferation of generative models, achieving fast sampling during inference without compromising sample diversity and quality remains challenging. Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of itera- tive steps. The Denoising Diffusion Generative Adversarial Networks (DDGAN) attempted to circumvent this limitation by integrating GAN model for larger jumps in the diffusion process. However, DDGAN encountered scalability limita- tions when applied to large datasets. To address these limitations, we introduce a novel approach that tackles the problem by matching implicit and explicit factors. More specifically, our approach involves utilizing an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. This combination allows us to effectively match the joint denoising distributions. Unlike DDPM but similar to DDGAN, we do not enforce parametric distribution for the reverse step, enabling us to take large steps during inference. Similar to the DDPM but unlike DDGAN, we take advantage exact form of the diffusion process. We demonstrate that our proposed method obtains comparable generative performance to diffusion babtudd models and vastly superior results models with small number of sampling steps. The code is available at http::/giitubb.co//yyaaav/u////
the
Introduction
Generative models have achieved significant success in various domains such image generation, Different types of generative models have been developed tackle specific challenges. Variational autoencoders (VAEs) g provide a variational lower bound for training models with explicit objectives. Generative adversarial networks (GANs) p introduce min-max game framework to implicitly model data distribution and enable one-step generation. Denoising diffusion probabilistic models (DDPMs)
video synthesis, audio generation, and point cloud generation 2 3 5 6
[0. [1. also known score-based generative models, recover the original data distribution through iterative denoising from an initial random Gaussian noise vector. However, these models face a common challenge known the "TRILEMMA" [IЛ, which involves ensuring high-quality sampling. mode coverage, and fast sampling speed simultaneously. Existing approaches, such as GANs, VAEs, and DDPMs struggle address all three aspects simultaneously. This paper focuses on tackling this TRILEMMA and developing models capable of effectively modelling large-scale data generation.
*Work done as a student researcher of Google. t Equal Contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While diffusion models excel in generating high-quality samples compared to VAEs and demonstrate better training convergence than GANs, they typically require thousands of iterative steps to obtain the highest-quality results. These long sampling steps are based on the assumption that the reversed diffusion distribution can approximated by Gaussian distributions when the noise addition in the forward diffusion process is small. However, if the noise addition is significant, the reversed diffusion distribution becomes a non- Gaussaan multimodal distribution [[]. Consequently, reducing the number of sampling steps for faster generation would violate this assumption and introduce bias in the generated samples.
To address this issue, DDGANs use reformulation of forward diffusion sampling and model the undefined denoising distribution using a conditional GAN. This approach enables faster sampling without compromising the quality of the generated samples. Additionally, DDGANs exhibit improved convergence and stability during training compared to pure GANs. However, DDGANS still face limitations in generating diverse large-scale datasets like ImageNet. We propose hypothesis that the effectiveness of implicit adversarial learning in capturing the joint distribution of variables at adjacent steps is limited. This limitation arises from the fact that the discriminator needs to operate on the high-dimensional concatenation of adjacent variables, which can be challenging.
In order to achieve fast sampling speed and the ability to generate large-scale datasets, we introduce a novel approach called Semi-Impliee Denoising Diffusion Model (SIDDM). Our model reformulates the denoising distribution of diffusion models and incorporate implicit and explicit training objectives. Specifically, we decompose the denoising distribution into two components: a marginal distribution of noisily sampled data and a conditional forward diffusion distribution. Together, these components jointly formulate the denoising distribution at each diffusion step. Our proposed SIDDMs employ an implicit GAN objective and an explicit 1.2 reconstruction loss as the final training objectives. The implicit GAN objective is applied to the marginal distribution, while the explicit L2 reconstruction loss is adopted for the conditional distribution, where we name the process of matching conditional distributions as auxiliary forward diffusion AFD in our obejetives. This combination ensures superior training convergence without introducing additional computational overhead compared to DDGANs. To further enhance the generative quality of our models, we incorporate an Unet-like structure for the discriminator. Additionally, we introduce new regularization technique that involves auxiliary denoising task. This regularization method effectively stabilizes the training of the discriminator without incurring any additional computational burden.
In summary, our method offers several key contributions. Firstly, we introduce novel formulation of the denoising distribution for diffusion models. This formulation incorporates an implicit and an explicit training objectives, enabling fast sampling while maintaining high-generation quality. Lastly, we propose new regularization method specifically targeting the discriminator. This regularization technique enhances the overall performance the model, further improving its generative capabilities. Overall, our approach presents a comprehensive solution that addresses the challenges of fast sampling, high generation quality, scalability to large-scale datasets, and improved model performance through proper regularization.
2 Background
Diffusion models contain two processes: a forward diffusion process and the reversion process. The forward diffusion gradually generates corrupted data from xp ~ g(zo) via interpolating between the sampled data and the Gaussian noise as follows:
T
((11r|10)) = II (())
g(<c|xr-1) == -N(z;; V1 B,Ir-1,
1=1
((z2)20)= = Nax Varzo. (1- a,),,, ā := (1 B.),
(I)
==1
where T denotes the maximum time steps and Br E (0, 1] is the variance schedule. The parameterized reversed diffusion can be formulated correspondingly:
T
Po(Xo.T) == po(xT) II
Po (x, 126,, Po(2--1la)) == N (1 o.
(2)

---

where we can parameterize po (=( as Gaussian distribution when the noise addition between each adjacent step is sufficiently small. The denoised function po produces the mean value of the adjacent predictions, while the determination of the variance relies on Br. The optimization
objective can be written as follows:
L Fyyoo)
150
which indirectly maximizes the ELBO of the likelihood (o())). When Ti is given, the posterior 0(1e-1/=4, Zo) is Gaussian. Thus, the above objective becomes L2 distance between the sampled Ft-1 from the posterior and the predicted denoised data. However, we want to achieve fast sampling with small number of steps, The assumption that po (0 |2,) follows Gaussian does not hold [ and the 122 reconstruction cannot be applied to model the K. divergence.
To tackle this, DDGANs employ an adversarial learning scheme to match the conditional distribution between g( (zx)) and (= lar, via a conditional GAN and enable random large noise addition between adjacent diffusion steps for few-steps denoising. Their formulation can be summarized as follows:
min Dode (et)
where Dadu tries to distinguish the difference between the predicted and sampled denoising distribu- tion, while the predicted model tries to make them less distinguishable. The objective above can be rewritten as the following expectation:
min max D&
E wlw.o//momlloooaeeeeooo
t))]
+ BP log(1 2e( (5)
While this formulation allows more flexible modeling of po (zt-1 1 the pure implicit adversarial learning the concatenation of rz-1 and 2; is statistically inefficient, especially when rq is high- dimensional. We bypothesize that this a major reason why DDGANs cannot scale up well on the large-scale dataset with more complex data distributions. In contrast, will explore the inherent structure in the forward difussion process to develop a more efficient semi-implicit method, which is detailed in the following section.
Semi-Implicit Denoising Diffusion Models
This section will present proposed semi-implicit denoising diffusion models (SIDDMs). We first discuss how we reformulate the denoising distribution, which enables fast sampling as DDGANs and high-quality generation as DDPMs. Then, we will introduce how optimize our model in the training time. At last, we introduce a free discriminator regularizer to boost the model performance further. We show the simplified model structure in Figure I
3.1 Revisiting Denoising Distribution and the Improved Decomposition
Let us reconsider the training objective presented in Equation 5 which can be reformulated as follows:
min Dadv (g(xx (o:
(6)
where DDGANs' formulation indirectly matches the conditional distribution of g( (xx) and pe(xz-1lzz) via matching the joint distribution between g(#,-1, and (1x To keep it concise, we use (o( to represent po( afterwards.
Starting from this, we can factorize the two joint distributions in the reverse direction and get q(I2-1,
(((=))) and po(2:-1, Po (2 ( (xtr-1)). The conditional distri- butions are forward diffusion; we name them auxiliary forward diffusion (AFD) in our distribution matching objectives. In this decomposition, we have a pair of marginal distributions of denoised data q(F+=1),Po(F;-) and pair of conditional distribution ( (.,- po (x Because the marginal distributions do not have explicit forms, can match them implicitly by minimizing

---

++
Fake/ True
t(z,t,:)
+
(a) DDPMs.
(b) DDGANs.
Dia 1.1)
Fake/
30 1 Tr
((F
C(zen11t)
1
Auxiliary Forward Diffusion
(C) SIDDMs (ours).
Figure 1: In this figure, show three different diffusion models, which are DDPMs, DDGANs and our SIDDMs. These models share some common structures and diffusion processes. Our model shows improved decomposition of the denoising distribution with the adversarial marginal matching and the auxiliary forward diffusion (AFD) for conditional matching.
the Jensen-Shanma divergence (JSD) via adversarial learning. For the conditional distribution of forward diffusion, since q (1))) has an explicit form of Gaussian, can match them via KL. The following theorem states that matching these two pairs of distributions separately can approximately match the joint distribution.
Let q(E1-1.21) and po (o( z1) denote the data distribution from forward diffusion and the denoising distribution specified by the denoiser Gg. respectively, and we have the following inequality:
Theorem
Po
- 2c1 21SDD((
+ 202v 2D (De (2, (g((5((tt=))
where and are upper bounds of [g(xx//; [u(( and 1 f |p(x,- (2((2=)))))) is o-finite measure), respectively. A proof of Theorem Iis provided in Section II the Supplementary.
3.2 Semi-Implich Objective
Based on the above analysis, we formulate our SIDDMs distribution matching objectives as follows:
min Dede (edeDaalaaaaaa (g(=+=))llp(((==))) + -
where the XAFD the weight for the matching of AFD. In Equation the adversarial part Dady the standard GAN objective. To match the distributions of AFD via KL, expand it as:
po(x 21-1) log (2, / pe(x,, (0 log (x)
DEL DxL/foorr/la===ll(eezeezz
- (pe
(po( (2) (x)))))
(8)
which is the combination of the negative entropy of (1))))))1)) and ((t)) In our scenario, optimizing the cross-entres cross-entropy term is straightforward because we can easily represent the cross-ent cross-entropy between the empirical and Gaussian distributions us- ing mean square error. This is possible because the forward diffusion, denoted as q( (x follows Gaussian distribution. However, the negative entropy term - -H (Po(p) (x+11) is intractable. How- ever, (ay (x+-1) can be estimated on samples from the denoiser that models pa (ay). Thus need another parametric distribution Py (x=(0;)1) to approximately compute Hf (p)
and the cross entropy between

---

In the following formulation, we show the maximizing of the conditional entropy can be approximated by the following adversarial training objective:
min max log Po (112)
B
Given 0 the i-th iteration, the max step estimates the conditional distribution Pg- (Irg ((0e(2211)) and we have Per (Telzz-1) po (zz| Then, in the next iteration, the min step updates the parameter 0 in the generator given v² and obtain the updated 0i+1 Thus, this iterative min-max game between the generator P9 and the conditional estimator Pe can minimize this negative conditional entropy - [p ( [=]) that mentioned in the DKz decomposition of Equation g This adversarial process can perform as long as we can access the likelihood to (=(2)(2111) In our case, it forward diffusion and follows the Gaussian distribution.
Similar to DDGANS, we also define (o == 124,-1124 2o Go(a,, 8)) via the posterior distribution. In the distribution matching objective above, we apply the GANs to minimize the JSD of marginal distributions and the L2 reconstruction to optimize the cross entropy. We also define r/-1 1 as the data sampled from the newly defined distribution, and z are sampled from 2'-1 via forward diffusion. Our final training objective can be formulated as follows:
min max
Σ 120
Bodtn)))dees--1lbbolllleee [nog( (2t-11111 f))l + [og(( - D (=(
8 Co.Co
+DAFD (1- Bz) | - xz - (c)( x''l3
(10)
3g
where Cg denotes the regression model that learns to minimize this negative conditional entropy. In the implementation, share the most layers between the discriminator and the regression model. We put the detailed derivation of the following training objective in Section I of the supplementary.
Compared with DDGANs, our model abandons the purely adversarial training objective and de- composes into a marginal and one conditional distribution, where the conditional distribution can be optimised with less complex training objective and leads to stable training for updating the denoising model. Secondly, our model can share the same model structure as DDPMs and be improved based on the advanced DDPMs network structure. Thirdly, our decomposition does not bring any overhead compared to DDGANs and can achieve steady performance improvement.
3.3 Regularizer of Discriminator
The UnetGANs 12
proposes adopting an Unet structure for the discriminator, demonstrating more details in the generated samples. Unlike the common design of discriminators, which only output global binary logit of "True/Fake", an Unet-like discriminator can distinguish details from different levels. The denoising process in the diffusion models can also benefit from pixel-level distribution matching. Thus, our paper shares the same network structure between the denoiser Go and the discriminator Da- Inspired by our decomposition formulation, the reconstruction term provides better gradient estimation and boosts the model performance. We also apply the same strategy to the discriminator as a stand-alone contribution. That is, getting the denoising output from the discriminator and reconstructing it with the ground truth xo. We formulate the regularizer follows:
min Bu(zo)((=1-11o0))) (mo) Lz DD D3
zo):
(11)
where this regularization only applies to the sampled data q(x:). Different from the commonly used spectral norm [33, Wasserstein GANs 14] and RI regularization [15] our regularization does not bring any side effect, such restriction model capacity, requiring additional overhead or grid search of the ppper-prraeeters on each dataset. Our regularizer can be casily plugged into our model and DDGANs and does not require extra network design, which is specifically designed for boosting diffusion models with GANs.
Related Works
The pioneer works 10 I of the diffusion models introduce the discrete time-step diffusion models and the parameterized reversal process for generating novel data. Later, the score-based perspec- tive [16] proposes the continuous-tin diffusion models and unifies the denoising diffusion model
5

---

and the denoising score matching models [8. The adversarial score matching [99] utilizes the extra adversarial loss to improve the unimodal Gaussian distribution matching, which differs from the goal of DDGANs and ours. Another branch of works introduces some inductive biases for improving the diffusion models further, the ADM, UDM and EDM 2 20 21 propose the classifier guidance, unbounded score matching and better sampling strategy for diffusion-based models respectively.
Although diffusion- -based models achieve better quality and diversity, it still suffers from the sampling speed, which usually takes thousands of steps to achieve the best quality. Several methods also boost
the sampling speed via knowledge distillation [2 231, learning the nosing schedule of forward
diffusion (24]. The DDIM [25] and FastDPM [26 propose Mon-Markovian diffusion processes
for the sampling boosting. For the score- base- continuous-time diffusion models, [27] provides faster SDE solvers. The DDGANs [IT] is most related to our proposed methods, which proposed an adversarial formulation for the denoising diffusion distribution and enable a few sampling steps
without compromising the generated quality too much.
Unlike DDGANs, we propose new decomposition for the denoising distribution and achieve better results. From the perspective of better distribution decomposition, we share a similar insight with the related works of conditional GANs (cGANs) 28 29 AC-GAN 28 proposes to model the conditional distribution of conditional generative models via decomposing the joint between the label and the data to marginal and the conditional distribution via an auxiliary classifier. TAC-GAN lately fix the missing term of the AC-GAN decomposition. These two works can only be applied to the conditional generation while our proposed decomposition can be applied both unconditional and conditional generation. Thus, their works are fundamentally different from ours. The other related GANs propose to enhance GANs with data argumentation BD 31 or diffusion noise (32, which would also explain why GAN-ehanced denoising diffusion process can function well in practical.
5 Experiments
In our experiments, we evaluate our proposed method in the simulated Mixture of Gaussians and several popular public datasets, Cifar10-32 [33], CelebA-HQ-256 (34] and the (m0geNet10000641 To study the effects of our model components, we also conduct the ablations to identify their sensitivity and effectiveness.
For the model architectures, we apply the Unet [36] as the ADM (2), and we follow the same efficient strategy as Imagen [4] change the order of downsampling and upsampling layer in the Unet blocks. In our method, we also design our discriminator as an Unet. Thus, we apply the identical structure for the generator G and the discriminator D. The only difference between them the input and output channels for fitting in different inputs and outputs in our formulations.
5.1 MOG Synthetic Data
Identifying the effectiveness of generative mod- els in nigh-dimensional data is tricky, and we cannot directly visualize the mode coverage of the generated data. Also, popular metries like FID and Inception Score can only be referred as quality evaluation. Thus, test our mod- els and the baseline DDGANS on generating Mixture of Gaussians. To generate the data, we sample 25 groups of data from the Gaus- sians independently and each distribution has different mean but with the same variance. In Figure 3, we show the synthesized results of different models and the denoising steps and the quantitative results are shown in Table [ We can observe that for the extremely few denoising steps of 2 and many as 16 steps. DDGANs fail to converge to real data distribution, and our pro- posed method can recover well on the different steps. Also, to identify if the adversarial term takes the main job in the generation, we remove the auxiliary forward term and find that our model cannot recover the original distribution, proving the proposed decomposition's 's effectiveness. To evaluate our model's 's effectiveness on real-world generation tasks, we test our models on the tiny image dataset CIFAR1O, the fine- ine-grained generation task on the Celeb-HQ, and the large-scale challenging dataset Imagennet. We pick small amount of generated images from our model and
Model steps
2
16
vanilla GANs
12.04
DDGAN [I] (IDDMs(ours)
7.27
0.99
0.49
0.53
0.14
1.21
0.30
0,23
SIDDMs w/o AFD (ours)
21.19
53.22
7.04
14,37
Table 1: MOG 5x5 results, FID

---

(a) Generated samples CIFAR 10.
(b) Generated samples CelehHQ.
220
(c) Generated samples on ImageNet.
Figure 2: pick subset images generated from our model, which produces the paper results. From the top left to the bottom, they are the generated results on CIFARIO. Celeb-HO HQ and ImageNet.
show them in the Figure 2 We also quantitatively evaluate the quality the generated samples and collect all of the results in Table 2 and 4 Visually, our model generates samples with high fidelity and rich diversity. For the evaluation metric, we choose the Fréchet inception distance (FID) 47 and Inception Score [48] for sample fidelity and the improved recall score 49 to measure the sample diversity. Compared with the GANs-based method, our method achieves better sample quality and diversity, identifying the benefit of enhancing GANs with the diffusion formulation. Compared with the baseline DDGANs, our model shows superior quantitative results than the baseline DDGANs. Although our method does not reach the same quality the diffusion- (or score-based) models
7

---

DOCGAA
Tw2
T:36
1+31
Figure 3: We show the generative results for the 5 x 5 Mixture of Gaussians, which can straight- forwardly show the effectiveness of our proposed method. We include our full model, our model without the auxiliary forward diffusion term and the baseline DDGANs. Our model can recover the original distribution even in the small diffusion steps of 2, but the DDGANs fail. Also, without the completed formulation, our model creates biased results.
Model
IST
FIDJ
Recallt
NFE
Time (s)
Model
FIDJ
DDGANS (1) T=1
9.63
3.75
0.57
0.20
SIDDMs (oars)
7.37
Diffiusionn models
DDGANs
9.46
Score SDE
16
7.23
EDM
9.84
2.04
36
Score SDB (VF)
9,89
2.20
01.99
2000
423.2
UDM
20
7.16
Score SDE (VP)
9.68
2000
Insrowed
4000
NVAE
29.7
UDM
20
10.1
2.33
2000
VAFRM 31
20.4
Distillation Diffusion Models
DDPM Distilation
3.00
PGGAN 43
8.03
CD 22
2.93
9.75
VQ-GAN
45
10.2
models
StyleGAN2 w/o ADA 31
9.18
8.32
0.4
0.04
Table 3: Generative results on CQl-bA-HQ-256
Sty SlyleG w/ ADA RI
983
2.92
0.49
0.04
StyleGiAN
Table 2: Generative results on CIFAR10
with small gap. our model still has the large advantage of fast sampling. To be notified, we apply four denoising steps for the CIFAR10 and two steps for the CelebA-HQ generation, identical the DDGANs main results.
5.2 Generation Real Data
For the ImageNet, choose four denoising steps to get the best sam- ple quality on the ImageNet. ImageNet has 1000 categories, each containing thousands of images and distinguishing itself as large- scale dataset with diversity. To train generative model with high fidelity on the dataset, we usually require other inductive biases such as regularizers model scaling-up methods [3] 50). The DDGANs fail retain the high-fidelity samples on this dataset without additional inductive bias. However, with the same model capacity, our proposed method can achieve comparable results w.r.t. the diffusion-basser mod- els. This result shows that our model can handle large-scale generation and potentially be applied boost the sampling speed of large-scale tex{2image models.
Model
FID,
SIDDMs (ours)
3.13
DDGANs (III)
20.63
CT 2P]
11.10
ADM 2
2.07
EDM 21
2.44
Improved DDPM
37
2.90
CD
4.07
Table Generative results on ImageNet1000

---

Ablations
Table 5: Study the effect of Auxiliary Forward Diffusion (AFD), FIDJ. The weight of 0.0 means that we remove the AFD term and only train the model with adversarial loss. The weight of oc means we only keep the AFD term without adversarial loss. This table also evaluates our model performance with different denoising steps and without our regularizer for the discriminator. All of the scores are evaluated on the CIFARIO.
Weight of AFD
0.0
0.1
0.5
1.0
5.0
SIDDMs<ours)
2.55
Diffusion Steps
8
16
32
SIDDMs(ours) wio
SIDDMs(ouns)
27.39
3.17
2,21
2.31
2,15
2/16
Regularizer of Discriminntor
51.67
4.64
4.79
3.20
2.95
$1.55
(a) AFD weight 0.0.
(b) AFD weight 1.0.
(c) AFD weight cc.
Figure 4: Generated samples on CIFARIO, we show two severe cases where only adversarial term (weight 0.0) and only AFD term (weight oo). We also show full model results in the middle for comparison.
To identify the function of the components in our formulation, we conduct controlledexperi- ments on the effects of the adversarial and the AFD term. We set the weights of AFD be [0.0.0.1 0.1, 0.5, 1.0, 5.0, ool, where 0.0 represents only the adversarial term and oo denotes only the AFD term in our training. We apply the same sampling strategy as our full models and adopt four denoising steps. The FID scores are reported in Table and the generated samples are shown in Figure We can see that if missing any of these two terms in our formulation, we cannot recover the original image distribution. In addition, we found that our model is not sensitive to the weights between these two components w.r.t. the FID scores when both terms participate in the training, which further identifies the effectiveness of our formulation. Also, we train the model without the regularizer for the discriminator, and we also identify that the proposed auxiliary tasks for the discriminator further enhance the model performance for our full formulation.
7 Conclusion
In conclusion, the quest for fast sampling with diverse and high-quality samples in generative models continues to pose a significant challenge. Existing models, such as Denoising Diffusion Probabilistic Models (DDPM), encounter limitations due the inherent slowness associated with their iterative steps. On the other hand, Denoising Diffusion Generative Adversarial Networks (DDGAN) faces scalability issues when dealing with large-scale datasets. To address these challenges, we propose a novel approach that effectively addresses the limitations of previous models by leveraging a combination of implicit and explicit factors. Specifically, we introduce an implicit model that enables us to match the marginal distribution of random variables the reverse diffusion process. Additionally, we model explicit distributions between pairs of variables in reverse steps, which allows us to effectively utilize the Lullback-Leible (KL) divergence for the reverse distribution. To estimate the negative entropy component, incorporate a min-max game into our framework. Moreover, we adopt the L.2 reconstruction loss to accurately represent the cross-entropy term in the KL divergence. Unlike DDPM but similar DDGAN, we do not impose a parametric distribution for the reverse step in our approach. This design choice empowers us to take larger steps during the inference process, contributing to enhanced speed and efficiency. Additionally, similar to DDPM, we effectively leverage the exact form of the diffusion process to further improve our model's 's performance. Our proposed approach exhibits comparable generative performance to DDPM while surpassing models with fewer sampling steps.

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[2] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image symthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[4] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[5] Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[6] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.
[7] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
[8] Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors. Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[10] Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamies. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265, Lille, France, 07-09 Jul 2015.
PMLR.
[11] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations, 2022.
[12] Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8207-8216, 2020.
[13] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[14) Martin Arjovsky, Soumith Chintala, and Léon Bottou. arxiv:1701.0782 777777
Wasserstein gan, 2017. cite
[15] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7559-7570. Curran Associates, Inc., 2020.
10

---

[16] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[17] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compulation, 23(7):1661-166) 2011.
[18] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. Curran Associates Inc., Red Hook, NY, USA, 2019.
[19] Alexia Molicoeur-Martineau, Rémi Piché-Taillefee Ioannis Mitliagkas, and Remi Tachet des Combes. Adversarial score matching and improved sampling for image generation. In Interna- tional Conference on Learning Representations, 2021.
[20] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Score matching model for unbounded data score. arXiv preprint 106i::200.00527,, 2021.
[21] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
[22] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.
[23]
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[24] Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models. arXiv preprint (xXxx:2104.026000 2021.
[25] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[26] Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint drXiv:2106.00//33 2021.
[27] Alexia Jolicoeur- Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefés Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.1400 2021.
[28] Augustus Odema, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans, 2017.
[29] Mingming Gong. Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 1330- 1339. Curran Associates, Inc., 2019.
[30] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in neural information processing systems, 2020.
[31] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in neural information processing systems, 2020.
[32] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gam Training gans with diffusion. arXiv preprint arXiv:2206.02262 2022.
[33] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[34] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CORR, abs/1710.10196, 2017.
[35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211252 2015.

---

[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015, cite Srdd:1150.045977oommee. conditionally accepted at MICCAI 2015.
[37] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, arXiv preprint arXiv:2102.09672 2021.
[38] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02238, 2021.
[39] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in neural information processing systems, 2021.
[40] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances neural information processing systems, 2020.
[41] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy- -baed models. In International Conference on Learning Representations, 2021.
[42] Jyoti Aneja, Alexander Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Neural Information Processing Systems, 2021.
[43] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representa- tions, 2018.
[44] Stanislav Pidhorskyi, Donald Adjeroh, and Gianfranco Doretto. Adversarial latent autoen- coders. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2020.
[45] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE conference computer vision and pattern recognition, 2021.
[46] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEFE conference on computer vision and paltern recognition, 2021.
[47] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 2017.
[48] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 2016.
[49] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. arXiv preprint VrXXiv1904.000 06991, 2019.
[50] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. volume abs/2201.00277 2022.
12


Reviews:
[{"review_id": "xBuFiBSaF6", "reviewer": "Reviewer_azcW", "summary": "The paper addresses the issue of making diffusion models faster (faster sampling) while still yielding high quality, diverse samples, from a large phenomenological space (scaling to large datasets). This is achieved by a novel semi-implicit denoising diffusion model (SIDDM). The idea extends the approach followed in DDGANs by reformulating the objective, and a novel decomposition into two components: (a) a pair of marginal distributions over the denoised data (at each step) with an implicit form to be aligned with the JSD via a learnt critic, and (b) a pair of conditional, forward diffusion components having explicit forms to be aligned using the KL divergence. This mixed objective allows better scaling than DDGANs. A novel regularizer is also introduced for the discriminator which allows a more granular distribution matching. A proof of concept validation is performed using a synthetic, Mixture of Gaussians dataset. Ablations demonstrate the benefit of both the innovations.  SIDDM is benchmarked against the art on the CIFAR10, CelebA-HQ-256, and ImageNet datasets demonstrating performance competitive with DDGANs, better scaling to ImageNet, and much higher sampling speeds than classical DDPMs. \n\n", "questions": "Kindly address the following:\n- Explain the behavior on the MoG dataset using a larger suite of metrics (see Borji, CVIU 2019; and its 2021 update for example). \n- Explain the behavior (non-convergence to data distribution) when using a larger number of steps and the scalability implications. \n- Did you investigate the tradeoffs between a smaller step size with larger number of steps, and smaller number of steps with a larger step size? How does one decide this? Does the theoretical formulation lend itself to such a guidance?\n-  Kindly share details of the hardware, implementation, and other details which will aid reproducibility.\n- Will the code be shared?\n- Discuss limitations. \n", "limitations": "Authors don’t address limitations of the paper. \n\nI don’t think there are any direct negative societal implications. Other limitations and opportunities for improvement are addressed in my responses to previous questions.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Relevance** The paper addresses an important problem – speeding up diffusion based generative image models while not compromising on the image quality, with good scalability to large, complex datasets. This should be of relevance to the community working in these areas but also of interest to a much wider audience. \n\n**Originality** The proposed decomposition is interesting and novel, balancing both a non-parametric, population-level statistical alignment of distributions and a direct (simple, parametric) objective which decomposes into a sample level objective over Auxiliary Forward Diffusion (AFD). It stands to reason that this can have a significant impact on the training of the diffusion reversal DNN.   \n\n**Technical Quality** \n- The technical approach mostly appears sound. It creates a new, direct and stronger learning signal which augments the critic based learning objective. This can lead to better learning outcomes over more complex distributions representing larger datasets like ImageNet. In addition, regularizing the discriminator via the UnetGAN formulation seems reasonable too. \n- The mathematical formalism used in the paper, including the upper bound in Theorem 1, add to the strength of the contribution, making it principled, and beyond the heuristic choices made. I haven’t verified the derivations in the paper entirely but have followed the structure of the derivations and the proof and find them reasonable.\n\n**Experimental Validation** There are several positives in the choices made for evaluation:\n- The choice of using a synthetic dataset based on Mixture of Gaussians (MoGs) allows for validating the core idea.  \n- While the results on CIFAR10 and CelebA-HQ-256 are comparable with DDGANs, the FID scores on ImageNet demonstrate a performance comparable to SOTA diffusion models while DDGANs performance seems to demonstrate a failure to scale to ImageNet. This validates the core motivation of the paper. \n- The ablation study, with and without the AFD term, validates the benefit of the decomposition of the training objective that forms the basis of SIDDMs; as well as the benefit due to the UNetGAN regularizer in (14)     \n\n**Significance** The approach has the potential for significant impact over an important area of research once the approach has been reproduced and ‘hardened’.", "weaknesses": "**Experimental Validation** \n\n(a) The MoG experiment (Section 5.1) demonstrates that SIDDM achieves good results with very small number of time steps, and has better stability than DDGANs.  However,\n- It is not discussed why it is unable to converge to the simple MoG data distribution, when the number of time steps increases (similar to DDGAN).  \n- Table 1 shows the FID score. These trends don’t tally well with the behavior shown in Figure 3. Perhaps using other metrics (JSD, LPIPS, etc.) may explain the results better?  \n- The implications of the above on the learning on real datasets, and the scalability thereof on larger,  complex datasets like ImageNet is unclear. Such tasks may require more time steps (it is not clear whether this is the case) in which case, convergence to the data distribution becomes an issue.\n\n(b) The results on CelebA-HQ-256 are typically used to show performance on high-resolution data. The images (depictions) of the generated samples, both in the paper and in the supplementary, are too small. Similarly, the ImageNet depictions are very small.\n\n(c) While NFE (number of function evaluations) and the wall-clock time are shared, no details of the hardware are shared.\n\n**Clarity** (l. 230) It is not quite clear why the material on training with real datasets like CIFAR10, Celeb-HQ etc. is in Section 5.1 and not Section 5.2. Kindly fix.\n\n**Reproducibility** Implementation details are missing. The authors also don’t mention that the code will be released. Given this, I suspect it may be hard to reproduce results. \n\n**Discussion of Limitations** No discussion of limitations in the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Explain the behavior on the MoG dataset using a larger suite of metrics\n\nWe pick the Unbiased FID mentioned in Borji, CVIU 2021 and MMD metric for our evaluation of the MOG generative quantitation, and the results are shown in Table 6 MMD is a kernel-based nonparametric metric to measure the difference between two distributions. It has nice theoretical properties and is suitable for the MoG data because here we do not need deep networks to learn representations. For the additional results for Unbaised FID and MMD, we found the scores are pretty close and our model still perform overall better than the DDGAN. There are also other advanced metrics in Borji, CVIU 2021, but some are specifically designed for image datasets or conditional generation cases.\n\n> Explain the behavior (non-convergence to data distribution) when using a larger number of steps and the scalability implications\n\nIn the baseline model DDGANs [11], they also observe that increasing the number of diffusion steps degrades the generative quality. The authors of DDGANs hypothesize that increasing the number of diffusion steps needs more capacity for Discriminator as we need a conditional GAN for each denoising step, where Conditional GANs are difficult to be trained when the number of category labels is large. However, the exact reason still remains unknown. According to our simulation results, it is possible that increasing the number of steps can lead to worse results. Fortunately, we observe that with a small number of diffusion steps our method can already obtain high-quality generations. In our paper, we train four-step diffusion on the Imagenet1000, and here we additionally show the preliminary results of our SSIDMs on the text-conditional Laion4B dataset with a small UNet. The preliminary results on a small UNet are shown in rebuttal pdf Figure 6.\n\n>  the tradeoffs between a smaller step size with larger number of steps, and smaller number of steps with a larger step size\n\nIn DDGANs and our proposed method, we do not have the Gaussian assumption for $p_{\\theta}(x_{t-1}|x_t)$. Thus theoretically, the model trained on any step size with the corresponding number of steps would result in the same distribution convergence at the end of training. However, this claim is based on the assumption that GANs would converge to the optimal, which is often empirically impossible. Our choice of step size with its corresponding number of steps mainly depends on the empirical and DDGANs' observations (we show additional results for the sensitivity of steps in the rebuttal pdf Table 7). While we agree that a thorough theoretical understanding of this issue is essential, conducting this analysis is non-trivial and we will definitely consider it in the near future.\n\n> Details of the hardware, implementation, and other details\n\nOur models are trained on the TPUv4 clusters. A single TPUv4 is around $1.2\\times$ faster as A100. And the code is implemented with JAX. Our model structure mostly follows the ADM [2] attention UNet which is also followed by IMAGEN. But instead of predicting the noise from our $G$ model, our model directly reconstructed $x_0'$. We have put our simulation code in the supplementary \".zip\" file containing our paper's main formulation. We will release the code for public research once our work gets into the final stage.\n\n> Discuss limitations\n\nOur model incorporated with UNet-like discriminator, and the discriminator needs to see both fake and real data. Thus we have at least two times memory usage than the Diffusion model. Also, in the GANs' training, the model contains two stages of training which results in at most half the training speed for each batch iteration compared with DDPM. These facts would lead to more CO2 emission. However, Distillation on the DDPM model would also take the same amount training time as training DDPM. Thus compared with Distillation+DDPM, our model probably cost the same amount energy and time but with less performance degradation."}, {"review_id": "8numBKTHK1", "reviewer": "Reviewer_g6aX", "summary": "The paper presents a method called Semi-Implicit Denoising Diffusion Model (SIDDM) aimed to accelerate the sampling process, enhance scalability to large datasets, and improve model performance. It improves upon the DDGAN model by reformulating the denoising distribution of diffusion models with explicit and implicit training objectives, which leverages an implicit GAN objective for the marginal distribution and an L2 reconstruction loss for the conditional distribution. To improve generative quality, the authors adopt a U-net-like structure for the discriminator and a new regularization technique involving an auxiliary denoising task. Experiments are conduct on CIFAR-10, CelebA-HQ-256, and ImageNet to demonstrate it effectiveness.", "questions": "Please see the weakness part.", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The experiment and comparison were comprehensive, and the results were good. Ablation was comprehensive.\n(2) The writing has a clear structure (just not sure if there are any issues with the derivation).\n(3) The method performs quite well on large datasets like ImageNet, whereas the previous ones didn't do well.", "weaknesses": "(1) There lacks a graph showing the FID-sampling speed tradeoff on ImageNet.\n(2) In Table 1, the 'SIDDMs w/o AFD (ours)' entry is quite close to DDGAN in terms of setting, but why is there such a big difference in performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. FID-sampling speed tradeoff on ImageNet.\nWe have shown the results of FID sample steps in Table 7 of the rebuttal pdf.\n\n2. 'SIDDMs w/o AFD (ours)' has a large gap on the performance compared with DDGANs.\nThis is a good question; In Equation 7, our GAN objective models distribution between marginals and the AFD models the matching between conditionals. They together match the joint distribution between $q(x_{t-1}, x_t)$ and $p_{\\theta}(x_{t-1}, x_t)$. Matching the distribution of joints can sufficiently guarantee the matching between the conditionals of $q(x_{t-1}|x_t)$ and $p_{\\theta}(x_{t-1}|x_t)$. In the DDGANs, they are also modelling the matching between the joint distribution. However, if we abandon the AFD, it results in only matching the marginals between $q(x_{t-1})$ and $p_{\\theta}(x_{t-1})$, which will not guarantee the matching between the conditionals of $q(x_{t-1}|x_t)$ and $p_{\\theta}(x_{t-1}|x_t)$. Ultimately, 'SIDDMs w/o AFD (ours)' will output biased distribution. "}, {"review_id": "D9JdZgk1vF", "reviewer": "Reviewer_SF8Z", "summary": "In this paper, the authors propose to use an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. Specifically, the adversarial loss is applied to the marginal distributions obtained by forward and backward process, and KL loss is used to regularize the different between the conditional distribution. Experiments show that the method works with small inference steps.", "questions": "Please see weakness part.", "limitations": "Please see weakness part.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Using the adversarial loss is a good idea to learn the conditional distribution of the backward process with large step size.\n* The experimental results with small steps seem to be good, both for toy examples and real datasets.", "weaknesses": "* Generally, the paper is hard to follow, the formulas should be presented in a better way:\n    * How to obtain equation (6) through (5)?\n    * Line 129-137 is really difficult to follow, since $q(x_t|x_{t-1})$ is Gaussian distribution, no matter the step size, why not directly use a Gaussian distribution to parameterize $p_\\theta(x_t|x_{t-1})$?\n    * It problematic to treat equation (11) and (12) as equivalent, since $\\psi$ is fixed in (11). \n\n* In the experiments, with smaller steps, the method works better for both the toy and real datasets. It is not reasonable. Any explanation about this phenomenon?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi, thanks for the questions, we would love to make the details more readable. Also, due to rendering issue, we place it in the rebuttal pdf.\n\n> Line 129-137 is really difficult to follow\n\nWe agree with the reviewer that those line are not written well. Here is our new rewrite and we hope it clarifies that paragraph:\n\nThere are two terms in Eq 8. The second term, $H(p(x_t|x_{t-1}),q(x_t|x_{t-1}))$ is a cross entropy which is simply reconstruction loss. This loss can be easily computed because the $q$ generates data, and $p$ does the denoising using $G$. The first term, $H(p (x_t | x_{t-1}))$ is challenging to estimate. The challenges stem from two facts. First, we $p (x_t | x_{t-1})$ is unknown in general, and second, estimating entropy entails computing a high-dimensional integral. However, we observe that  *at the convergence*, $p (x_t | x_{t-1})$ is the same $q (x_t | x_{t-1})$ which is a Gaussian distribution, and we only need to estimate its parameters, $\\psi$. Our method can be viewed as a continuous generalization of the method proposed in [29]. \n\n> Obtaining equation (6) through (5).\n\nPlease refer to the Equation (A) in the rebuttal pdf, where we can rewrite Equation 5 as a typical adversarial training objective with non-saturated loss in the middle expectation, and then, it becomes the joint distribution JSD matching under the sampling strategy proposed by DDGANs. Here we have to admit that  Eq (6) has a typo, where the $E_{q(x_0)q(x_{t-1}|x_0)q(x_t|x_{t-1})}$ in the last joint matching equation should not appear in it. Thanks for your carefully checking.\n\n> It problematic to treat equation (11) and (12) as equivalent\n\nWe are sorry for the confusion. We re-wrote that part as the Equation (B) in the rebuttal pdf.  Due to markdown rendering issue, we represent $E$ as the expectation symbol. The $H(X)=E_{q(X)}\\log q(X)$ typically represents the entropy for variable $X$. And $H(X,Y) = E_{q(Y)}\\log q(X)$  denotes the cross-entropy between two variables $X,Y$. In Equation (10), we have the maximization for the negative cross-entropy and optimize $\\psi$, which will result in $p_\\psi(x_t|x_{t-1})= p_\\theta(x_t|x_{t-1})$ at the optimal. In practice, there might be possible approximation error, i.e., $p_\\psi(x_t|x_{t-1})\\approx p_\\theta(x_t|x_{t-1})$. Thus under this condition, if we try to minimize the cross entropy and optimize $\\theta$ while fixing $\\psi$, we can rewrite the minimization of the cross entropy as Equation (B).\n\n>. Smaller steps works better.\n\nIn the baseline model DDGANs [11], they also observe that increasing the number of diffusion steps degrades the generative quality. The authors of DDGANs hypothesize that increasing the number of diffusion steps needs more capacity for Discriminator as we need a conditional GAN for each denoising step, where Conditional GANs are difficult to be trained when the number of category labels is large. However, the exact reason still remains unknown. According to our simulation results, it is possible that increasing the number of steps can lead to worse results. Fortunately, we observe that with a small number of diffusion steps our method can already obtain high-quality generations. While we agree that a thorough theoretical understanding of this issue is essential, conducting this analysis is non-trivial and we will definitely consider it in the near future."}, {"review_id": "WJdMVORuOF", "reviewer": "Reviewer_3HLp", "summary": "This method propose a way to achieve fast sampling during inference without compromising sample diversity and quality of diffusion model, and shows their effectiveness on both conditional and unconditional generation, qualitatively and quantitatively. A theoretical framework is also proposed which looks reasonable to me.\n\n", "questions": "The overall story looks good to me. I wonder whether you would like to perfect your experiment with more qualitative/quantitative in the conditional setting.", "limitations": "As mentioned above, more analysis on the conditional setting is welcome, e.g., using Stable Diffusion as the base model.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This method basically proves that a very low FID can be obtained with very few sampling steps, compared to ADM and DDGAN. In addition, it claims that it can do both conditional and unconditional generation. Moreover, the theoretical framework looks solid to me, though I am not from the theory field so I cannot comment more on this.", "weaknesses": "However, it seems that there are no experiments on conditional generation (both qualitative and quantitative). Perhaps more relevant experiments are welcome to consolidate this submission and prove the effectiveness of the new sampling strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Conditional setting\n\nThanks for the agreement on our work and the suggestions. It is our mistake that we did not highlight our conditional experiments. In fact, our experiments and the comparisons are conditional generative models on the Imagenet1000  and we have some additional preliminary results on the text2image conditional generation on Laion4B of small UNet setting, shown in Figure 6 in the rebuttal pdf."}, {"review_id": "SWzIAaojbx", "reviewer": "Reviewer_6duC", "summary": "This paper proposes the Semi-Implicit Denoising Diffusion Model (SIDDM) to enable fast sampling while maintaining high-generation quality. Specifically, SIDDM applies an implicit model to match the marginal distributions of the reverse diffusion process. Meanwhile, SIDDM models explicit conditional distribution of the forward diffusion. Furthermore, a regularization method is proposed to enhance the performance. Experiments show that SIDDM has comparable performance to other diffusion models with fewer sampling steps.", "questions": "1. In Fig. 3 and Tab.5, the larger step (e.g., 16) performs worse than smaller step sizes (e.g., 2). This is different from the conventional situation in diffusion models. Please give some analysis.\n2. It is recommended to give the training and inference strategy (algorithm) of SIDDM.", "limitations": "The authors have discussed the societal impact in the supplementary material. It would be better if the authors discuss some limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors analyse the reasons for the limitations of DDGAN, and decomposite the denoising distribution to improve the training objective. The idea is reasonable.\n2. The experiments in the simulated Mixture of Gaussians and several popular public datasets demonstrate the effectiveness of the proposed method.\n3. The authors also provide the code for results reproduction, which shows the solidness of the work.", "weaknesses": "1. In Fig. 1, there are dashed lines of different colours in Figure 1. But the authors don't give some corresponding explanation. \n2. The implementation details are not clarified, like the structure of the regression model, the discriminator regularizer, and the denoiser. Meanwhile, the training settings (e.g., training iterations) are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  Larger step (e.g., 16) performs worse than smaller step sizes.\n\nIn the baseline model DDGANs [11], they also observe that increasing the number of diffusion steps degrades the generative quality. The authors of DDGANs hypothesize that increasing the number of diffusion steps needs more capacity for Discriminator as we need a conditional GAN for each denoising step, where Conditional GANs are difficult to be trained when the number of category labels is large. However, the exact reason still remains unknown. According to our simulation results, it is possible that increasing the number of steps can lead to worse results. Fortunately, we observe that with a small number of diffusion steps our method can already obtain high-quality generations. While we agree that a thorough theoretical understanding of this issue is essential, conducting this analysis is non-trivial and we will definitely consider it in the near future.\n\n>  Training and inference strategy (algorithm) of DDGAN.\n\nFor the training settings, we reimplement DDGANs with our proposed GAN training structure and found it would not stable during training without R1 constraint, including the time schedule and the network design. For the inference strategy, we follow the original DDGANs implementation with the posterior sampling which conditioned on the previous $x_t$ and the predicted $x'_0$ from the denoisier. To be more specifically, we write the following algorithm for the DDGAN \n\nTraining:\n\n1. Sampling $x_0\\sim q(x_0)), t\\sim \\text{Uniform}(\\{0, . . . , T-1\\}), \\epsilon_t\\sim N(0,1), \\epsilon_{t+1}\\sim N(0,1)$.\n2. $x_t=\\sqrt{\\bar \\alpha}x_0 + \\sqrt{1 - \\bar \\alpha}\\epsilon_t, x_{t+1}=\\sqrt{1-\\beta_{t+1}}x_t + \\sqrt{\\beta_{t+1}}\\epsilon_{t+1}$.\n3. $x_0'=G_\\theta(x_{t+1},t+1), x_t' \\sim q(x_t|x_0',x_{t+1})$, where $q(x_t|x_0',x_{t+1})$ is the posterior sampling from DDPM.\n4. D step: $\\nabla_\\phi (-\\log(D_\\phi(x_{t-1},x_t,t))-\\log(1-D_\\phi(x_{t-1}',x_t,t)))$\n5. G step: $\\nabla_\\theta (-\\log(D_\\phi(x_{t-1}',x_t,t)))$\n6. Repeat 1-5 until model converge.\n\nInference:\n1. $x_T\\sim N$\n2. for t = T,...,1 do\n\n   $\\epsilon \\sim N(0,1)$ if $t>1$ else $\\epsilon=0$,\n\n   $x_0'=G_\\theta(x_t,t)$,\n\n   $x_{t-1}' \\sim q(x_{t-1}|x_0',x_t)$\n\n   end for\n\n  return $x_0$,\n\nwhere we simply denote the posterior sampling of DDPM as $ x_{t-1}' \\sim q(x_{t-1}|x_0',x_t)$ due to the mark down rendering issue."}]

Author rebuttal:
We want to thank all reviewers evaluating the paper and will fully address all the reviewers' concerns. We put additional Tables and Figures in the rebuttal pdf file, please find the corresponding Tables and Figures for our reponses below. If our answers are satisfactory, we would be thankful if you could update your score. Otherwise, we are happy to answer any more questions.

Meta-review: There is an agreement between the reviewers to accept the paper. I think that the proposed approach is novel and the experimental validation is sufficient. Therefore, I recommend acceptance.
Decision: Accept (poster)