Title: Semi-Implicit Denoising Diffusion Models (SIDDMs)
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion Model

GAN

Semi-implicit Modeling
Abstract: Despite the proliferation of generative models, achieving fast sampling during inference without compromising sample diversity and quality remains challenging. Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. The Denoising Diffusion Generative Adversarial Networks (DDGAN) attempted to circumvent this limitation by integrating a GAN model for larger jumps in the diffusion process. However, DDGAN encountered scalability limitations when applied to large datasets. To address these limitations, we introduce a novel approach that tackles the problem by matching implicit and explicit factors. More specifically, our approach involves utilizing an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. This combination allows us to effectively match the joint denoising distributions. Unlike DDPM but similar to DDGAN, we do not enforce a parametric distribution for the reverse step, enabling us to take large steps during inference. Similar to the DDPM but unlike DDGAN, we take advantage of the exact form of the diffusion process. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.

Semi-Implicit Denoising Diffusion Models (SIDDMs)

Abstract

Despite the proliferation of generative models, achieving fast sampling during inference without compromising sample diversity and quality remains challenging. Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. The Denoising Diffusion Generative Adversarial Networks (DDGAN) attempted to circumvent this limitation by integrating a GAN model for larger jumps in the diffusion process. However, DDGAN encountered scalability limitations when applied to large datasets. To address these limitations, we introduce a novel approach that tackles the problem by matching implicit and explicit factors. More specifically, our approach involves utilizing an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. This combination allows us to effectively match the joint denoising distributions. Unlike DDPM but similar to DDGAN, we do not enforce a parametric distribution for the reverse step, enabling us to take large steps during inference. Similar to the DDPM but unlike DDGAN, we take advantage of the exact form of the diffusion process. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps. The code is available at https://github.com/xuyanwu/SIDDMs.

1 Introduction

Generative models have achieved significant success in various domains such as image generation, video synthesis, audio generation, and point cloud generation [1, 2, 3, 4, 5, 6, 7]. Different types of generative models have been developed to tackle specific challenges. Variational autoencoders (VAEs) [8] provide a variational lower bound for training models with explicit objectives. Generative adversarial networks (GANs) [9] introduce a min-max game framework to implicitly model data distribution and enable one-step generation. Denoising diffusion probabilistic models (DDPMs) [10, 1], also known as score-based generative models, recover the original data distribution through iterative denoising from an initial random Gaussian noise vector. However, these models face a common challenge known as the 'TRILEMMA' [11], which involves ensuring high-quality sampling, mode coverage, and fast sampling speed simultaneously. Existing approaches, such as GANs, VAEs, and DDPMs struggle to address all three aspects simultaneously. This paper focuses on tackling this TRILEMMA and developing models capable of effectively modelling large-scale data generation.

While diffusion models excel in generating high-quality samples compared to V AEs and demonstrate better training convergence than GANs, they typically require thousands of iterative steps to obtain the highest-quality results. These long sampling steps are based on the assumption that the reversed diffusion distribution can be approximated by Gaussian distributions when the noise addition in the forward diffusion process is small. However, if the noise addition is significant, the reversed diffusion distribution becomes a non-Gaussian multimodal distribution [11]. Consequently, reducing the number of sampling steps for faster generation would violate this assumption and introduce bias in the generated samples.

To address this issue, DDGANs [11] use a reformulation of forward diffusion sampling and model the undefined denoising distribution using a conditional GAN. This approach enables faster sampling without compromising the quality of the generated samples. Additionally, DDGANs exhibit improved convergence and stability during training compared to pure GANs. However, DDGANs still face limitations in generating diverse large-scale datasets like ImageNet. We propose a hypothesis that the effectiveness of implicit adversarial learning in capturing the joint distribution of variables at adjacent steps is limited. This limitation arises from the fact that the discriminator needs to operate on the high-dimensional concatenation of adjacent variables, which can be challenging.

In order to achieve fast sampling speed and the ability to generate large-scale datasets, we introduce a novel approach called Semi-Implicit Denoising Diffusion Model (SIDDM). Our model reformulates the denoising distribution of diffusion models and incorporate implicit and explicit training objectives. Specifically, we decompose the denoising distribution into two components: a marginal distribution of noisily sampled data and a conditional forward diffusion distribution. Together, these components jointly formulate the denoising distribution at each diffusion step. Our proposed SIDDMs employ an implicit GAN objective and an explicit L2 reconstruction loss as the final training objectives. The implicit GAN objective is applied to the marginal distribution, while the explicit L2 reconstruction loss is adopted for the conditional distribution, where we name the process of matching conditional distributions as auxiliary forward diffusion AFD in our obejctives. This combination ensures superior training convergence without introducing additional computational overhead compared to DDGANs. To further enhance the generative quality of our models, we incorporate an Unet-like structure for the discriminator. Additionally, we introduce a new regularization technique that involves an auxiliary denoising task. This regularization method effectively stabilizes the training of the discriminator without incurring any additional computational burden.

In summary, our method offers several key contributions. Firstly, we introduce a novel formulation of the denoising distribution for diffusion models. This formulation incorporates an implicit and an explicit training objectives, enabling fast sampling while maintaining high-generation quality. Lastly, we propose a new regularization method specifically targeting the discriminator. This regularization technique enhances the overall performance of the model, further improving its generative capabilities. Overall, our approach presents a comprehensive solution that addresses the challenges of fast sampling, high generation quality, scalability to large-scale datasets, and improved model performance through proper regularization.

2 Background

Diffusion models contain two processes: a forward diffusion process and the reversion process. The forward diffusion gradually generates corrupted data from x 0 ∼ q ( x 0 ) via interpolating between the sampled data and the Gaussian noise as follows:

where T denotes the maximum time steps and β t ∈ (0 , 1] is the variance schedule. The parameterized reversed diffusion can be formulated correspondingly:

where we can parameterize p θ ( x t -1 | x t ) as Gaussian distribution when the noise addition between each adjacent step is sufficiently small. The denoised function µ θ produces the mean value of the adjacent predictions, while the determination of the variance σ t relies on β t . The optimization objective can be written as follows:

which indirectly maximizes the ELBO of the likelihood p θ ( x 0 ) . When x 0 is given, the posterior q ( x t -1 | x t , x 0 ) is Gaussian. Thus, the above objective becomes L 2 distance between the sampled x t -1 from the posterior and the predicted denoised data. However, if we want to achieve fast sampling with a small number of steps, The assumption that p θ ( x t -1 | x t ) follows Gaussian does not hold [11], and the L 2 reconstruction cannot be applied to model the KL divergence.

To tackle this, DDGANs employ an adversarial learning scheme to match the conditional distribution between q ( x t -1 | x t ) and p θ ( x t -1 | x t ) via a conditional GAN and enable random large noise addition between adjacent diffusion steps for few-steps denoising. Their formulation can be summarized as follows:

where D adv tries to distinguish the difference between the predicted and sampled denoising distribution, while the predicted model tries to make them less distinguishable. The objective above can be rewritten as the following expectation:

While this formulation allows more flexible modeling of p θ ( x t -1 | x t ) , the pure implicit adversarial learning on the concatenation of x t -1 and x t is statistically inefficient, especially when x t is highdimensional. We hypothesize that this is a major reason why DDGANs cannot scale up well on the large-scale dataset with more complex data distributions. In contrast, we will explore the inherent structure in the forward difussion process to develop a more efficient semi-implicit method, which is detailed in the following section.

3 Semi-Implicit Denoising Diffusion Models

This section will present our proposed semi-implicit denoising diffusion models (SIDDMs). We first discuss how we reformulate the denoising distribution, which enables fast sampling as DDGANs and high-quality generation as DDPMs. Then, we will introduce how we optimize our model in the training time. At last, we introduce a free discriminator regularizer to boost the model performance further. We show the simplified model structure in Figure 1.

3.1 Revisiting Denoising Distribution and the Improved Decomposition

Let us reconsider the training objective presented in Equation 5, which can be reformulated as follows:

where DDGANs' formulation indirectly matches the conditional distribution of q ( x t -1 | x t ) and p θ ( x t -1 | x t ) via matching the joint distribution between q ( x t -1 , x t ) and p θ ( x t -1 | x t ) q ( x t ) . To keep it concise, we use p θ ( x t -1 , x t ) to represent p θ ( x t -1 | x t ) q ( x t ) afterwards.

Starting from this, we can factorize the two joint distributions in the reverse direction and get q ( x t -1 , x t ) = q ( x t | x t -1 ) q ( x t -1 ) and p θ ( x t -1 , x t ) = p θ ( x t | x t -1 ) p θ ( x t -1 ) . The conditional distributions are forward diffusion; we name them auxiliary forward diffusion (AFD) in our distribution matching objectives. In this decomposition, we have a pair of marginal distributions of denoised data q ( x t -1 ) , p θ ( x t -1 ) and a pair of conditional distribution q ( x t | x t -1 ) , p θ ( x t | x t -1 ) . Because the marginal distributions do not have explicit forms, we can match them implicitly by minimizing

<latexi sh

1_b

64="0zr

8

AFM37

dXukq

j

c

N

>

B

n

V

S

EJ

U

/

9L

C

Q

oWy2

pZ

I

P

Y

+O

gfH

G

D

w

T

R

K

v

5

m

<latexi sh

1_b

64="

CHnm3IOE

cJWU07

Tz

Sv

>A

B

V

N

8

r

q/

9L

y

Ro

Xj

Q

Z

Y

g

d

+P

2

w

p

5

KF

f

u

M

k

G

D

<latexi sh

1_b

64="p

m

Q

vn

Ow

8gq3rEo2

y

z0

>A

B7H

c

V

NS

J

U

/

9L

GP

K

W

Z

I

Cj

R

k

X

M

Y

d

+

T

u

D

F

f

5

<latexi sh

1_b

64="0zr

8

AFM37

dXukq

j

c

N

>

B

n

V

S

EJ

U

/

9L

C

Q

oWy2

pZ

I

P

Y

+O

gfH

G

D

w

T

R

K

v

5

m

<latexi sh

1_b

64="ndzGmZ

YpFH9

R

2gX7

I

>A

B8

c

V

NS

EJ3

W

O

q

K

M

C

/

rL

0

+

w

v

5f

y

Q

o

k

U

j

T

P

u

D

<latexi sh

1_b

64="HKU

kS

ZgXVMIPDvNG

>A

B

c

8

EJ3

r

q/

9L

0m

CF

2

oQ

+

z

u

j

Y

w

y

f

d

p

W

7

5

T

n

O

R

<latexi sh

1_b

64="

CHnm3IOE

cJWU07

Tz

Sv

>A

B

V

N

8

r

q/

9L

y

Ro

Xj

Q

Z

Y

g

d

+P

2

w

p

5

KF

f

u

M

k

G

D

<latexi sh

1_b

64="p

m

Q

vn

Ow

8gq3rEo2

y

z0

>A

B7H

c

V

NS

J

U

/

9L

GP

K

W

Z

I

Cj

R

k

X

M

Y

d

+

T

u

D

F

f

5

<latexi sh

1_b

64="w

j

y

zZ

pH9

F/XY

D

uV

>A

B73

c

NS8

EJ

W

O

Q

r

g

M

+

K

v

2

Td

C

P

m

k

I

5

R

0

q

o

n

f

U

G

L

<latexi sh

1_b

64="C/dX

o

5c7FI

Z

W

G

Mu

N

>A

B8

V

S

EJ3

O

Y

q

K

z

rL

m0

n

g

9

+

2

H

w

v

p

f

y

R

Q

k

U

j

P

T

D

<latexi sh

1_b

64="HKU

kS

ZgXVMIPDvNG

>A

B

c

8

EJ3

r

q/

9L

0m

CF

2

oQ

+

z

u

j

Y

w

y

f

d

p

W

7

5

T

n

O

R

<latexi sh

1_b

64="0zr

8

AFM37

dXukq

j

c

N

>

B

n

V

S

EJ

U

/

9L

C

Q

oWy2

pZ

I

P

Y

+O

gfH

G

D

w

T

R

K

v

5

m

<latexi sh

1_b

64="

CHnm3IOE

cJWU07

Tz

Sv

>A

B

V

N

8

r

q/

9L

y

Ro

Xj

Q

Z

Y

g

d

+P

2

w

p

5

KF

f

u

M

k

G

D

<latexi sh

1_b

64="w

j

y

zZ

pH9

F/XY

D

uV

>A

B73

c

NS8

EJ

W

O

Q

r

g

M

+

K

v

2

Td

C

P

m

k

I

5

R

0

q

o

n

f

U

G

L

<latexi sh

1_b

64="oKdBwp7/

0SZ

FnNW

zYC

>A

3

c

V

8

EJ

Ur

q

9

Iv

Q

Hg

M

2

u

m

G

T

y5

+

D

f

R

O

P

j

k

X

L

<latexi sh

1_b

64="p

m

Q

vn

Ow

8gq3rEo2

y

z0

>A

B7H

c

V

NS

J

U

/

9L

GP

K

W

Z

I

Cj

R

k

X

M

Y

d

+

T

u

D

F

f

5

<latexi sh

1_b

64="C/dX

o

5c7FI

Z

W

G

Mu

N

>A

B8

V

S

EJ3

O

Y

q

K

z

rL

m0

n

g

9

+

2

H

w

v

p

f

y

R

Q

k

U

j

P

T

D

<latexi sh

1_b

64="HKU

kS

ZgXVMIPDvNG

>A

B

c

8

EJ3

r

q/

9L

0m

CF

2

oQ

+

z

u

j

Y

w

y

f

d

p

W

7

5

T

n

O

R

<latexi sh

1_b

64="XIB

m

+ZyuM0VoP

S

W

Lq

A

>

83

c

D

gN

EO

r

fUY9

Q

H

Fj zk

2

w5

J

/

R

C

7

d

T

v

n

G

p

K

<latexi sh

1_b

64="z

HQf nq

R

d

7

IA

L

>

B83

c

VD

SgN

EOy

r

UY9

oM

u

5

TZ

j

P

K

v

G

F

/

p

0

W

2

J

k

w

X

m

+

C

<latexi sh

1_b

64="w

j

y

zZ

pH9

F/XY

D

uV

>A

B73

c

NS8

EJ

W

O

Q

r

g

M

+

K

v

2

Td

C

P

m

k

I

5

R

0

q

o

n

f

U

G

L

<latexi sh

1_b

64="

G

C0k3TFK

O/D

XwY

QcI

>A

B9H

V

LSgN

E

y

r

fU

oP

d

n

M

5

Z

z

u

R

q

7

v

W

8

j

p

+m

2

J

<latexi sh

1_b

64="7yRPdUI

v

q0

EJfG

TrQ

A

>

B

H

c

V

NS8

3

/

9L

O

F

oY

+

z

u

D

m

Z

j

w

M

XC

W

2

g

n

K

p

k

5

<latexi sh

1_b

64="HyKBk8NzZ

/

wn r3d

S

u

>A

+X

VDL

FJ

U

2vqE

g0Wo

T

R

5M

Q

j

P

7

p

9

G

O

Y

m

C

f

<latexi sh

1_b

64="

G

C0k3TFK

O/D

XwY

QcI

>A

B9H

V

LSgN

E

y

r

fU

oP

d

n

M

5

Z

z

u

R

q

7

v

W

8

j

p

+m

2

J

Diffusion Reverse

x

t

-

1

x

0

x

′

t

-

1

x

t

G

(

x

t

, t, z

)

z

(a) DDPMs.

x

t

-

1

x

0

x

′

t

-

1

x

t

D

(

x

t

, x

t

-

1

, t

)

Fake/True

G

(

x

t

, z, t

)

z

(b) DDGANs.

x

t

-

1

x

0

x

′

t

-

1

x

t

Fake/True

G

(

x

t

, z, t

)

z

D

(

x

t

-

1

, t

)

x

′

t

Auxiliary Forward Diffusion

C

(

x

t

-

1

, t

)

x

C

t

(c) SIDDMs (ours).

the Jensen-Shannon divergence ( JSD ) via adversarial learning. For the conditional distribution of forward diffusion, since q ( x t | x t -1 ) has an explicit form of Gaussian, we can match them via KL . The following theorem states that matching these two pairs of distributions separately can approximately match the joint distribution.

Theorem 1 Let q ( x t -1 , x t ) and p θ ( x t -1 , x t ) denote the data distribution from forward diffusion and the denoising distribution specified by the denoiser G θ , respectively, and we have the following inequality:

where c 1 and c 2 are upper bounds of 1 2 ∫ | q ( x t | x t -1 ) | µ ( x t -1 , x t ) and 1 2 ∫ | p ( x t -1 ) | µ ( x t -1 ) ( µ is a σ -finite measure), respectively. A proof of Theorem 1 is provided in Section II of the Supplementary.

3.2 Semi-Implicit Objective

Based on the above analysis, we formulate our SIDDMs distribution matching objectives as follows:

where the λ AFD is the weight for the matching of AFD. In Equation 7, the adversarial part D adv is the standard GAN objective. To match the distributions of AFD via KL, we can expand it as:

which is the combination of the negative entropy of p θ ( x t | x t -1 ) and the cross entropy between p θ ( x t | x t -1 ) and q ( x t | x t -1 ) . In our scenario, optimizing the cross-entropy term is straightforward because we can easily represent the cross-entropy between the empirical and Gaussian distributions using mean square error. This is possible because the forward diffusion, denoted as q ( x t | x t -1 ) , follows a Gaussian distribution. However, the negative entropy term -H ( p θ ( x t | x t -1 )) is intractable. However, p θ ( x t | x t -1 ) can be estimated on samples from the denoiser G that models p θ ( x t -1 | x t ) . Thus we need another parametric distribution p ψ ( x t | x t -1 ) to approximately compute -H ( p θ ( x t | x t -1 )) .

In the following formulation, we show the maximizing of the conditional entropy can be approximated by the following adversarial training objective:

Given θ i at the i -th iteration, the max step estimates the conditional distribution p θ i ( x t | x t -1 ) and we have p ψ i ( x t | x t -1 ) ≈ p θ i ( x t | x t -1 ) . Then, in the next iteration, the min step updates the parameter θ in the generator given ψ i and obtain the updated θ i +1 . Thus, this iterative min-max game between the generator p θ and the conditional estimator p ψ can minimize this negative conditional entropy -H ( p θ ( x t | x t -1 )) that we mentioned in the D KL decomposition of Equation 8. This adversarial process can perform as long as we can access the likelihood to p ψ ( x t | x t -1 ) . In our case, it is forward diffusion and follows the Gaussian distribution.

Similar to DDGANs, we also define p θ ( x t -1 | x t ) := q ( x t -1 | x t , x 0 = G θ ( x t , t )) via the posterior distribution. In the distribution matching objective above, we apply the GANs to minimize the JSD of marginal distributions and the L 2 reconstruction to optimize the cross entropy. We also define x ′ t -1 as the data sampled from the newly defined distribution, and x ′ t are sampled from x ′ t -1 via forward diffusion. Our final training objective can be formulated as follows:

where C ψ denotes the regression model that learns to minimize this negative conditional entropy. In the implementation, we share the most layers between the discriminator and the regression model. We put the detailed derivation of the following training objective in Section I of the supplementary.

Compared with DDGANs, our model abandons the purely adversarial training objective and decomposes it into a marginal and one conditional distribution, where the conditional distribution can be optimised with a less complex training objective and leads to stable training for updating the denoising model. Secondly, our model can share the same model structure as DDPMs and be improved based on the advanced DDPMs network structure. Thirdly, our decomposition does not bring any overhead compared to DDGANs and can achieve steady performance improvement.

3.3 Regularizer of Discriminator

The UnetGANs [12] proposes adopting an Unet structure for the discriminator, demonstrating more details in the generated samples. Unlike the common design of discriminators, which only output a global binary logit of "True/Fake", an Unet-like discriminator can distinguish details from different levels. The denoising process in the diffusion models can also benefit from pixel-level distribution matching. Thus, our paper shares the same network structure between the denoiser G θ and the discriminator D ϕ . Inspired by our decomposition formulation, the reconstruction term provides better gradient estimation and boosts the model performance. We also apply the same strategy to the discriminator as a stand-alone contribution. That is, getting the denoising output from the discriminator and reconstructing it with the ground truth x 0 . We formulate the regularizer as follows:

where this regularization only applies to the sampled data q ( x t ) . Different from the commonly used spectral norm [13], Wasserstein GANs [14] and R1 regularization [15], our regularization does not bring any side effect, such as restriction to model capacity, requiring additional overhead or grid search of the hyper-parameters on each dataset. Our regularizer can be easily plugged into our model and DDGANs and does not require extra network design, which is specifically designed for boosting diffusion models with GANs.

4 Related Works

The pioneer works [10, 1] of the diffusion models introduce the discrete time-step diffusion models and the parameterized reversal process for generating novel data. Later, the score-based perspective [16] proposes the continuous-time diffusion models and unifies the denoising diffusion model

and the denoising score matching models [17, 18]. The adversarial score matching [19] utilizes the extra adversarial loss to improve the unimodal Gaussian distribution matching, which differs from the goal of DDGANs and ours. Another branch of works introduces some inductive biases for improving the diffusion models further, the ADM, UDM and EDM [2, 20, 21] propose the classifier guidance, unbounded score matching and better sampling strategy for diffusion-based models respectively.

Although diffusion-based models achieve better quality and diversity, it still suffers from the sampling speed, which usually takes thousands of steps to achieve the best quality. Several methods also boost the sampling speed via knowledge distillation [22, 23], learning the nosing schedule of forward diffusion [24]. The DDIM [25] and FastDPM [26] propose non-Markovian diffusion processes for the sampling boosting. For the score-based continuous-time diffusion models, [27] provides faster SDE solvers. The DDGANs [11] is most related to our proposed methods, which proposed an adversarial formulation for the denoising diffusion distribution and enable a few sampling steps without compromising the generated quality too much.

Unlike DDGANs, we propose a new decomposition for the denoising distribution and achieve better results. From the perspective of better distribution decomposition, we share a similar insight with the related works of conditional GANs (cGANs) [28, 29]. AC-GAN [28] proposes to model the conditional distribution of conditional generative models via decomposing the joint between the label and the data to a marginal and the conditional distribution via an auxiliary classifier. TAC-GAN lately fix the missing term of the AC-GAN decomposition. These two works can only be applied to the conditional generation while our proposed decomposition can be applied to both unconditional and conditional generation. Thus, their works are fundamentally different from ours. The other related GANs propose to enhance GANs with data argumentation [30, 31] or diffusion noise [32], which would also explain why GAN-ehanced denoising diffusion process can function well in practical.

5 Experiments

In our experiments, we evaluate our proposed method in the simulated Mixture of Gaussians and several popular public datasets, Cifar10-32 [33], CelebA-HQ-256 [34] and the ImageNet1000-64[35]. To study the effects of our model components, we also conduct the ablations to identify their sensitivity and effectiveness.

For the model architectures, we apply the Unet [36] as the ADM [2], and we follow the same efficient strategy as Imagen [4] to change the order of downsampling and upsampling layer in the Unet blocks. In our method, we also design our discriminator as an Unet. Thus, we apply the identical structure for the generator G and the discriminator D . The only difference between them is the input and output channels for fitting in different inputs and outputs in our formulations.

5.1 MOGSynthetic Data

Identifying the effectiveness of generative models in high-dimensional data is tricky, and we cannot directly visualize the mode coverage of the generated data. Also, popular metrics like FID and Inception Score can only be referred as quality evaluation. Thus, we test our models and the baseline DDGANs on generating a Mixture of Gaussians. To generate the data, we sample 25 groups of data from the Gaus-

sians independently and each distribution has a different mean but with the same variance. In Figure 3, we show the synthesized results of different models and the denoising steps and the quantitative results are shown in Table 1. We can observe that for the extremely few denoising steps of 2 and many as 16 steps, DDGANs fail to converge to real data distribution, and our proposed method can recover well on the different steps. Also, to identify if the adversarial term takes the main job in the generation, we remove the auxiliary forward term and find that our model cannot recover the original distribution, proving the proposed decomposition's effectiveness. To evaluate our model's effectiveness on real-world generation tasks, we test our models on the tiny image dataset CIFAR10, the fine-grained generation task on the Celeb-HQ, and the large-scale challenging dataset Imagennet. We pick a small amount of generated images from our model and

(a) Generated samples on CIFAR10.

(b) Generated samples on CelebHQ.

?

(c) Generated samples on ImageNet.

show them in the Figure 2. We also quantitatively evaluate the quality of the generated samples and collect all of the results in Table 2,3 and 4. Visually, our model generates samples with high fidelity and rich diversity. For the evaluation metric, we choose the Fréchet inception distance (FID) [47] and Inception Score [48] for sample fidelity and the improved recall score [49] to measure the sample diversity. Compared with the GANs-based method, our method achieves better sample quality and diversity, identifying the benefit of enhancing GANs with the diffusion formulation. Compared with the baseline DDGANs, our model shows superior quantitative results than the baseline DDGANs. Although our method does not reach the same quality as the diffusion-based (or score-based) models

Ground Truth

T=1, vanilla GANs

DDGANs

SIDDMs

(ours)

SIDDMs

w/o AFD

(ours)

三

T=4

T=8

T=16

10

®

10

0.0072

0.0064

0.0056

0.0048

10

10

10

15

20

SI

0.0040

T=2

T=4

T=8

T=16

10

0.0032

20

20

®

- 0.0024

- 0.0016

SI

®

SI

?

ST

0.0008

?

10

10

10

10

15

20

- 0.0000

®

图

15

10

15

15

T=2

T=4

T=8

T=16

25

10

15

15

with a small gap, our model still has the large advantage of fast sampling. To be notified, we apply four denoising steps for the CIFAR10 and two steps for the CelebA-HQ generation, identical to the DDGANs' main results.

5.2 Generation on Real Data

For the ImageNet, we choose four denoising steps to get the best sample quality on the ImageNet. ImageNet has 1000 categories, each containing thousands of images and distinguishing itself as a largescale dataset with diversity. To train a generative model with high fidelity on the dataset, we usually require other inductive biases such as regularizers or model scaling-up methods [13, 50]. The DDGANs fail to retain the high-fidelity samples on this dataset without additional inductive bias. However, with the same model capacity, our proposed method can achieve comparable results w.r.t. the diffusion-based models. This result shows that our model can handle large-scale generation and potentially be applied to boost the sampling speed of large-scale text2image models.

6 Ablations

Weight of AFD

0.0

0.1

0.5

1.0

5.0

∞

SIDDMs(ours)

77.15

3.32

2.63

2.24

2.55

41.27

SIDDMs(ours) w/o

Regularizer of Discriminator

51.67

4.64

4.79

3.20

2.95

51.55

Diffusion Steps

1

2

4

8

16

32

SIDDMs(ours)

27.39

3.17

2.24

2.31

2.15

2.46

(a) AFD weight 0.0.

(b) AFD weight 1.0.

(c) AFD weight ∞ .

X

To identify the function of the components in our formulation, we conduct controlledexperiments on the effects of the adversarial and the AFD term. We set the weights of AFD to be [0 . 0 , 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , ∞ ] , where 0 . 0 represents only the adversarial term and ∞ denotes only the AFD term in our training. We apply the same sampling strategy as our full models and adopt four denoising steps. The FID scores are reported in Table 5, and the generated samples are shown in Figure 4. We can see that if missing any of these two terms in our formulation, we cannot recover the original image distribution. In addition, we found that our model is not sensitive to the weights between these two components w.r.t. the FID scores when both terms participate in the training, which further identifies the effectiveness of our formulation. Also, we train the model without the regularizer for the discriminator, and we also identify that the proposed auxiliary tasks for the discriminator further enhance the model performance for our full formulation.

7 Conclusion

In conclusion, the quest for fast sampling with diverse and high-quality samples in generative models continues to pose a significant challenge. Existing models, such as Denoising Diffusion Probabilistic Models (DDPM), encounter limitations due to the inherent slowness associated with their iterative steps. On the other hand, Denoising Diffusion Generative Adversarial Networks (DDGAN) faces scalability issues when dealing with large-scale datasets. To address these challenges, we propose a novel approach that effectively addresses the limitations of previous models by leveraging a combination of implicit and explicit factors. Specifically, we introduce an implicit model that enables us to match the marginal distribution of random variables in the reverse diffusion process. Additionally, we model explicit distributions between pairs of variables in reverse steps, which allows us to effectively utilize the Kullback-Leibler (KL) divergence for the reverse distribution. To estimate the negative entropy component, we incorporate a min-max game into our framework. Moreover, we adopt the L2 reconstruction loss to accurately represent the cross-entropy term in the KL divergence. Unlike DDPM but similar to DDGAN, we do not impose a parametric distribution for the reverse step in our approach. This design choice empowers us to take larger steps during the inference process, contributing to enhanced speed and efficiency. Additionally, similar to DDPM, we effectively leverage the exact form of the diffusion process to further improve our model's performance. Our proposed approach exhibits comparable generative performance to DDPM while surpassing models with fewer sampling steps.

References

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations , 2021.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021.

Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations , 2022.

Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8207-8216, 2020.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations , 2018.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017. cite arxiv:1701.07875.

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 7559-7570. Curran Associates, Inc., 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution . Curran Associates Inc., Red Hook, NY, USA, 2019.

Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Ioannis Mitliagkas, and Remi Tachet des Combes. Adversarial score matching and improved sampling for image generation. In International Conference on Learning Representations , 2021.

Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527 , 2021.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS , 2022.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations , 2022.

Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models. arXiv preprint arXiv:2104.02600 , 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132 , 2021.

Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080 , 2021.

Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans, 2017.

Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 13301339. Curran Associates, Inc., 2019.

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in neural information processing systems , 2020.

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in neural information processing systems , 2020.

Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.02262 , 2022.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR , abs/1710.10196, 2017.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. cite arxiv:1505.04597Comment: conditionally accepted at MICCAI 2015.

Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672 , 2021.

Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 , 2021.

Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in neural information processing systems , 2021.

Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in neural information processing systems , 2020.

Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations , 2021.

Jyoti Aneja, Alexander Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Neural Information Processing Systems , 2021.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations , 2018.

Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2020.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2021.

Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 2017.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 2016.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. arXiv preprint arXiv:1904.06991 , 2019.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. volume abs/2201.00273, 2022.

Peer reviews:
comments: SUMMARY:
This paper proposes the Semi-Implicit Denoising Diffusion Model (SIDDM) to enable fast sampling while maintaining high-generation quality. Specifically, SIDDM applies an implicit model to match the marginal distributions of the reverse diffusion process. Meanwhile, SIDDM models explicit conditional distribution of the forward diffusion. Furthermore, a regularization method is proposed to enhance the performance. Experiments show that SIDDM has comparable performance to other diffusion models with fewer sampling steps.

STRENGTHS:
1. The authors analyse the reasons for the limitations of DDGAN, and decomposite the denoising distribution to improve the training objective. The idea is reasonable.
2. The experiments in the simulated Mixture of Gaussians and several popular public datasets demonstrate the effectiveness of the proposed method.
3. The authors also provide the code for results reproduction, which shows the solidness of the work.

WEAKNESSES:
1. In Fig. 1, there are dashed lines of different colours in Figure 1. But the authors don't give some corresponding explanation. 
2. The implementation details are not clarified, like the structure of the regression model, the discriminator regularizer, and the denoiser. Meanwhile, the training settings (e.g., training iterations) are not provided.

QUESTIONS:
1. In Fig. 3 and Tab.5, the larger step (e.g., 16) performs worse than smaller step sizes (e.g., 2). This is different from the conventional situation in diffusion models. Please give some analysis.
2. It is recommended to give the training and inference strategy (algorithm) of SIDDM.

LIMITATIONS:
The authors have discussed the societal impact in the supplementary material. It would be better if the authors discuss some limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

comments: SUMMARY:
This method propose a way to achieve fast sampling during inference without compromising sample diversity and quality of diffusion model, and shows their effectiveness on both conditional and unconditional generation, qualitatively and quantitatively. A theoretical framework is also proposed which looks reasonable to me.

STRENGTHS:
This method basically proves that a very low FID can be obtained with very few sampling steps, compared to ADM and DDGAN. In addition, it claims that it can do both conditional and unconditional generation. Moreover, the theoretical framework looks solid to me, though I am not from the theory field so I cannot comment more on this.

WEAKNESSES:
However, it seems that there are no experiments on conditional generation (both qualitative and quantitative). Perhaps more relevant experiments are welcome to consolidate this submission and prove the effectiveness of the new sampling strategy.

QUESTIONS:
The overall story looks good to me. I wonder whether you would like to perfect your experiment with more qualitative/quantitative in the conditional setting.

LIMITATIONS:
As mentioned above, more analysis on the conditional setting is welcome, e.g., using Stable Diffusion as the base model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
In this paper, the authors propose to use an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. Specifically, the adversarial loss is applied to the marginal distributions obtained by forward and backward process, and KL loss is used to regularize the different between the conditional distribution. Experiments show that the method works with small inference steps.

STRENGTHS:
* Using the adversarial loss is a good idea to learn the conditional distribution of the backward process with large step size.
* The experimental results with small steps seem to be good, both for toy examples and real datasets.

WEAKNESSES:
* Generally, the paper is hard to follow, the formulas should be presented in a better way:
    * How to obtain equation (6) through (5)?
    * Line 129-137 is really difficult to follow, since $q(x_t|x_{t-1})$ is Gaussian distribution, no matter the step size, why not directly use a Gaussian distribution to parameterize $p_\theta(x_t|x_{t-1})$?
    * It problematic to treat equation (11) and (12) as equivalent, since $\psi$ is fixed in (11). 

* In the experiments, with smaller steps, the method works better for both the toy and real datasets. It is not reasonable. Any explanation about this phenomenon?

QUESTIONS:
Please see weakness part.

LIMITATIONS:
Please see weakness part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-12

comments: SUMMARY:
The paper presents a method called Semi-Implicit Denoising Diffusion Model (SIDDM) aimed to accelerate the sampling process, enhance scalability to large datasets, and improve model performance. It improves upon the DDGAN model by reformulating the denoising distribution of diffusion models with explicit and implicit training objectives, which leverages an implicit GAN objective for the marginal distribution and an L2 reconstruction loss for the conditional distribution. To improve generative quality, the authors adopt a U-net-like structure for the discriminator and a new regularization technique involving an auxiliary denoising task. Experiments are conduct on CIFAR-10, CelebA-HQ-256, and ImageNet to demonstrate it effectiveness.

STRENGTHS:
(1) The experiment and comparison were comprehensive, and the results were good. Ablation was comprehensive.
(2) The writing has a clear structure (just not sure if there are any issues with the derivation).
(3) The method performs quite well on large datasets like ImageNet, whereas the previous ones didn't do well.

WEAKNESSES:
(1) There lacks a graph showing the FID-sampling speed tradeoff on ImageNet.
(2) In Table 1, the 'SIDDMs w/o AFD (ours)' entry is quite close to DDGAN in terms of setting, but why is there such a big difference in performance?

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of making diffusion models faster (faster sampling) while still yielding high quality, diverse samples, from a large phenomenological space (scaling to large datasets). This is achieved by a novel semi-implicit denoising diffusion model (SIDDM). The idea extends the approach followed in DDGANs by reformulating the objective, and a novel decomposition into two components: (a) a pair of marginal distributions over the denoised data (at each step) with an implicit form to be aligned with the JSD via a learnt critic, and (b) a pair of conditional, forward diffusion components having explicit forms to be aligned using the KL divergence. This mixed objective allows better scaling than DDGANs. A novel regularizer is also introduced for the discriminator which allows a more granular distribution matching. A proof of concept validation is performed using a synthetic, Mixture of Gaussians dataset. Ablations demonstrate the benefit of both the innovations.  SIDDM is benchmarked against the art on the CIFAR10, CelebA-HQ-256, and ImageNet datasets demonstrating performance competitive with DDGANs, better scaling to ImageNet, and much higher sampling speeds than classical DDPMs.\n\nSTRENGTHS:\n**Relevance** The paper addresses an important problem – speeding up diffusion based generative image models while not compromising on the image quality, with good scalability to large, complex datasets. This should be of relevance to the community working in these areas but also of interest to a much wider audience. \n\n**Originality** The proposed decomposition is interesting and novel, balancing both a non-parametric, population-level statistical alignment of distributions and a direct (simple, parametric) objective which decomposes into a sample level objective over Auxiliary Forward Diffusion (AFD). It stands to reason that this can have a significant impact on the training of the diffusion reversal DNN.   \n\n**Technical Quality** \n- The technical approach mostly appears sound. It creates a new, direct and stronger learning signal which augments the critic based learning objective. This can lead to better learning outcomes over more complex distributions representing larger datasets like ImageNet. In addition, regularizing the discriminator via the UnetGAN formulation seems reasonable too. \n- The mathematical formalism used in the paper, including the upper bound in Theorem 1, add to the strength of the contribution, making it principled, and beyond the heuristic choices made. I haven’t verified the derivations in the paper entirely but have followed the structure of the derivations and the proof and find them reasonable.\n\n**Experimental Validation** There are several positives in the choices made for evaluation:\n- The choice of using a synthetic dataset based on Mixture of Gaussians (MoGs) allows for validating the core idea.  \n- While the results on CIFAR10 and CelebA-HQ-256 are comparable with DDGANs, the FID scores on ImageNet demonstrate a performance comparable to SOTA diffusion models while DDGANs performance seems to demonstrate a failure to scale to ImageNet. This validates the core motivation of the paper. \n- The ablation study, with and without the AFD term, validates the benefit of the decomposition of the training objective that forms the basis of SIDDMs; as well as the benefit due to the UNetGAN regularizer in (14)     \n\n**Significance** The approach has the potential for significant impact over an important area of research once the approach has been reproduced and ‘hardened’.\n\nWEAKNESSES:\n**Experimental Validation** \n\n(a) The MoG experiment (Section 5.1) demonstrates that SIDDM achieves good results with very small number of time steps, and has better stability than DDGANs.  However,\n- It is not discussed why it is unable to converge to the simple MoG data distribution, when the number of time steps increases (similar to DDGAN).  \n- Table 1 shows the FID score. These trends don’t tally well with the behavior shown in Figure 3. Perhaps using other metrics (JSD, LPIPS, etc.) may explain the results better?  \n- The implications of the above on the learning on real datasets, and the scalability thereof on larger,  complex datasets like ImageNet is unclear. Such tasks may require more time steps (it is not clear whether this is the case) in which case, convergence to the data distribution becomes an issue.\n\n(b) The results on CelebA-HQ-256 are typically used to show performance on high-resolution data. The images (depictions) of the generated samples, both in the paper and in the supplementary, are too small. Similarly, the ImageNet depictions are very small.\n\n(c) While NFE (number of function evaluations) and the wall-clock time are shared, no details of the hardware are shared.\n\n**Clarity** (l. 230) It is not quite clear why the material on training with real datasets like CIFAR10, Celeb-HQ etc. is in Section 5.1 and not Section 5.2. Kindly fix.\n\n**Reproducibility** Implementation details are missing. The authors also don’t mention that the code will be released. Given this, I suspect it may be hard to reproduce results. \n\n**Discussion of Limitations** No discussion of limitations in the paper.\n\nQUESTIONS:\nKindly address the following:\n- Explain the behavior on the MoG dataset using a larger suite of metrics (see Borji, CVIU 2019; and its 2021 update for example). \n- Explain the behavior (non-convergence to data distribution) when using a larger number of steps and the scalability implications. \n- Did you investigate the tradeoffs between a smaller step size with larger number of steps, and smaller number of steps with a larger step size? How does one decide this? Does the theoretical formulation lend itself to such a guidance?\n-  Kindly share details of the hardware, implementation, and other details which will aid reproducibility.\n- Will the code be shared?\n- Discuss limitations.\n\nLIMITATIONS:\nAuthors don’t address limitations of the paper. \n\nI don’t think there are any direct negative societal implications. Other limitations and opportunities for improvement are addressed in my responses to previous questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}