Title: Moment Matching Denoising Gibbs Sampling
Venue: NeurIPS 2023
Accepted: yes
Keywords: denoising score-matching

gibbs sampling

diffusion model
Abstract: Energy-Based Models (EBMs) offer a versatile framework for modelling complex data distributions. However, training and sampling from EBMs continue to pose significant challenges. The widely-used Denoising Score Matching (DSM) method for scalable EBM training suffers from inconsistency issues, causing the energy model to learn a noisy data distribution. In this work, we propose an efficient sampling framework: (pseudo)-Gibbs sampling with moment matching, which enables effective sampling from the underlying clean model when given a noisy model that has been well-trained via DSM.  We explore the benefits of our approach compared to related methods and demonstrate how to scale the method to high-dimensional datasets.

Moment Matching Denoising Gibbs Sampling

Abstract

Energy-Based Models (EBMs) offer a versatile framework for modeling complex data distributions. However, training and sampling from EBMs continue to pose significant challenges. The widely-used Denoising Score Matching (DSM) method [41] for scalable EBM training suffers from inconsistency issues, causing the energy model to learn a 'noisy' data distribution. In this work, we propose an efficient sampling framework, (pseudo)-Gibbs sampling with moment matching, which enables effective sampling from the underlying clean model when given a 'noisy' model that has been well-trained via DSM. We explore the benefits of our approach compared to related methods and demonstrate how to scale the method to high-dimensional datasets.

1 Energy-Based Models

Energy-Based Models (EBMs) have attracted a lot of attention in the generative model literature [25, 46, 7, 35]. EBMs are a type of non-normalized probabilistic model that determines the probability density function without a known normalizing constant. For continuous data x , the density function of an EBM is specified as q θ ( x ) = exp( -f θ ( x )) /Z ( θ ) where the f θ ( x ) is a nonlinear function with parameter θ and Z ( θ ) = ∫ exp( -f θ ( x )) d x is the normalization constant that is independent of x . The energy parameterization allows for greater flexibility in model parameterization and the ability to model a wider range of probability distributions. However, the lack of a known normalizing constant makes training these models challenging. We start by giving a brief introduction of how to estimate θ in EBMs and refer the reader to [37] for a detailed overview of different training techniques for continuous EBMs.

Likelihood-based training: A classic method to learn θ is to minimize the KL divergence between the data distribution p d and the model density q θ , which is defined as

where we use . = to denote the equivalence up to a constant that is independent of θ . The integration of p d ( x ) can be approximated by Monte Carlo with the training dataset X train = { x 1 , · · · , x N } ∼ p d ( x ) ; in this case, it is equivalent to the maximum likelihood estimate (MLE) [5]. However, for EBMs, minimizing the KL divergence requires the estimation of Z ( θ ) , which is intractable for nonlinear f θ ( x ) defined by a neural network. Various methods have been proposed to alleviate the

intractability by introducing techniques like Markov chain Monte Carlo (MCMC) [13, 26, 8, 11] or adversarial training [18, 47, 6].

Score-based training: Alternatively, [16] proposes to minimize the Fisher divergence to learn θ , which is defined as

where we use s p ( x ) to denote the score function of distribution p : s p ( x ) ≡ ∇ x log p ( x ) . Under certain regularity conditions, the Fisher divergence is equivalent to the score-matching (SM) objective [16],

which does not require estimation of the intractable Z ( θ ) . However, this objective needs to calculate the Hessian trace ∇ x s q θ ( x ) = -∇ 2 x f θ ( x ) in every gradient step during training, which is computationally expensive and does not scale to high dimensional data or requires approximation [38]. In this paper, we will focus on another training method, denoising score matching [41], which overcomes the tractability and scalability issues mentioned above, and is introduced in the next section.

1.1 Denoising Score Matching

For the target data density p d ( x ) , a noise distribution p (˜ x | x ) = N ( x, σ 2 I ) is introduced to construct a noised data distribution ˜ p d (˜ x ) = ∫ p d ( x ) p (˜ x | x ) d x . Denoising score matching (DSM) [41] minimizes the Fisher divergence between the noised data distribution ˜ p d (˜ x ) and an energy-based model ˜ q θ (˜ x ) = exp( -f θ (˜ x )) /Z ( θ ) , with

where the last equation is due to ∇ ˜ x log p (˜ x | x ) being tractable for the Gaussian distribution p (˜ x | x ) .

̸

Compared to the KL or SM objectives, the DSM objective is scalable and well-defined when the data distribution is singular 2 [49] and can alleviate the blindness problem of score matching [35, 45, 51]. On the other hand, there is a notable disadvantage associated with the DSM objective: for a fixed σ > 0 , the DSM objective is not a consistent objective for learning the underlying data distribution p d since FD(˜ p d || ˜ q θ ) = 0 = ⇒ ˜ p d = ˜ q θ = p d . A common solution is to anneal σ → 0 during training. However, Equation 4 is not defined when σ = 0 since the division in Equation 4 will make (˜ x -x ) /σ 2 unbounded, which results in an inconsistent objective. Annealing σ increases the variance of the training gradients [37, 42], which makes the optimization challenging in practice.

To overcome the challenges, we propose an alternative data generation scheme: we use DSM with a fixed σ > 0 to train a 'noisy' energy model and then construct a sampler which targets the underlying 'clean' model. Specifically, our contributions are summarized as follows:

We demonstrate that for an EBM that learns a noisy data distribution, there exists a unique underlying clean model which recovers the true data distribution.

We introduce a pseudo-Gibbs sampling scheme incorporating an analytical momentmatching approximation of the denoising distribution. This allows us to sample from the underlying clean model without requiring additional training.

We illustrate how to scale our method for high-dimensional data and demonstrate the generation of high-quality images using only a single level of fixed noise. Furthermore, we showcase the application of our proposed method in multi-level noise scenarios, closely resembling a diffusion model.

2 Clean Model Identification

For a fixed σ > 0 , DSM can only learn a 'noisy' data distribution ˜ q θ (˜ x ) even in the ideal case where the Fisher divergence is exactly minimized, since FD(˜ p d || ˜ q θ ∗ ) = 0 → ˜ p d = ˜ q θ = p d . In this case, the following theorem shows that there exists a 'clean' model that is implicitly defined that learns the true data distribution.

̸

Theorem 2.1 (Existence of the underlying clean model for optimal ˜ q θ (˜ x ) ) . When the Fisher divergence goes to 0, FD(˜ p d || ˜ q θ ) = 0 → ˜ p d = ˜ q θ , there exists an unique underlying clean model q ( x ) such that ˜ q θ (˜ x ) = ∫ q ( x ) p (˜ x | x ) d x and q ( x ) = p d ( x ) .

See Appendix A.1 for proof. This theorem shows that despite training an EBM on noisy data, there is an implicit model within it that can recover the true data distribution. Therefore, instead of annealing the noise σ → 0 to recover the true data distribution, we will demonstrate how to directly sample from the implicitly-defined clean model given the noisy energy-based model in the next section.

We want to highlight that the 'perfect fit' assumption, i.e. achieving FD(˜ p d || ˜ q θ ) = 0 , may not hold for a complex data distribution p d or underpowered EBMs. Therefore, we provide general sufficient conditions for the existence of the clean model for an imperfect EBM in Appendix A.2.

2.1 Gibbs Sampling with Gaussian Moment Matching

Given a well-trained noisy energy-based model ˜ q θ (˜ x ) = ˜ p d (˜ x ) , the clean model has the form

where the denoising distribution can be written as p ( x | ˜ x ) ∝ p (˜ x | x ) q ( x ) . We notice that, since the noise distribution p (˜ x | x ) = N ( x, σ 2 I ) is known, a Gibbs sampling scheme can be constructed to sample from the underlying clean model if we know the denoising distribution p ( x | ˜ x ) , with

where the initial sample x 0 ∼ p 0 ( x ) can be drawn from a standard Gaussian p 0 ( x ) = N (0 , I ) . However, as the denoising distribution p ( x | ˜ x ) is usually intractable for complex p d , we propose an analytical Gaussian moment matching approximation of p ( x | ˜ x ) .

Denote the mean and covariance of p ( x | ˜ x ) as

The classic Gaussian moment matching method [24] specifies a Gaussian approximation p ( x | ˜ x ) ≈ N ( µ (˜ x ) , Σ(˜ x )) , which matches the first and second moment of p ( x | ˜ x ) . When ˜ q θ = ˜ p d , the first mean of the denoised distribution has a well-known analytical form [39, 2, 9, 29]

we include the derivation in Appendix A.3. Using this identity, we can rewrite Equation 4 as

where we can see that the Fisher divergence only depends on µ (˜ x ) . Since FD(˜ p d || ˜ q θ ) = 0 ⇔ q = p d (Theorem 2.1), the function µ (˜ x ) fully characterizes the distribution q . Therefore, µ (˜ x ) and p (˜ x | x ) = N ( x, σ 2 I ) can provide sufficient information to determine p ( x | ˜ x ) ∝ q ( x ) p (˜ x | x ) . As a consequence, the following theorem shows that the covariance function can also be analytically derived.

Theorem 2.2 (Analytical Covariance Identity) . Given a clean model q ( x ) such that ∫ q ( x ) p (˜ x | x ) d x = ˜ q θ (˜ x ) = ˜ p d (˜ x ) with p (˜ x | x ) = N ( x, σ 2 I ) , the µ (˜ x ) and Σ(˜ x ) of the p ( x | ˜ x ) ∝ q ( x ) p (˜ x | x ) has the following relations

See Appendix A.3 for proof. This analytical covariance identity can be seen as a high-dimensional generalization of the 2nd-order Tweedie's Formula [9, 29]. Therefore, the analytical full-covariance moment matching approximation can be written as

We want to highlight that since the Gaussian moment matching is only an approximation of p ( x | ˜ x ) , the sampling scheme in Equation 6 is a 'pseudo' Gibbs sampler unless the true p ( x | ˜ x ) is also a Gaussian distribution 3 , which is not true for general non-Gaussian p d . However, since µ (˜ x ) and p (˜ x | x ) = N ( x, σ 2 I ) are already sufficient to specify p ( x | ˜ x ) , it should be possible to derive expressions for higher-order moments which themselves involve only µ (˜ x ) and σ ; we leave this to future work. To our knowledge, the ˜ x -conditioned full covariance Gaussian moment matching approximation to p ( x | ˜ x ) has not been derived previously. In the next section, we briefly discuss the connections between our method and other related approaches.

2.2 Connection to Covariance Learning Approaches

Bengio et al. [4] proposes to approximate the true posterior p ( x | ˜ x ) ∝ p d ( x ) p (˜ x | x ) with a variational distribution q θ ( x | ˜ x ) . The parameter θ is then learned by minimizing the joint KL divergence

where ˜ p d (˜ x ) = ∫ p d ( x ) p (˜ x | x ) d x . The joint KL divergence in Equation 12 encourages q θ ( x | ˜ x ) to match the moments of the true posterior p ( x | ˜ x ) , and defines an upper bound of the marginal KL [48]

where the model is implicitly defined as the marginal of the joint q θ ( x ) = ∫ q θ ( x | ˜ x )˜ p d (˜ x ) . When q θ ( x | ˜ x ) is a consistent estimator of p ( x | ˜ x ) , this asymptotic distribution of the Gibbs sampling will converge to the true data distribution p d ( x ) [4]. For continuous data, the variational distribution is chosen as a Gaussian distribution q θ ( x | ˜ x ) = N ( µ θ (˜ x ) , Σ θ (˜ x )) , where the mean µ θ ( · ) and the covariance Σ θ ( · ) are parameterized by neural networks. We note that the only difference between the KL and DSM objective (Equation 9) is that the KL objective additionally learns the covariance. We thus show that the optimal covariance under KL minimization is the proposed analytical covariance.

Theorem 2.3 (Optimal Gaussian Approximation) . Let p (˜ x | x ) = N (0 , σ 2 I ) and assume Gaussian distribution q θ ( x | ˜ x ) = N ( µ q (˜ x ) , Σ q (˜ x )) , then the optimal q ∗ such that

has the mean and covariance with the form

see Appendix A.4 for proof. Therefore, when the optimal mean function is learned µ θ (˜ x ) = µ ∗ q (˜ x ) , the optimal Σ(˜ x ) can be analytically derived, making the learning of Σ θ (˜ x ) redundant. In addition to the training inefficiency caused by more parameters, the amortized covariance network may suffer from poor generalization [50]. Moreover, the KL objective is also not well-defined for learning data distributions which lie on a low-dimensional manifold, e.g. MNIST, see Section 4 for a detailed discussion. In this case, the learned Σ θ (˜ x ) may be a degenerate matrix, making the Gaussian density function q ( x | ˜ x ) ill-defined [49] which impedes the training, see Figure 5 for an example.

Paper[23] proposes a higher-order score-matching loss to simultaneously learn both the first order score ∇ ˜ x log ˜ q θ (˜ x ) and the second order score ∇ 2 ˜ x log ˜ q θ (˜ x ) . However, our findings indicate that the mean function µ (˜ x ) (or the first order score ∇ ˜ x log p (˜ x ) ) already contains all the moment information of the underlying true distribution p d , and the optimal moment can be derived using the mean function. Therefore, learning the second-order score is redundant and may lead to sub-optimal inference.

2.3 Connection to Analytic DDPM

The recent paper Bao et al. [2] considers a constrained variational family q θ ( x | ˜ x ) = N ( µ θ (˜ x ) , σ 2 q I ) in the context of diffusion model and derive the optimal σ ∗ q as

which can also be rewritten using the score function

In Appendix B, we provide a detailed derivation to show how this approximation can be linked to our method using the Fisher information identity [10]. This approximation has two potential limitations: first, compared to full covariance moment matching, the assumed isotropic covariance structure may be insufficiently flexible to capture the true posterior; second, the covariance is independent of ˜ x .

The second assumption only holds when µ (˜ x ) is a linear function of ˜ x 4 (e.g. when p d ( x ) is Gaussian) and does not hold for other nonGaussian p d ( x ) . Therefore, our ˜ x -dependent full-covariance approximation offers a more versatile approximation family, which ultimately results in a more precise estimation. However, in certain applications such as accelerating the sampling procedure of a diffusion model [2], it is advantageous to use a ˜ x -independent isotropic covariance due to its inexpensive estimation. On the other hand, our ˜ x -dependent covariance necessitates the computation of the Hessian for each ˜ x , making it inefficient for highdimensional data. In Section 3, we will explore approaches to mitigate this limitation.

2.4 Posterior Approximation Comparison

We now consider a toy example to compare the three denoising posterior approximations discussed above. Let p d ( x ) be a Mixture of Gaussians (MoG) p d ( x ) = 1 4 ∑ k =4 k =1 g k ( x ) whose components g [1:4] are 2D Gaussians with means [ -1 , -1] , [ -1 , 1] , [1 , 1] , [1 , -1] and isotropic covariance σ 2 g I with σ g = 0 . 2 . The noise distribution is p (˜ x | x ) = N ( x, σ 2 I ) with σ = 0 . 2 , so ˜ p (˜ x ) = ∫ p ( x ) p (˜ x | x ) d x is an MoG with the same component means and diagonal covariance ( σ 2 g + σ 2 ) I ; see Figure 1a for a visualization. In this case the true posterior p ( x | ˜ x ) does not allow a tractable form. Fortunately, given a noisy sample ˜ x ′ and an evaluation point x ′ , we can evaluate the true density p ( x | ˜ x ′ ) using Bayes rule: p ( x ′ | ˜ x = ˜ x ′ ) = p (˜ x ′ | x = x ′ ) p d ( x ′ ) / ˜ p d (˜ x ′ ) . Figure 1 shows the true posteriors given four different ˜ x ′ where we use grid data in x -space to visualize the density.

To train the model, we sample 10,000 data points from p d as our training data. For the KL-trained

p

d

(

x

)

˜

p

d

(˜

x

)

∫

p

d

(

x

)

p

(˜

x

|

x

) d

x

(1.0, 1.0)

(0.0,0.0)

(0.2, 0.2)

(1.0,0.0)

Gibbs sampler described in Section 2.2, we use a network with 3 hidden layers with 400 hidden units, Swish activation [28] and output size 4 to generate both mean and log standard deviation of the Gaussian approximation. For the moment-matching Gibbs sampler (including both full and isotropic covariance), we use the same network architecture but with output size 1 to get the scalar energy and DSM as the training objective. Both networks are trained with batch size 100 and Adam [19] optimizer with learning rate 1 × 10 -4 for 100 epochs. For the ˜ x -independent isotropic covariance, we use the Monte Carlo approximation to estimate the variance [2] with 10000 samples from ˜ p d (˜ x ) .

Figure 1 visualizes the approximations to the denoising posterior p ( x | ˜ x ) estimated by each of the three methods described in the previous sections. We surprisingly find that although the KL objective in Equation 12 encourages q θ ( x | ˜ x ) to match the moments of

p ( x | ˜ x ) , the learned covariance in Figure 1e still underestimates the variance of the posterior. This shows the redundancy of covariance learning can degrade the variational approximation performance. Additionally, the ˜ x -independent covariance fails to account for the relative positions of ˜ x ′ and lacks the ability to predict the posterior's elliptical shape due to its isotropic nature. In contrast, our ˜ x -dependent full covariance approximation overcomes these limitations, enabling more accurate predictions that capture the intricate geometry of the posterior distribution.

We then use the estimated posterior to conduct (pseudo) Gibbs sampling to generate samples 5 . Specifically, we initialize the first sample x 0 ∼ N (0 , 0 . 1) and run one Markov Chain with 10,000 time steps to generate 10,000 samples. In addition to the mixture of Gaussian datasets, we also train and generate samples from the 2D Swiss roll and two-ring datasets. For numerical evaluation, we calculate the Maximum Mean Discrepancy (MMD) [12] between 10k samples generated by a single-chain Gibbs sampler and 10k samples from the training dataset respectively. The kernel insides MMD is a sum over 5 Gaussian kernels with bandwidth ranging over [2 -2 , 2 -1 , 2 0 , 2 1 , 2 2 ] . The MMD results (including both mean and std) are calculated using 5 random seeds. We find that Gibbs sampling with the

1

(i) True data (j) Learn. diag. (k) Analytic iso. (l) Analytic full

proposed analytical full covariance achieves the best results; numerical results are in Table 1, with a visual comparison in Figure 2.

3 Scalable Implementations for Image Data

Scalable Diagonal Hessian Approximation As we discussed in Section 2.3, the proposed full covariance Gaussian approximation in Equation 10 requires calculating an D × D Hessian ∇ 2 ˜ x log ˜ q (˜ x ) for each ˜ x with size D , which brings both memory and computation difficulties for high-dimensional data. A naive diagonal Hessian method (only using the diagonal entries in the Hessian) will address the memory bottleneck but still needs D times backward passes for the exact computation of the diagonal term [22]. In this paper, we use the following diagonal Hessian approximation [3],

where v s ∼ p ( v ) is a Rademacher random variable with entries ± 1 and ⊙ denotes the element-wise product 6 . This estimator will converge to the exact Hessian diagonals when S → ∞ [3]. The computation for each v s can be computed by two forward-backward passes. It is worth emphasizing that our ˜ x -dependent diagonal moment matching approach provides a comparable level of flexibility to the variational method proposed in [4] while eliminating the need for additional training of the

万

5

4

5

7

9

女

6

?30

827

0358

2

3

5

3

G.

9

5

0

4

2

交

&

2

8

M

K

?

M

2

n

A

6

Q

diagonal covariance. Furthermore, our method remains more flexible than the isotropic ˜ x -independent moment matching method proposed by [2].

Energy or Score Parameterization For the full-covariance moment matching in Equation 10, we require ∇ 2 ˜ x log ˜ q θ (˜ x ) to be symmetric to obtain a valid Gaussian approximation. However, if we learn the score function ∇ ˜ x log p (˜ x ) = s θ (˜ x ) using a network s θ ( · ) : R D → R D , its Jacobian is not guaranteed to be symmetric. In this case, we follow [34] and directly parameterize the density function ˜ q θ ( · ) with a neural network f θ ( · ) : R D → R and let the score function ∇ ˜ x log ˜ q θ (˜ x ) = -∇ x f θ (˜ x ) . This can be obtained by AutoDiff packages like PyTorch [27], and this parameterization guarantees ∇ 2 ˜ x f θ (˜ x ) to be symmetric. We also notice that when using the diagonal Hessian approximation (Equation 18), we only need entries in Diag( H ) to be positive in order to obtain a valid Gaussian approximation. In this case, the score parameterization remains applicable and offers more efficient training compared to the energy parameterization. Therefore, the combination of full/diagonal covariance and energy/score parameterization provides a tradeoff between flexibility and inference speed, allowing for a flexible approach while maintaining computational efficiency during training.

4 Image Generation with a Single Noise Level

We then apply the proposed method to model the grey-scale MNIST [21] dataset. We use the standard U-Net architecture [35, 31] with a single fixed noise level σ = 0 . 5 ; the effect of varying σ is explored in Appendix C.1. For the KL training objective, the output channel size is 2 to generate both mean and log-std at the same time. For DSM training, we take the sum of the U-Net output to obtain the scalar energy evaluation which also relates to the product-of-experts model described in [34]. We train both networks for 300 epochs with learning rate 1 × 10 -4 and batch-size 100.

As discussed in Section 1.1, the KL divergence is not well-defined for manifold data distributions. This limitation becomes evident when working with MNIST, where the presence of constant black pixels in the boundary areas leads to a rapid decrease towards 0 in the variance of q θ ( x | ˜ x ) during training. Consequently, the likelihood value log q θ ( x | ˜ x ) tends to approach infinity, resulting in unstable training. In contrast, the DSM objective is welldefined for manifold data, providing

a stable training process even in the presence of such boundary effects. Figure 5 provides a visual comparison of the two training procedures, demonstrating the improved stability and effectiveness of the DSM objective in handling manifold data distributions.

For the sample generation process, calculating the full-covariance Gaussian posterior becomes challenging. We therefore apply the scalable diagonal Hessian approximation described in Section 3 to approximate the diagonal Gaussian covariance of p ( x | ˜ x ) . We find that the estimated diagonal Hessian occasionally contains small negative values due to approximation error; we, therefore, use the max( · , ϵ ) function with ϵ > 0 to ensure the pos-

(a) KL loss

(b) DSM loss

10000

8000

6000

4000

2000

0

2000

0

25000

50000

75000 100000 125000 150000 175000

0

2500050000

75000 100000 125000 150000 175000

itivity of the diagonal covariance. The ˜ x -independent isotropic covariance and the proposed ˜ x -dependent diagonal covariance share the same mean function.

We first visualize the covariance estimated by three different methods in Figure 4. We use 100 Rademacher samples in estimating the diagonal Hessian (Equation 18) and 50,000 samples in estimating the isotropic variance (Equation 17). We find that both ˜ x -dependent diagonal covariance approximations can capture the posterior structure whereas the isotropic

(a) Data

x

(b)

1

sample

(c) 10 samples (d) 100 samples

˜ x -independent covariance is just Gaussian noise since the variance is shared between different digit and pixel locations. In Figure 3, we plot the sample comparison for three methods.

Since the isotropic covariance has the same variance in each dimension, the generated samples in Figure 3b contain white noise in the black background, whereas the proposed full-covariance sampler

Algorithm 1 Sampling with Langevin Dynamics

Algorithm 2 Sampling with the proposed pseudo Gibbs Sampling

can generate a clean black background in Figure 3a. On the other hand, the samples generated by the KL-trained Gibbs sampler (Figure 3c) have worse sample quality due to the unstable training.

We then apply the same method to model the more complicated CIFAR 10 [20] dataset. We use the same U-Net structure as used in [36] and directly parameterize the score function rather than the energy function to speed up the training. The noise level is fixed at 0 . 3 . We train the model using Adam optimizer with learning rate 1 × 10 -4 and batch size 100 for 1000 epochs. We visualize the denoising posterior diagonal covariance in Figure 6 when using different numbers of Rademacher samples (Equation 18). We observe that better covariance estimation can be obtained by increasing the number of samples. To balance efficiency and accuracy, we use a sample number of 10 in the subsequent Gibbs sampling stage. Figure 7 shows three independent Markov chains with the samples plotted every 10 Gibbs steps, which demonstrates that sharp images can be generated with even one fixed level of noise.

Limitation: In the CIFAR experiment, we observe a mode collapse phenomenon when running multiple independent Markov chains for a longer time. This phenomenon is likely due to the small noise level σ = 0 . 3 , which prevents the sampler from exploring the full space, as commonly found with MCMC methods [30]. This effect is visually represented in Figure 8, where we assess the Fréchet Inception Distance (FID) values for 50,000 images sampled with varying numbers of Gibbs steps. Notably, the FID increases beyond 40 Gibbs steps, and visual evidence of mode collapse is observed (Figure 12). In the ensuing section, we will demonstrate the application of our method to settings with multiple noise levels, an approach that may help mitigate the issue of mode collapse.

5 Image Generation Using Multiple Noise Levels

The success of diffusion models and lessons from prior work on score-based generative models point to the importance of using multiple noise levels [14, 35] when modelling data with complex multi-modal distributions. Intuitively, by learning to denoise data at a range of noise levels, a single network can learn both the fine and global structure of the distribution, which in turn allows for more effective sampling algorithms capable of efficiently exploring diverse modes [35]. We therefore propose to adapt the denoising Gibbs sampling procedure to sample from distributions corrupted with multiple noise levels. For this purpose, we use a noise-conditioned score network trained by Song and Ermon [36], who generated high-quality samples using a procedure inspired by annealed Langevin dynamics [43]. This procedure involves generating samples from a sequence of distributions p T ( x T ) , ..., p 0 ( x 0 ) , corrupted by progressively decreasing levels of Gaussian noise (parameterized via standard deviations σ t , with σ T >, · · · , > σ 0 ). At a given step t in the sequence, Langevin dynamics is used to sample from the corresponding noised distribution p t ( x t ) , using the

score network s θ ( x t , σ t ) to approximate the gradient of the noised distribution. The outputs of this Langevin dynamics run are then used to initialize the same procedure at the next noise level, leading the sampling procedure to converge gradually towards the data distribution as the noise level tends to zero (i.e. p 0 ( x 0 ) ≈ p d ( x ) , σ 0 ≈ 0 ). The algorithm of the annealed Langevin dynamics with multi-level noise used in [35, 36] is summarized in Algorithm 1.

We show that the proposed Gibbs sampling scheme can be directly applied to a pre-trained scorebased generative model as a drop-in replacement for Langevin dynamics MCMC in the generation stage. At each noise level, we use samples x t +1 from the previous noise level to initialise a Gibbs sampling chain targeting the marginal distribution at the current noise level p t ( x t ) , which now plays the role of the 'clean' distribution in Equation 5. Therefore, the noisy distribution at time step t is a Gaussian p ( x t +1 | x t ) = N ( x t , σ 2 t +1 -σ 2 t ) . The optimal denoising distribution q ( x t | x t +1 ) is thus a function of the level of noise at step t relative to the level of noise at the previous step t +1 . For generation efficiency, we employ 3 Gibbs steps at each noise level, using 3 Rademacher samples to approximate the diagonal Hessian. The sampling procedure is summarized in Algorithm 2. In Figure 9 and 10, we visualize the samples from models that are trained on CIFAR10 and CelebA separately. Further experimental details can be found in Appendix C.2.

For direct comparison with the results of [36] on CIFAR10, we retain the same schedule of noise levels used to generate samples with Langevin dynamics. We generate 50000 samples using this approach and report FID and Inception scores in Table 2. Our multi-level Gibbs sampling scheme produces samples of equivalent quality to the multi-level Langevin

dynamics of [36], confirming its applicability to complex natural image data. The FID is also notably superior to that of the single-noise level Gibbs sampling, and the samples exhibit significant visual diversity (Figure 9). This underlines the importance of employing multi-level noise in our approach. Recent advances in sampling strategies for score-based models leveraging the framework of stochastic differential equations [39] have led to significant further improvements in generation quality as shown in Table 2; we leave the exploration of possible applications of our method to this framework to future work.

6 Conclusion

This paper focuses on addressing the inconsistency problem in training energy-based models (EBMs) using denoising score matching. Specifically, we identify the presence of an underlying clean model within a 'noisy' EBM and propose an efficient sampling scheme for the clean model. We demonstrate how this method can be effectively applied to high-dimensional data and showcase image generation results in both single and multi-level noise settings. More broadly, we hope our more accurate denoising posterior opens new avenues for future work on score-based methods in machine learning.

References

M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

F. Bao, C. Li, J. Zhu, and B. Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503 , 2022.

C. Bekas, E. Kokiopoulou, and Y. Saad. An estimator for the diagonal of a matrix. Applied numerical mathematics , 57(11-12):1214-1229, 2007.

Y. Bengio, L. Yao, G. Alain, and P. Vincent. Generalized denoising auto-encoders as generative models. Advances in neural information processing systems , 26, 2013.

C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

A. J. Bose, H. Ling, and Y. Cao. Adversarial contrastive estimation. arXiv preprint arXiv:1805.03642 , 2018.

Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems , 32, 2019.

Y. Du, S. Li, J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316 , 2020.

B. Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association , 106(496):1602-1614, 2011.

R. A. Fisher. Theory of statistical estimation. In Mathematical proceedings of the Cambridge philosophical society , volume 22, pages 700-725. Cambridge University Press, 1925.

R. Gao, Y. Lu, J. Zhou, S.-C. Zhu, and Y. N. Wu. Learning generative convnets via multi-grid modeling and sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9155-9164, 2018.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation , 19(2):433-450, 1990.

A. Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

A. Jolicoeur-Martineau, R. Piché-Taillefer, I. Mitliagkas, and R. T. des Combes. Adversarial score matching and improved sampling for image generation. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id= eLfqMl3z3lq .

T. Kim and Y. Bengio. Deep directed generative models with energy-based probability estimation. arXiv preprint arXiv:1606.03439 , 2016.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Y. LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

J. Martens, I. Sutskever, and K. Swersky. Estimating the hessian by back-propagating curvature. arXiv preprint arXiv:1206.6464 , 2012.

C. Meng, Y. Song, W. Li, and S. Ermon. Estimating high order gradients of the data distribution by denoising. Advances in Neural Information Processing Systems , 34:25359-25369, 2021.

T. P. Minka. Expectation propagation for approximate bayesian inference. arXiv preprint arXiv:1301.2294 , 2013.

J. Ngiam, Z. Chen, P. W. Koh, and A. Y. Ng. Learning deep energy models. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 1105-1112, 2011.

E. Nijkamp, M. Hill, S.-C. Zhu, and Y. N. Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems , 32, 2019.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.

P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

H. E. Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory , pages 388-394. Springer, 1992.

C. P. Robert, G. Casella, and G. Casella. Monte Carlo statistical methods , volume 2. Springer, 1999.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

T. Salimans and J. Ho. Should ebms model the energy or the score? In Energy Based Models Workshop-ICLR 2021 , 2021.

S. Saremi. On approximating ∇ f with neural networks. arXiv preprint arXiv:1910.12744 , 2019.

S. Saremi, A. Mehrjou, B. Schölkopf, and A. Hyvärinen. Deep energy estimator networks. arXiv preprint arXiv:1805.08306 , 2018.

Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019.

Y. Song and S. Ermon. Improved techniques for training score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 92c3b916311a5517d9290576e3ea37ad-Abstract.html .

Y. Song and D. P. Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 , 2021.

Y. Song, S. Garg, J. Shi, and S. Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence , pages 574-584. PMLR, 2020.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=PxTIG12RRHS .

T. Tao. An introduction to measure theory , volume 126. American Mathematical Society Providence, 2011.

P. Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Z. Wang, S. Cheng, L. Yueru, J. Zhu, and B. Zhang. A wasserstein minimum velocity approach to learning unnormalized models. In International Conference on Artificial Intelligence and Statistics , pages 3728-3738. PMLR, 2020.

M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688, 2011.

H. Wendland. Scattered data approximation , volume 17. Cambridge university press, 2004.

L. Wenliang and H. Kanagawa. Blindness of score-based methods to isolated components and mixing proportions. arXiv preprint arXiv:2008.10087 , 2020.

J. Xie, Y . Lu, S.-C. Zhu, and Y . Wu. A theory of generative convnet. In International Conference on Machine Learning , pages 2635-2644. PMLR, 2016.

S. Zhai, Y. Cheng, R. Feris, and Z. Zhang. Generative adversarial networks as variational training of energy based models. arXiv preprint arXiv:1611.01799 , 2016.

M. Zhang, T. Bird, R. Habib, T. Xu, and D. Barber. Variational f-divergence minimization. arXiv preprint arXiv:1907.11891 , 2019.

M. Zhang, P. Hayes, T. Bird, R. Habib, and D. Barber. Spread divergence. In International Conference on Machine Learning , pages 11106-11116. PMLR, 2020.

M. Zhang, P. Hayes, and D. Barber. Generalization gap in amortized inference. Advances in neural information processing systems , 2022.

M. Zhang, O. Key, P. Hayes, D. Barber, B. Paige, and F.-X. Briol. Towards healing the blindness of score matching. arXiv preprint arXiv:2209.07396 , 2022.

A Proof and Derivations

A.1 Proof of Theorem 2.1

The existence is straightforward, since FD(˜ p d || ˜ q θ ∗ ) = 0 → ˜ p d = ˜ q θ ∗ , we can simply let q ( x ) = p d ( x ) , which makes ∫ q ( x ) p (˜ x | x ) d x = ∫ p d ( x ) p (˜ x | x ) d x = ˜ p d . To show the uniqueness, we denote density k ( ϵ ) = N (0 , σ 2 I ) , so ˜ q θ (˜ x ) and ˜ p d ( x ) can be written as convolutions

we then have

where F denotes the Fourier transform. Since the Fourier transform of a Gaussian is also a Gaussian, so F ( k ) > 0 everywhere, we have

Therefore, q = p d is the unique distribution that makes ˜ p d = ˜ q θ . This technique has also been used to construct spread KL divergence (we denote as ˜ KL ) [49], which is defined as ˜ KL( p d || q θ ) ≡ KL( p d ∗ k || q θ ∗ k ) where k ( ϵ ) = N (0 , σ 2 I ) , to train implicit model q θ . Different from the DSM situation, when ˜ KL( p d || q θ ) = 0 , the underlying model q θ = p d is directly available, whereas the EBM ˜ q θ trained by DSM learns to be the noisy distribution ˜ q θ = p d ∗ k .

A.2 General Conditions Characterising the Existence of the Clean Model

In the previous section, we assume for a flexible neural network parameterized f θ , the energy-based model ˜ q θ (˜ x ) = exp( -f (˜ x )) /Z ( θ ) trained by Equation 4 can recover the target noisy data distribution ˜ q θ ∗ = ˜ p d so there exists an underlying model q such that ˜ q θ ∗ = q ∗ k and q = p d . This assumption is commonly used in the literature on score-based methods. For example, in the score-based diffusion models literature [35, 14, 2], for any data x ∈ R D , the score function ∇ ˜ x log ˜ q θ (˜ x ) is usually parameterized by a neural network NN θ ( · ) : R D → R D . However, this parameterization cannot guarantee NN θ (˜ x ) is a conservative vector field, or in other words, there doesn't exist a distribution ˜ q θ (˜ x ) such that ∇ ˜ x ˜ q θ (˜ x ) = ∇ ˜ x log ˜ q θ (˜ x ) and ∇ 2 ˜ x log ˜ q (˜ x ) is symmetric [32, 33]. Therefore, perfect score estimation ∇ ˜ x log ˜ p d (˜ x ) = ∇ ˜ x log ˜ q θ (˜ x ) is implicitly assumed to allow an EBM interpretation.

̸

However, the underlying clean model doesn't always exist for imperfect model ˜ q θ = ˜ p d . We here provide the sufficient and necessary conditions which guarantee the existence of the underlying clean model.

Theorem A.1 (Necessary and Sufficient conditions for the existence of the underlying clean model.) . For a model ˜ q θ with the convolutional noise distribution k ( ϵ ) = N (0 , σ 2 I ) , there exists an underlying model q such that q ∗ k = ˜ q if and only if F (˜ q θ ) / F ( k ) is positive semi-definite 7 . Additionally, the underlying distribution q can be written as

where F -1 is the inverse Fourier transform. This theorem is a straightforward corollary of Bochner's Theorem 8 . However, for the energy model ˜ q θ (˜ x ) ∝ exp( -f θ (˜ x )) , it's difficult to design a functioning family of f that satisfies the positive semi-definite condition and have the tractable score function at the same time 9 . We thus leave the design of better energy function parameterizations as a promising future direction.

A.3 Proof of Theorem 2.2

Derivation of the Mean Identity

We let ˜ q θ (˜ x ) = ∫ k (˜ x | x ) q θ ( x ) d˜ x , where k (˜ x | x ) = N ( x, σ 2 I ) , we have

where we define the model denoising posterior using Bayes rule q θ ( x | ˜ x ) ≡ k (˜ x | x ) q θ ( x ) / ˜ q θ (˜ x ) . The second equality is due to the following Gaussian distribution property

Derivations of the Analytical Full Covariance Identity

We have derived the mean identity

Taking the gradient over x in both side and scaling with σ 2 , we have

We can also expand the hessian of the log ˜ q θ (˜ x ) :

Therefore, we obtain the analytical full covariance identity.

A.4 Proof of Theorem 2.3

Lemma A.2 (KL to Gaussian [2]) . Let p ( x ) be a distribution with mean µ p and covariance Σ p and q ( x ) = N ( µ q , Σ q ) , denote the differential entropy as H( p ) ≡ -∫ p ( x ) log p ( x ) d x , we have

The proof can be found in [2] Lemma 2.

We can then prove Theorem 2.3. Since p (˜ x | x ) p d ( x ) = p ( x | ˜ x )˜ p d (˜ x ) , where ˜ p d (˜ x ) = ∫ p d ( x ) p (˜ x | x ) d x , we have

Assume Gaussian distribution q ( x | ˜ x ) = N ( µ q (˜ x ) , Σ q (˜ x )) and denote the mean and covariance of the true posterior are µ p (˜ x ) and Σ p (˜ x ) , then the optimal q ∗ is

Therefore, the optimal q ( x | ˜ x ) = N ( µ q (˜ x ) , Σ q (˜ x )) under the joint KL has the mean and covariance µ ∗ q (˜ x ) = µ p (˜ x ) , Σ ∗ q (˜ x )) = Σ p (˜ x ) .

B Connection to Analytical DDPM

Paper [2] considers the constrained variational family q θ ( x | ˜ x ) = N ( µ θ (˜ x ) , σ 2 q I ) and derive the optimal σ ∗ q as

which can also be rewritten using the score function

To make a deep connection, we can also plug our analytical full covariance (Equation 10) into Equation 16

which recovers Equation 17, where the first equality is due to the well-known Fisher information identity [10].

C Experiments

All the experiments conducted in this paper are run on one single NVDIA GTX 3090.

C.1 Effect of the Single Noise Choice on MNIST

Figure 11 shows the samples generated by our method with the EBM trained with difference σ ∈ { 0 . 3 , 0 . 5 , 0 . 8 } in the noise distribution p (˜ x | x ) , we can find the image quality also heavily depends on the choice of the noise scale and σ = 0 . 5 achieves the best visual quality, we then use this hyper-parameter in the subsequent comparisons.

C.2 Multi-level Noise Details

For full details on the architecture and noise schedule used in the multi-level noise experiments in Section 5, we refer to Appendix B of [36]. For our multi-level Gibbs sampling procedure, we used 3 Gibbs steps at each noise level and 3 Rademacher samples for each diagonal Hessian computation. Following [36], we used a total of 232 noise levels, distributed according to their proposed geometric schedule, and applied a final denoising step in which the mean of the clean distribution conditioned on the final output of the sampling procedure is returned (the final output of the sampling procedure is a sample from the noised distribution from the noise distribution at the smallest noise level). This denoising step was previously found to improve FID scores [17] significantly.

(a) σ = 0 . 3

(b) σ = 0 . 5

(c) σ = 0 . 8

762

7

2

0

8080

212

55

7

2

3

O

9672

W

8

3

805

5

5

8906

6382

8

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new denoising scoring matching (DSM) technique to train energy-based models. They address the well-known inconsistency problem of DSM objectives by training a \"noisy\" model and then sampling \"clean\" samples from the \"noisy\" model by learning a denoising distribution.\n\nThe contributions of the paper are as follows:\n\n1) Theoretically, they show that a \"clean\" model is uniquely identified by an EBM learning a noisy data distribution.\n2) A sampling strategy inspired from Gibbs sampling to approximate the denoising distribution, which allows to sample from the underlying clean model without the need to retrain the noisy model.\n3) Practically, they show how to scale their approach to high-dimensional data by overcoming the quadratic cost of evaluating the Hessian of the DSM for every $\\tilde{x}$.\n\nSTRENGTHS:\n- Well-written paper, easy to follow and most claims are justified either by citing previous work or in appendix / math proofs\n- Prior work is duly cited\n- Experiments on synthetic and real-world (MNIST/CIFAR) datasets are appreciated to assess the quality of the technique proposed\n\nWEAKNESSES:\nI'm not an expert in this field and I have a hard time evaluating the overall contribution of this paper to the field of \"Generative Models\". In particular, it feels like this denoising techniques introduce a scalability problem (although partly discussed and solved by contribution 3) in order to solve the inconsistency problem of score-based matching training. A discussion on the quality of the approximation and comparing the full Hessian with its approximation could help answer my doubts.\n\nBesides, I think that describing the full sampling algorithm could help the reader get a better picture of the whole training and sampling procedure. This would clarify what has been made in the experiments section.\n\nI list below a few comments / typos I have found while reviewing the paper:\n\n1) l.33: there is a missing $\\nabla_x$ in the definition of the score matching function. Otherwise, I don't see how $\\nabla_x s_{p_{\\theta}}(x) = -\\nabla^2_x f_{\\theta}(x)$.\n2) l.26: \"it is equivalent to MLE\". A reference is welcome. \n3) l.34: \"under certain regularity conditions, the Fisher divergence is equivalent to SM\". Reference?\n\nQUESTIONS:\nAs stated above in the weaknesses section, my main doubt revolves around how does the Hessian approximation affect the training stability and overall output samples quality.\n\nLIMITATIONS:\nThe authors have partly discussed the limitations in section 3, but an experiment comparing the quality of the approximation of the generated samples with the full Hessian and the approximated Hessian is welcome to strengthen the demonstration.\n\nNo potential negative societal impact at this stage.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This paper proposed a sampling scheme called (pseudo)-Gibbs sampling with moment machine. It can be applicable to the sampling problem from  from the underlying clean model when a noisy model hat has been well-trained via DSM. The proposed sampling technique is used to sample images for  image generation and showed good efficiency on toy dataset.

STRENGTHS:
1. The paper is well written
2. The problem is a basic problem with many applications
3.  The idea is new and seems working well.

WEAKNESSES:
The improvement in image generation seems minor.

QUESTIONS:
NIL

LIMITATIONS:
NIL
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper addressed the inconsistency problem in training energy-based models (EBMs) using denoising score matching (DSM). An efficient sampling framework, called (pseudo)-Gibbs sampling with moment matching, was proposed to enable effective sampling from the underlying clean model when given a well-trained 'noisy' model obtained through DSM. The presence of an underlying clean model within the noisy EBM was identified, and a pseudo-Gibbs sampling scheme incorporating analytical moment matching approximation was introduced for sampling from the clean model without requiring additional training. The effectiveness of the method was demonstrated in generating high-quality images for both single and multi-level noise settings. Additionally, the scalability of the method to high-dimensional datasets was illustrated. Comparisons with related methods highlighted the advantages of the proposed approach.

STRENGTHS:
The theoretical derivation and presentation are clear and well-defined.

WEAKNESSES:
The limited number of experiments conducted to validate the proposed method. While the paper provides some demonstration of the effectiveness of the approach, further experiments across a wider range of scenarios and datasets would enhance the robustness and generalizability of the findings.

The authors should provide more details for those experiments in Supplementary Material.

The authors failed to demonstrate the wide applications of the proposed method.

QUESTIONS:
In experiments, the authors used a noise-conditioned score network trained in [32], does the proposed model also excel in other settings?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This work proposes a sampling method from implicitly defined clean model given a noisy model without retraining and demonstrates its applicability to high-dimensional data / images.

STRENGTHS:
- this work presents interesting theoretical results such as the uniqueness of underlying clean model given noisy data distribution, analytical covariance identity, and optimal Gaussian approx. 
- this work presents interesting corrections to prior works such as analytical DDPM, KL training Gibbs sampling.

WEAKNESSES:
- It is unclear if the proposed method (e.g., Eq. (12)) is effective in general. Some of the toy examples look promising, but some image examples may not look good (see Table 2 for large performance gaps with DDPM and NCSN++ / possibly more with recent methods). While this work emphasized that it will work well with a single level of fixed noise, eventually the proposed method with multi-level noise was used for more complex image cases without the issue of mode collapse. This inconsistency is my major concern.
- Weak experimental results: some of the experiments may not effectively demonstrate the performance of the proposed scheme well. For example, it is unclear what this work wanted to show in Figures 5, 6 and 8. Comparison experiments may be needed.

QUESTIONS:
Please address the comments in the Weaknesses section. 

- What if log scale is used in Fig 3(a)? Then, it may show similar curves like Fig 3(b) unlike the claims in this work.
- It is unclear why the comparable results of NCSNv2 with Langevin / the proposed method is ok. Are they using the same multi-level noise? Then, what is the advantage of the proposed method over the conventional Langevin dynamics?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a sampling framework, i.e., (pseudo)-Gibbs sampling with moment matching, which aims at sampling from the implicitly-defined clean model given the noisy energy-based model well-trained by the denoising score matching method. It first provides a theoretical justification of the existence of the underlying clean model for a well-trained energy-based model. It then introduces the gibbs sampling framework based on the proposed analytical Gaussian moment matching approximation method for the denoising distribution. Experiments on several toy datasets and MNIST and CIFAR10 datasets are conducted to demonstrate the effectiveness of the proposed approximation and sampling method.\n\nSTRENGTHS:\n- The paper provides theoretical justification of the existence of the underlying clean model of a well-trained noisy EBM, and discusses the existence of the clean model for an imperfect EBM. \n- It derives an analytical Gaussian moment matching approximation method for the denoising distribution and validates the design on several toy datasets, and discusses some scalable implementations of the proposed method for image data.\n- The presentation in general is clear.\n\nWEAKNESSES:\n- The full covariance moment matching requires calculating the Hessian matrix, which could be computationally expensive even with diagonal approximation methods.\n- The experiments are conducted on toy datasets and MNIST and CIFAR-10 datasets. More results on other image dataset could provide better intuitive understanding of the proposed method.\n- The performance of the proposed sampling method is close to the existing methods. The significance of the proposed sampling method is not very clear.\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nThe authors have mentioned the limitations in Sec 3. Scalable Implementations for Image Data and Sec 4. Image Generation with a Single Noise Level.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}