Title: One-Line-of-Code Data Mollification Improves Optimization of Likelihood-based Generative Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative Models

Normalizing Flows

Variational Autoencoders
Abstract: Generative Models (GMs) have attracted considerable attention due to their tremendous success in various domains, such as computer vision where they are capable to generate impressive realistic-looking images. Likelihood-based GMs are attractive due to the possibility to generate new data by a single model evaluation. However, they typically achieve lower sample quality compared to state-of-the-art score-based Diffusion Models (DMs). This paper provides a significant step in the direction of addressing this limitation. The idea is to borrow one of the strengths of score-based DMs, which is the ability to perform accurate density estimation in low-density regions and to address manifold overfitting by means of data mollification. We propose a view of data mollification within likelihood-based GMs as a continuation method, whereby the optimization objective smoothly transitions from simple-to-optimize to the original target. Crucially, data mollification can be implemented by adding one line of code in the optimization loop, and we demonstrate that this provides a boost in generation quality of likelihood-based GMs, without computational overheads. We report results on real-world image data sets and UCI benchmarks with popular likelihood-based GMs, including variants of variational autoencoders and normalizing flows, showing large improvements in FID score and density estimation.

One-Line-of-Code Data Mollification Improves Optimization of Likelihood-based Generative Models

Abstract

Generative Models (GMs) have attracted considerable attention due to their tremendous success in various domains, such as computer vision where they are capable to generate impressive realistic-looking images. Likelihood-based GMs are attractive due to the possibility to generate new data by a single model evaluation. However, they typically achieve lower sample quality compared to state-of-the-art score-based Diffusion Models (DMs). This paper provides a significant step in the direction of addressing this limitation. The idea is to borrow one of the strengths of score-based DMs, which is the ability to perform accurate density estimation in low-density regions and to address manifold overfitting by means of data mollification. We propose a view of data mollification within likelihood-based GMs as a continuation method, whereby the optimization objective smoothly transitions from simple-to-optimize to the original target. Crucially, data mollification can be implemented by adding one line of code in the optimization loop, and we demonstrate that this provides a boost in generation quality of likelihood-based GMs, without computational overheads. We report results on real-world image data sets and UCI benchmarks with popular likelihood-based GMs, including variants of variational autoencoders and normalizing flows, showing large improvements in FID score and density estimation.

1 Introduction

Generative Models (GMs) have attracted considerable attention recently due to their tremendous success in various domains, such as computer vision, graph generation, physics and reinforcement learning [see e.g., 37, 39, 75, 76, and references therein]. Given a set of data points, GMs attempt to characterize the distribution of such data so that it is then possible to draw new samples from this. Popular approaches include Variational Autoencoders (VAEs), Normalizing Flows (NFs), Generative Adversarial Networks (GANs), and score-based Diffusion Models (DMs).

In general, the goal of any GMs is similar to that of density estimation with the additional aim to do so by constructing a parametric mapping between an easy-to-sample-from distribution p s and the desired data distribution p data. While different GMs approaches greatly differ in their optimization strategy and formulation, the underlying objectives share some similarity due to their relation to the optimal transport problem, defined as arg min π ∫ ‖ x -y ‖ 2 dπ ( x , y ) . Here π is constrained to belong to the set of joint distributions with marginals p s , p data, respectively [19, 42]. This unified

perspective is explicitly investigated for GANs and VAEs [19] for example, whereas other works study NFs [50]. Similarly, DMs can be connected to Schrodinger Bridges [8], which solve the problem of entropy-regularized optimal transport [55]. Given that extensions of the regularized optimal transport case are available also for other generative models [65, 58], we should expect that, in principle, any technique should allow generation of samples with similar quality, provided it is properly tuned. However, this is not true in practice. The different formulations lead to a variety of properties associated with GMs, and pros and cons of each formulation can be understood through the so-called GM tri-lemma [84]. The three desirable properties of GMs are high sample quality, mode coverage, and fast sampling, and it has been argued that such goals are difficult to be satisfied simultaneously [84] .

The state-of-the-art is currently dominated by score-based DMs, due to their ability to achieve high sample quality and good mode coverage. However, generating new samples is computationally expensive due to the need to simulate stochastic differential equations. Likelihood-based GMs are complementary, in that they achieve lower sample quality, but sampling requires one model evaluation per sample and it is therefore extremely fast. While some attempts have been made to bridge the gap by combining GANs with DMs [84] or training GANs with diffusions [82], these still require careful engineering of architectures and training schedules. The observation that all GMs share a common underlying objective indicates that we should look at what makes DMs successful at optimizing their objective. Then, the question we address in this paper is: can we borrow the strengths of score-based DMs to improve likelihood-based GMs, without paying the price of costly sample generation?

One distinctive element of score-based DMs is data mollification, which is typically achieved by adding Gaussian noise [69] or, in the context of image data sets, by blurring [61]. A large body of evidence points to the manifold hypothesis [63], which states that the intrinsic dimensionality of image data sets is much lower than the dimensionality of their input. Density estimation in this context is particularly difficult because of the degeneracy of the likelihood for any density concentrated on the manifold where data lies [43]. Under the manifold hypothesis, or even when the target density is multi-modal, the Lipschitz constant of GMs has to be large, but regularization, which is necessary for robustness, is antagonist to this objective [64, 9]. As we will study in detail in this paper, the process of data mollification gracefully guides the optimization mitigating manifold overfitting and enabling a desirable tail behavior, yielding accurate density estimation in low-density regions. In likelihood-based GMs, data mollification corresponds to some form of simplification of the optimization objective. This type of approach, where the level of data mollification is annealed throughout training, can be seen as a continuation method [83, 47], which is a popular technique in the optimization literature to reach better optima.

Strictly speaking, data mollification in score-based DMs and likelihood-based GMs are slightly different. In the latter, the amount of noise injected in the data is continuously annealed throughout training. At the beginning, the equivalent loss landscape seen by the optimizer is much smoother, due to the heavy perturbation of the data, and a continuous reduction of the noise level allows optimization to be gracefully guided until the point where the level of noise is zero [83, 47]. DMs, instead, are trained at each step of the optimization process by considering all noise levels simultaneously, where complex amortization procedures, such as self-attention [70], allow the model to efficiently share parameters across different perturbation levels. It is also worth mentioning that score-based DMs possess another distinctive feature in that they perform gradient-based density estimation [69, 28]. It has been conjectured that this can be helpful to avoid manifold overfitting by allowing for the modeling of complex densities while keeping the Lipschitz constant of score networks low [64]. In this work, we attempt to verify the hypothesis that data mollification is heavily responsible for the success of score-based DMs. We do so by proposing data mollification for likelihood-based GMs, and provide theoretical arguments and experimental evidence that data mollification consistently improves their optimization. Crucially, this strategy yields better sample quality and it is extremely easy to implement, as it requires adding very little code to any existing optimization loop.

We consider a large set of experiments involving VAEs and NFs and some popular image data sets. These provide a challenging test for likelihood-based GMs due to the large dimensionality of the input space and to the fact that density estimation needs to deal with data lying on manifolds. The results show systematic, and in some cases dramatic, improvements in sample quality, indicating that this is a simple and effective strategy to improve optimization of likelihood-based GMs models. The paper is organized as follows: in § 2 we illustrate the challenges associated with generative modeling when data points lie on a manifold, particularly with density estimation in low-density regions and

manifold overfitting; in § 3 we propose data mollification to address these challenges; § 4 reports the experiments with a discussion of the limitations and the broader impact, while § 5 presents related works, and § 6 concludes the paper.

2 Challenges in Training Deep Generative Models

We are interested in unsupervised learning, and in particular on the task of density estimation. Given a dataset D consisting of N i.i.d samples D ∆ = { x i } N i =1 with x i ∈ R D , we aim to estimate the unknown continuous generating distribution p data ( x ) . In order to do so, we introduce a model p θ ( x ) with parameters θ and attempt to estimate θ based on the dataset D . A common approach to estimate θ is to maximize the likelihood of the data, which is equivalent to minimizing the following objective:

There are several approaches to parameterize the generative model p θ ( x ) . In this work, we focus on two widely used likelihood-based Generative Models (GMs), which are Normalizing Flows (NFs) [52, 39] and Variational Autoencoders (VAEs) [36, 59]. Although NFs and VAEs are among the most popular deep GMs, they are characterized by a lower sample quality compared to GANs and score-based DMs. In this section, we present two major reasons behind this issue by relying on the manifold hypothesis.

2.1 The Manifold Hypothesis and Density Estimation in Low-Density Regions

The manifold hypothesis is a fundamental concept in manifold learning [63, 73, 1] stating that real-world high-dimensional data tend to lie on a manifold M characterized by a much lower dimensionality compared to the one of the input space (ambient dimensionality) [48]. This has been verified theoretically and empirically for many applications and datasets [51, 48, 57, 72]. For example, [57] report extensive evidence that natural image datasets have indeed very low intrinsic dimension relative to the high number of pixels in the images.

The manifold hypothesis suggests that density estimation in the input space is challenging and ill-posed. In particular, data points on the manifold should be associated with high density, while points outside the manifold should be considered as lying in regions of nearly zero density [45]. This implies that the target density in the input space should be characterized by high Lipschitz constants. The fact that data is scarce in regions of low density makes it difficult to expect that models can yield accurate density estimation around the tails. These pose significant challenges for the training of deep GMs [9, 45, 69]. Recently, diffusion models [69, 25, 70] have demonstrated the ability to mitigate this problem by gradually transforming a Gaussian distribution, whose support spans the full input space, into the data distribution. This observation induces us to hypothesize that the

Data distribution

Estimated distribution

data mollification mechanism in score-based DMs is responsible for superior density estimation in low-density regions.

To demonstrate the challenges associated with accurate estimation in low-density regions, we consider a toy experiment where we use a REAL-NVP flow [14] to model a two-dimensional mixture of Gaussians, which is a difficult test for NFs in general. Details on this experiment are provided in the Appendix D. Fig. 1 depicts the true and estimated densities, and their corresponding scores, which are the gradient of the log-density function with respect to the data [28]. Note that the use of 'score' here is slightly different from that from traditional statistics where score usually refers to the gradient of the log-likelihood with respect to model parameters. As it can be seen in the figure, in regions of low data density, p θ ( x ) is completely unable to model the true density and scores. This problem is due to the lack of data samples in these regions and may be more problematic under the manifold hypothesis and for high-dimensional data such as images. In § 4, we will demonstrate how it is possible to considerably mitigate this issue by means of data mollification.

2.2 Manifold Overfitting

The manifold hypothesis suggests that overfitting on a manifold can occur when the model p θ ( x ) assigns an arbitrarily large likelihood in the vicinity of the manifold, even if the distribution does not accurately capture the true distribution p data ( x ) [10, 43]. This issue is illustrated in Fig. 2 of [43] and it will be highlighted in our experiment ( § 4.1), where the true data distribution p data ( x ) is supported on a one-dimensional curve manifold M in two-dimensional space R 2 . Even when the model distribution p θ ( x ) poorly approximates p data ( x ) , it may reach a high likelihood value by concentrating the density around the correct manifold M . If p θ ( x ) is flexible enough, any density defined on M may achieve infinite likelihood and this might be an obstacle for retrieving p data ( x ) .

A theoretical formalization of the problem of manifold overfitting appears in [43] and it is based on the concept of Riemannian measure [56]. The Riemannian measure on manifolds holds an analogous role to that of the Lebesgue measure on Euclidean spaces. To begin, we establish the concept of smoothness for a probability measure on a manifold.

Definition 1. Let M be a finite-dimensional manifold, and p be a probability measure on M . Let g be a Riemannian metric on M and µ ( g ) M the corresponding Riemannian measure. We say that p is smooth if p glyph[lessmuch] µ ( g ) M and it admits a continuous density p : M→ R > 0 with respect to µ ( g ) M .

We now report Theorem 1 from [43] followed by a discussion on its implications for our work.

Theorem 1. (Gabriel Loaiza-Ganem et al. [43]). Let M ⊂ R D be an analytic d -dimensional embedded submanifold of R d with d < D , µ D is the Lebesgue measure on R D , and p † a smooth probability measure on M . Then there exists a sequence of probability measures { p ( t ) θ } ∞ t =0 on R D such that:

Theorem 1 holds for any smooth probability measure supported in M . This is an important point because this includes the desired p data, provided that this is smooth too. The key message in [43] is that, a-priori, there is no reason to expect that for a likelihood-based model to converge to p data out of all the possible p † . Their proof is based on convolving p † with a Gaussian kernel with variance σ 2 t that decreases to zero as t → ∞ , and then verify that the stated properties of p ( t ) θ hold. Our analysis, while relying on the same technical tools, is instead constructive in explaining why the proposed data mollification allows us to avoid manifold overfitting. The idea is as follows: at time step t = 0 , we select the desired p data convolved with a Gaussian kernel with a large, but finite, variance σ 2 (0) as the target distribution for the optimization. Optimization is performed and p (0) θ targets this distribution, without any manifold overfitting issues, since the dimensionality of the corrupted data is non-degenerate. At the second step, the target distribution is obtained by convolving p data with the kernel with variance σ 2 (1) < σ 2 (0) , and again manifold overfitting is avoided. By iteratively repeating this procedure, we can reach the point where we are matching a distribution convolved with an arbitrarily small variance σ 2 ( t ) , without ever experiencing manifold overfitting. When removing the last bit of perturbation we fall back to the case where we experience manifold overfitting. However, when we operate in a stochastic setting, which is the typical scenario for the GMs considered here, we avoid ending up in solutions for which the density is degenerate and with support which is exactly the data manifold. Another way to avoid instabilities is to adopt gradient clipping. Note that, as mentioned in [43] and verified by ourselves in earlier investigations, a small constant amount of noise does not provide any particular benefits over the original scheme, whereas gradually reducing the level of data mollification improves optimization dramatically.

2.3 Data Mollification as a Continuation Method

We can view the proposed data mollification approach as a continuation method [83, 47]. Starting from the target objective function, which in our case is L ( θ ) in Eq. 4 (or a lower bound in the case of VAEs), we construct a family of functions H ( θ , γ ) parameterized by an auxiliary variable

γ ∈ [0 , 1] so that H ( θ , 0) = L ( θ ) . The objective functions H ( θ , γ ) are defined so that the higher γ the easier is to perform optimization. In our case, when γ = 1 we operate under a simple regime where we target a Gaussian distribution, and likelihood-based GMs can model these rather easily. By annealing γ from 1 to 0 with a given schedule, the sequence of optimization problems with objective H ( θ , γ ) is increasinly more complex to the point where we target L ( θ ) . In essence, the proposed data mollification approach can be seen as a good initialization method, as the annealing procedure introduces a memory effect in the optimization process, which is beneficial in order to obtain better optima.

3 Generative Models with Data Mollification

Motivated by the aforementioned problems with density estimation in low-density regions and manifold overfitting, we propose a simple yet effective approach to improve likelihood-based GMs. Our method involves mollifying data using Gaussian noise, gradually reducing its variance, until recovering the original data distribution p data ( x ) . This mollification procedure is similar to the reverse process of diffusion models, where a prior noise distribution is smoothly transformed into the data distribution [69, 25, 70]. As already mentioned, data mollification alleviates the problem of manifold overfitting and it induces a memory effect in the optimization which improves density estimation in regions of low density.

Training Iteration

Algorithm 1:

Gaussian mollification

1

for

t

1

,

2

, ..., T

do

2

x

∼

p

data

(

x

)

// Sample training

data

3

˜

x

t

=

α

t

x

+

σ

t

ε

data

with

α

t

, σ

2

t

γ

(

t/T

)

and

ε

∼ N

(

0

,

I

)

4

θ

t

UPDATE

(

θ

t

-

1

,

˜

x

t

)

// Train model

Gaussian Mollification. Given that we train the model p θ ( x ) for T iterations, we can create a sequence of progressively less smoothed versions of the original data x , which we refer to as mollified data ˜ x t . Here, t ranges from t = 0 (the most mollified) to t = T (the least mollified). For any t ∈ [0 , T ] , the distribution of the mollified data ˜ x t , conditioned on x , is given as follows:

where α t and σ 2 t are are positive scalar-valued functions of t . In addition, we define the signal-tonoise ratio SNR ( t ) = α 2 t /σ 2 t . and we assume that it monotonically increases with t , i.e., SNR ( t ) ≤ SNR ( t +1) for all t ∈ [0 , T -1] . In other words, the mollified data ˜ x t is progressively less smoothed as t increases. In this work, we adopt the variance-preserving formulation used for diffusion models [67, 25, 32], where α t = √ 1 -σ 2 t and σ 2 t = γ ( t/T ) . Here, γ ( · ) is a monotonically decreasing function from 1 to 0 that controls the rate of mollification. Intuitively, this procedure involves gradually transforming an identity-covariance Gaussian distribution into the distribution of the data. Algorithm 1 summarizes the proposed Gaussian mollification procedure, where the red line indicates a simple additional step required to mollify data compared with vanilla training using the true data distribution.

Noise schedule. The choice of the noise schedule γ ( · ) has an impact on the performance of the final model. In this work, we follow common practice in designing the noise schedule based on the literature of score-based DMs [49, 26, 7]. In particular, we adopt a sigmoid schedule [31], which has recently been shown to be more effective in practice compared to other choices such as linear [25] or cosine schedules [49].

Iteration 5000

Iteration 20000

Iteration 0

Estimated distribution

Target distribution

True scores

2

.

3

4

.

61

MMD

2

[log]

0

5000 10000 15000 20000

1

.

15

1

.

38

Training loss [log]

Vanilla

Iteration

Mollification

The sigmoid schedule γ ( t/T ) [31] is defined through the sigmoid function:

where τ is a temperature hyper-parameter. This function is then scaled and shifted to ensure that γ (0) = 1 and γ (1) = 0 . We encourage the reader to check the implementation of this schedule, available in Appendix C. Fig. 3 illustrates the sigmoid schedule and the corresponding log( SNR ) with different values of τ . We use a default temperature of 0 . 7 as it demonstrates consistently good results in our experiments.

4 Experiments

In this section, we demonstrate emprically the effectiveness of our proposal through a wide range of experiments on synthetic data, and some popular real-world tabular and image data sets. Appendix D contains a detailed description of each experiment to guarantee reproducibility.

4.1 2D Synthetic Data Sets

We begin our experimental campaign with two 2D synthetic data sets. The two-dimensional nature of these data sets allows us to demonstrate the effectiveness of Gaussian mollification in mitigating the challenges associated with density estimation in low-density regions and manifold overfitting. Here, we consider p θ ( x ) to be a REAL-NVP flow [14], which comprises five coupling bijections, each consisting of a two-hidden layer multilayer perceptron (MLP). To assess the capability of p θ ( x ) to recover the true data distribution, we use Maximum Mean Discrepancy (MMD) [21] with a radial basis function (RBF) kernel on a held-out set. In these experiments, we employ the Gaussian mollification strategy presented in the previous section and compare the estimated density with the vanilla approach where we use the original training data without any mollification.

Mixture of Gaussians. First, we consider a target distribution that is a mixture of two Gaussians, as depicted in Fig. 4. As discussed in § 2.1, the vanilla training procedure fails to accurately estimate the true data distribution and scores, particularly in the low-density regions. The estimated densities and the mollified data during the training are depicted in Fig. 4. Initially, the mollification process considers a simpler coarse-grained version of the target density, which is easy to model. This is demonstrated by the low training loss at the beginning of the optimization, as depicted in Fig. 5. Subsequently, the method gradually reduces the level of noise allowing for a progressive refinement of the estimated versions of the target density. This process uses the solution from one level of

0

0

.

5

1

0

0

.

5

1

t/T

σ

2

0

0

.

5

1

0

10

t/T

log(

SNR

)

mollification as a means to guiding optimization for the next. As a result, Gaussian mollification facilitates the recovery of the modes and enables effective density estimation in low-density regions. The vanilla training procedure, instead, produces a poor estimate of the target density, as evidenced by the trace-plot of the MMD 2 metric in Fig. 5 and the visualization of the scores in Fig. 4.

Von Mises distribution. We proceed with an investigation of the von Mises distribution on the unit circle, as depicted in Fig. 6, with the aim of highlighting the issue of manifold overfitting [43]. In this experiment, the data lies on a one-dimensional manifold embedded in a two-dimensional space. The vanilla training procedure fails to approximate the target density effectively, as evidenced by the qualitative results and the substantially high value of MMD 2 ( ≈ 383 . 44 ) shown in Fig. 6. In contrast, Gaussian mollification gradually guides the estimated density towards the target, as depicted in Fig. 6, leading to a significantly lower MMD 2 ( ≈ 6 . 13 ). Additionally, the mollification approach enables the estimated model not only to accurately learn the manifold but also to capture the mode of the density correctly.

4.2 Image Experiments

Setup. We evaluate our method on image generation tasks on CIFAR10 [40] and CELEBA 64 [41] datasets, using a diverse set of likelihood-based GMs. The evaluated models include the vanilla VAE [36], the β -VAE [24], and the VAE-IAF [35] which employs an expressive inverse autoregressive flow for the approximate posterior. To further obtain flexible approximations of the posterior of latent variables as well as a tight evidence lower bound (ELBO), we also select the Hamiltonian-VAE (HVAE) [5] and the importance weighted VAE (IWAE) [4]. For flow-based models, we consider the REAL-NVP [14] and GLOW [34] models in our benchmark. We found that further training the model on the original data after the mollification procedure leads to better performance. Hence, in our approach we apply data mollification during the first half of the optimization phase, and we continue optimize the model using the original data in the second half. Nevertheless, to ensure a fair comparison, we adopt identical settings for the vanilla and the proposed approaches, including random seed, optimizer, and the total number of iterations.

Target distribution

t=500

t=1000

t=1200

t=18000

t=20000

Blurring mollification. Even though Gaussian mollification is motivated by the manifold hypothesis, it is not the only way to mollify the data. Indeed, Gaussian mollification does not take into account certain inductive biases that are inherent in natural images, including their multi-scale nature. Recently, [61, 27, 11] have proposed methods that incorporate these biases in diffusion-type generative models. Their approach involves stochastically reversing the heat equation, which is a partial differential equation (PDE) that can be used to erase fine-scale information when applied locally to the 2D plane of an image. During training, the model first learns the coarse-scale structure of the data, which is easier to learn, and then gradually learns the finer details. It is therefore interesting to assess whether this form of data mollification is effective in the context of this work compared to the addition of Gaussian noise. Note, however, that under the manifold hypothesis, this type of mollification produces the opposite effect to the addition of Gaussian noise in that at time t = 0 mollified images lie on a 1D manifold and they are gradually transformed to span the dimension of the data manifold; more details on blurring mollification can be found in Appendix B.

Image generation. We evaluate the quality of the generated images using the popular Fr´ echet Inception Distance (FID) score [22] computed on 50K samples from the trained model using the pytorch-fid 1 library. The results, reported in Table 1, indicate that the proposed data mollification consistently improves model performance compared to vanilla training across all datasets and models. Additionally, mollification through blurring, which is in line with recent results from diffusion models [61], is less effective than Gaussian mollification, although it still enhances the vanilla training in most cases. We also show intermediate samples in Fig. 8 illustrating the progression of samples from pure random noise or completely blurred images to high-quality images. Furthermore, we observe that Gaussian mollification leads to faster convergence of the FID score for VAE-based models, as shown in Fig. 7. We provide additional results in Appendix E. As a final experiment, we consider a recent large-scale VAE model for the CIFAR10 data set, which is a deep hierarchical VAE (NVAE) [78]. By applying Gaussian mollification without introducing any additional complexity, e.g., step-size annealing, we improve the FID score from 53 . 64 to 52 . 26 .

0

50

100

150

200

5

.

07

5

.

53

5

.

99

Epoch

VAE - FID [log]

0

20

40

60

80

4

.

61

5

.

76

Epoch

GLOW - FID [log]

Epoch 10

Epoch 20

Epoch 80

Vanilla

Gauss.

Blurring

VAE Choice of noise schedule. We ablate on the choice of noise schedule for Gaussian mollification. Along with the sigmoid schedule, we also consider the linear [25] and cosine [49] schedules, which are also popular for diffusion models. As shown in Table 2, our method consistently outperforms the vanilla approach under all noise schedules. We also observe that the sigmoid schedule consistently produced good results. Therefore, we chose to use the sigmoid schedule in all our experiments.

Comparisons with the two-step approach in [43]. Manifold overfitting in likelihood-based GMs has been recently analyzed in [43], which provides a two-step procedure to mitigate the issue. The first step maps inputs into a low-dimensional space to handle the intrinsic low dimensionality of the data. This step is then followed by likelihood-based density estimation on the resulting lower-dimensional representation. This is achieved by means of a generalized autoencoder, which relies on a certain set of explicit deep GMs, such as VAEs. Here, we compare our proposal with this two-step approach; results are reported in Table 3. To ensure a fair comparison, we use the same network architecture for our VAE and their generalized autoencoder, and we rely on their official implementation 2 . Following [43], we consider a variety of density estimators in the low-dimensional space such as NFs, Autoregressive Models (ARMs) [77], Adversarial Varitional Bayes (AVB) [46] and Energy-Based Models (EBMs) [16]. We observe that Gaussian mollification is better or comparable with these variants. In addition, our method is extremely simple and readily applicable to any likelihood-based GMs without any extra auxilary models or the need to modify training procedures.

4.3 Density Estimation on UCI Data Sets

Wefurther evaluate our proposed method in the context of density estimation tasks using UCI data sets [17], following [68]. We consider three different types of normalizing flows: masked autorgressive flows (MAF) [53], REAL-NVP [14] and GLOW [34]. To ensure a fair comparison, we apply the same experimental configurations for both the vanilla and the proposed method, including random seeds, network architectures, optimizer, and the total number of iterations. Table 1 shows the average log-likelihood (the higher the better) on the test data. Error bars correspond to the standard deviation computed over 4 runs. As it can be seen, our proposed Gaussian mollification approach consistently and significantly outperforms vanilla training across all models and all datasets.

Limitations. One limitation is that this work focuses exclusively on likelihood-based GMs. On image data, the improvements in FID score indicate that the performance boost is generally substantial, but still far from being comparable with state-of-the-art DMs. While this may give an impression of a low impact, we believe that this work is important in pointing to one of the successful aspects characterizing DMs and show how this can be easily integrated in the optimization of likelihood-based GMs. A second limitation is that, in line with the literature on GMs for image data, where models are extremely costly to train and evaluate, we did not provide error bars on the results reported in the tables in the experimental section. Having said that, the improvements reported in the experiments have been shown on a variety of models and on two popular image data sets. Furthermore, the results are supported theretically and experimentally by a large literature on continuation methods for optimization.

Broader impact. This work provides an efficient way to improve a class of GMs. While we focused mostly on images, the proposed method can be applied to other types of data as shown in the density

estimation experiments on the UCI datasets. Like other works in this literature, the proposed method can have both positive (e.g., synthesizing new data automatically or anomaly detection) and negative (e.g., deep fakes) impacts on society depending on the application.

5 Related work

Our work is positioned within the context of improving GMs through the introduction of noise to the data. One popular approach is the use of denoising autoencoders [81], which are trained to reconstruct clean data from noisy samples. Building upon this, [2] proposed a framework for modeling a Markov chain whose stationary distribution approximates the data distribution. In addition, [80] showed a connection between denoising autoencoders and score matching, which is an objective closely related to recent diffusion models [67, 25]. More recently, [45] introduced a two-step approach to improve autoregressive generative models, where a smoothed version of the data is first modeled by adding a fixed level of noise, and then the original data distribution is recovered through an autoregressive denoising model. In a similar vein, [44] recently attempted to use Tweedie's formula [62] as a denosing step, but surprisingly found that it does not improve the performance of NFs and VAEs. Our work is distinct from these approaches in that Gaussian mollification guides the estimated distribution towards the true data distribution in a progressive manner by means of annealing instead of fixing a noise level. Moreover, our approach does not require any explicit denoising step, and it can be applied off-the-shelf to the optimization of any likelihood-based GMs without any modifications.

6 Conclusion

Inspired by the enormous success of score-based Diffusion Models (DMs), in this work we hypothesized that data mollification is partially responsible for their impressive performance in generative modeling tasks. In order to test this hypothesis, we introduced data mollification within the optimization of likelihood-based Generative Models (GMs), focusing in particular on Normalizing Flows (NFs) and Variational Autoencoders (VAEs). Data mollification is extremely easy to implement and it has nice theoretical properties due to its connection with continuation methods in the optimization literature, which are well-known techniques to improve optimization. We applied this idea to challenging generative modeling tasks involving imaging data and relatively large-scale architectures as a means to demonstrate systematic gains in performance in various conditions and input dimensions. We measured performance in quality of generated images through the popular FID score.

While we are far from closing the gap with DMs in achieving competing FID score, we are confident that this work will serve as the basis for future works on performance improvements in state-of-the-art models mixing DMs and likelihood-based GMs, and in alternative forms of mollification to improve optimization of state-of-the-art GMs. For example, it would be interesting to study how to apply data mollification to improve the training of GANs; preliminary investigations show that the strategy proposed here does not offer significant performance improvements, and we believe this is due to the fact that data mollification does not help in smoothing the adversarial objective. Also, while our study shows that the data mollification schedule is not critical, it would be interesting to study whether it is possible to derive optimal mollification schedules, taking inspiration, e.g., from [30]. We believe it would also be interesting to consider mixture of likelihood-based GMs to counter problems due to the union of manifolds hypothesis, whereby the intrinsic dimension changes over the input [3]. Finally, it would be interesting to investigate other data, such as 3D point cloud data [85] and extend this work to deal with other tasks, such as supervised learning.

Acknowledgments and Disclosure of Funding

MF gratefully acknowledges support from the AXA Research Fund and the Agence Nationale de la Recherche (grant ANR-18-CE46-0002 and ANR-19-P3IA-0002).

References

Y. Bengio, A. C. Courville, and P. Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35:1798-1828, 2012.

Y. Bengio, E. Laufer, G. Alain, and J. Yosinski. Deep Generative Stochastic Networks Trainable by Backprop. In Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 226-234, Bejing, China, 22-24 Jun 2014. PMLR.

B. C. Brown, A. L. Caterini, B. L. Ross, J. C. Cresswell, and G. Loaiza-Ganem. Verifying the Union of Manifolds Hypothesis for Image Data. In International Conference on Learning Representations , 2023.

Y. Burda, R. Grosse, and R. Salakhutdinov. Importance Weighted Autoencoders. In International Conference on Learning Representations , 2015.

A. L. Caterini, A. Doucet, and D. Sejdinovic. Hamiltonian Variational Auto-Encoder. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

C. Chadebec, L. Vincent, and S. Allassonniere. Pythae: Unifying Generative Autoencoders in Python - A Benchmarking Use Case. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 21575-21589. Curran Associates, Inc., 2022.

T. Chen. On the Importance of Noise Scheduling for Diffusion Models. arXiv preprint arXiv:2301.10972 , 2023.

T. Chen, G.-H. Liu, and E. Theodorou. Likelihood Training of Schr¨ odinger Bridge using Forward-Backward SDEs Theory. In International Conference on Learning Representations , 2022.

R. Cornish, A. Caterini, G. Deligiannidis, and A. Doucet. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 2133-2143. PMLR, 13-18 Jul 2020.

B. Dai and D. Wipf. Diagnosing and Enhancing VAE Models. In International Conference on Learning Representations , 2019.

G. Daras, M. Delbracio, H. Talebi, A. Dimakis, and P. Milanfar. Soft Diffusion: Score Matching with General Corruptions. Transactions on Machine Learning Research , 2023.

N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders. arXiv preprint arXiv:1611.02648 , 2016.

L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-linear Independent Components Estimation. In International Conference on Learning Representations , 2015.

L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density Estimation Using Real NVP. In International Conference on Learning Representations , 2017.

H. M. Dolatabadi, S. Erfani, and C. Leckie. Invertible Generative Modeling using Linear Rational Splines. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 4236-4246. PMLR, 26-28 Aug 2020.

Y. Du and I. Mordatch. Implicit Generation and Modeling with Energy Based Models. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

D. Dua and C. Graff. UCI Machine Learning Repository, 2017.

C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural Spline Flows. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

A. Genevay, G. Peyr, and M. Cuturi. GAN and VAE from an Optimal Transport Point of View. arXiv preprint arXiv:1706.01807 , 2017.

M. Germain, K. Gregor, I. Murray, and H. Larochelle. MADE: Masked Autoencoder for Distribution Estimation. In Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 881-889, Lille, France, 07-09 Jul 2015. PMLR.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch¨ olkopf, and A. Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research , 13(25):723-773, 2012.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 6626-6637, 2017.

I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In International Conference on Learning Representations , 2017.

J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

E. Hoogeboom, J. Heek, and T. Salimans. simple diffusion: End-to-end Diffusion for High Resolution Images. arXiv preprint arXiv:2301.11093 , 2023.

E. Hoogeboom and T. Salimans. Blurring Diffusion Models. In International Conference on Learning Representations , 2023.

A. Hyv¨ arinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research , 6(24):695-709, 2005.

S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 448-456, Lille, France, 07-09 Jul 2015. PMLR.

H. Iwakiri, Y. Wang, S. Ito, and A. Takeda. Single Loop Gaussian Homotopy Method for Non-convex Optimization. In Advances in Neural Information Processing Systems , volume 35, pages 7065-7076. Curran Associates, Inc., 2022.

A. Jabri, D. Fleet, and T. Chen. Scalable Adaptive Computation for Iterative Generation. arXiv preprint arXiv:2212.11972 , 2022.

D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational Diffusion Models. In Advances in Neural Information Processing Systems , volume 34, pages 21696-21707. Curran Associates, Inc., 2021.

D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations , 2015.

D. P. Kingma and P. Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Improved Variational Inference with Inverse Autoregressive Flow. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations , 2014.

D. P. Kingma and M. Welling. An Introduction to Variational Autoencoders. Foundations and Trends in Machine Learning , 12(4):307-392, 2019.

A. Klushyn, N. Chen, R. Kurle, B. Cseke, and P. van der Smagt. Learning Hierarchical Priors in VAEs. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

I. Kobyzev, S. J. D. Prince, and M. A. Brubaker. Normalizing Flows: An Introduction and Review of Current Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(11):3964-3979, 2021.

A. Krizhevsky and G. Hinton. Learning Multiple Layers of Features from Tiny Images. Master's thesis, Department of Computer Science, University of Toronto , 2009.

Z. Liu, P. Luo, X. Wang, and X. Tang. Deep Learning Face Attributes in the Wild. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 3730-3738. IEEE Computer Society, 2015.

A. Liutkus, U. Simsekli, S. Majewski, A. Durmus, and F.-R. St¨ oter. Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4104-4113. PMLR, 09-15 Jun 2019.

G. Loaiza-Ganem, B. L. Ross, J. C. Cresswell, and A. L. Caterini. Diagnosing and Fixing Manifold Overfitting in Deep Generative Models. Transactions on Machine Learning Research , 2022.

G. Loaiza-Ganem, B. L. Ross, L. Wu, J. P. Cunningham, J. C. Cresswell, and A. L. Caterini. Denoising Deep Generative Models. In I Can't Believe It's Not Better Workshop at NeurIPS 2022 , 2022.

C. Meng, J. Song, Y. Song, S. Zhao, and S. Ermon. Improved Autoregressive Modeling with Distribution Smoothing. In International Conference on Learning Representations , 2021.

L. Mescheder, S. Nowozin, and A. Geiger. Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 2391-2400. PMLR, 06-11 Aug 2017.

H. Mobahi and J. W. Fisher. A theoretical analysis of optimization by gaussian continuation. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence , AAAI'15, page 12051211. AAAI Press, 2015.

H. Narayanan and S. Mitter. Sample Complexity of Testing the Manifold Hypothesis. In Advances in Neural Information Processing Systems , volume 23. Curran Associates, Inc., 2010.

A. Q. Nichol and P. Dhariwal. Improved Denoising Diffusion Probabilistic Models. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8162-8171. PMLR, 18-24 Jul 2021.

D. Onken, S. W. Fung, X. Li, and L. Ruthotto. OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9223-9232, 2021.

A. Ozakin and A. Gray. Submanifold Density Estimation. In Advances in Neural Information Processing Systems , volume 22. Curran Associates, Inc., 2009.

G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research , 22(57):1-64, 2021.

G. Papamakarios, T. Pavlakou, and I. Murray. Masked Autoregressive Flow for Density Estimation. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

M. Pavon, G. Trigila, and E. G. Tabak. The Data-Driven Schr¨ odinger Bridge. Communications on Pure and Applied Mathematics , 74(7):1545-1573, 2021.

X. Pennec. Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements. Journal of Mathematical Imaging and Vision , 25:127-154, 2006.

P. Pope, C. Zhu, A. Abdelkader, M. Goldblum, and T. Goldstein. The Intrinsic Dimension of Images and Its Impact on Learning. In International Conference on Learning Representations , 2021.

D. Reshetova, Y. Bai, X. Wu, and A. ¨ Ozg¨ ur. Understanding Entropic Regularization in GANs. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 825-830. IEEE, 2021.

D. Rezende and S. Mohamed. Variational Inference with Normalizing Flows. In Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.

D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 , volume 32 of JMLR Workshop and Conference Proceedings , pages 1278-1286. JMLR.org, 2014.

S. Rissanen, M. Heinonen, and A. Solin. Generative Modelling with Inverse Heat Dissipation. In International Conference on Learning Representations , 2023.

H. Robbins. An Empirical Bayes Approach to Statistics. In Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probability, 1956 , volume 1, pages 157-163, 1956.

S. T. Roweis and L. K. Saul. Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science , 290(5500):2323-2326, 2000.

A. Salmona, V. De Bortoli, J. Delon, and A. Desolneux. Can Push-forward Generative Models Fit Multimodal Distributions? In Advances in Neural Information Processing Systems , volume 35, pages 10766-10779. Curran Associates, Inc., 2022.

M. Sanjabi, J. Ba, M. Razaviyayn, and J. D. Lee. On the Convergence and Robustness of Training GANs with Regularized Optimal Transport. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

C. K. Sø nderby, T. Raiko, L. Maalø e, S. r. K. Sø nderby, and O. Winther. Ladder Variational Autoencoders. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 22562265, Lille, France, 07-09 Jul 2015. PMLR.

J. Song and S. Ermon. Bridging the Gap Between f-GANs and Wasserstein GANs. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 9078-9087. PMLR, 13-18 Jul 2020.

Y. Song and S. Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations , 2021.

V. Stimper, D. Liu, A. Campbell, V. Berenz, L. Ryll, B. Sch¨ olkopf, and J. M. Hern´ andez-Lobato. normflows: A PyTorch Package for Normalizing Flows. arXiv preprint arXiv:2302.12014 , 2023.

P. Tempczyk, R. Michaluk, L. Garncarek, P. Spurek, J. Tabor, and A. Golinski. LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 21205-21231. PMLR, 17-23 Jul 2022.

J. B. Tenenbaum, V. d. Silva, and J. C. Langford. A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science , 290(5500):2319-2323, 2000.

J. Tomczak and M. Welling. VAE with a VampPrior. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1214-1223. PMLR, 09-11 Apr 2018.

B.-H. Tran, S. Rossi, D. Milios, P. Michiardi, E. V . Bonilla, and M. Filippone. Model Selection for Bayesian Autoencoders. In Advances in Neural Information Processing Systems , volume 34, pages 19730-19742. Curran Associates, Inc., 2021.

B.-H. Tran, B. Shahbaba, S. Mandt, and M. Filippone. Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 34409-34430. PMLR, 23-29 Jul 2023.

B. Uria, I. Murray, and H. Larochelle. RNADE: The Real-valued Neural autoregressive Density-estimator. In Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

A. Vahdat and J. Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. In Advances in Neural Information Processing Systems , volume 33, pages 19667-19679. Curran Associates, Inc., 2020.

R. van den Berg, L. Hasenclever, J. M. Tomczak, and M. Welling. Sylvester Normalizing Flows for Variational Inference. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018 , pages 393-402. AUAI Press, 2018.

P. Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation , 23:1661-1674, 2011.

P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of the 25th International Conference on Machine Learning , pages 1096-1103, 2008.

Z. Wang, H. Zheng, P. He, W. Chen, and M. Zhou. Diffusion-GAN: Training GANs with Diffusion. In International Conference on Learning Representations , 2023.

A. Witkin, D. Terzopoulos, and M. Kass. Signal matching through scale space. In Proceedings of the Fifth AAAI National Conference on Artificial Intelligence , AAAI'86, page 714719. AAAI Press, 1986.

Z. Xiao, K. Kreis, and A. Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In International Conference on Learning Representations , 2022.

G. Yang, X. Huang, Z. Hao, M. Liu, S. J. Belongie, and B. Hariharan. PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pages 4540-4549. IEEE, 2019.

A A Primer on Normalizing Flows and VAEs

Given a dataset D consisting of N i.i.d samples D ∆ = { x i } N i =1 with x i ∈ R D , we aim at estimating the unknown continuous generating distribution p data ( x ) . In order to do so, we introduce a model p θ ( x ) with parameters θ and attempt to estimate θ based on the dataset D . A common approach to estimate θ is to maximize the likelihood of the data, which is equivalent to minimizing the following objective:

Optimization for this objective can be done through a stochastic gradient descent algorithm using minibatches of samples from p data ( x ) .

A.1 Normalizing Flows

In flow-based generative models [52, 39], the generative process is defined as:

where z ∈ R D is a latent variable, and p φ ( z ) is a tractable base distribution with parameters φ , such as an isotropic multivariate Gaussian. The function f ψ : R D → R D is invertible, such that given any input vector x we have z = f -1 ψ ( x ) . A Normalizing Flow (NF) [59] defines a sequence of invertible transformations f = f 1 ◦ f 2 ◦ · · · f K , such that the relationship between x and z can be written as:

where h k = f -1 k ( h k -1 ; ψ k ) and ψ k are the parameters of the transformation f k . For the sake of simplicity, we define h 0 ∆ = x and h K ∆ = z . The likelihood of the model given a datapoint can be computed analytically using the change of variables as follows:

where log | det( ∂ h k /∂ h k -1 ) | is the logarithm of absolute value of the determinant of the Jacobian matrix ∂ h k /∂ h k -1 . This term accounts for the change of measure when going from h k -1 to h k using the transformation f k . The resulting NF model is then characterized by the set of parameters θ = { φ } ∪ { ψ k } K k =1 , which can be estimated using the maximum likelihood estimation (MLE) objective Eq. 4.

Though NFs allow for exact likelihood computation, they require f k to be invertible and to have a tractable inverse and Jacobian determinant. This restricts the flexibility to certain transformations that can be used within NFs [see e.g., 52, 39, and references therein], such as affine coupling [13, 14], invertible convolution [34], spline [18, 15], or inverse autoregressive transformations [35].

A.2 Variational Autoencoders

Variational Autoencoders (VAEs) [36, 59] introduce a low-dimensional latent variable z ∈ R P , with P glyph[lessmuch] D , to the generative process as follows:

Here, p ( z ) is a tractable prior distribution over the latent variables z , and p θ ( x | z ) , which is also known as a decoder, is usually implemented by a flexible neural network parameterized by θ . Different from NFs, VAEs employ a stochastic transformation p θ ( x | z ) to map z to x . Indeed, NFs can be viewed as VAEs where the decoder and encoder are modelled by Dirac deltas p θ ( x | z ) = δ ( f θ ( x ) ) and q φ ( z | x ) = δ ( f -1 θ ( x ) ) respectively, using a restricted family of transformations f θ .

The marginal likelihood of VAEs is intractable and given by:

A variational lower bound on ther marginal likelihood can be obtained by introducing a variational distribution q φ ( z | x ) , with parameters φ , which acts as an approximation to the the unknown posterior p ( z | x ) :

where, L ELBO ( θ , φ ) is known as the ELBO, and the expectation can be approximated by using Monte Carlo samples from q φ ( z | x ) . This objective can be optimized with stochastic optimization w.r.t. parameters θ and φ in place of Eq. 4.

To tighten the gap between the ELBO and the true marginal likelihood, VAEs can be employed with an expressive form of the approximate posterior q φ ( z | x ) such as importance weighted sampling [4] or normalizing flows [35, 79]. In addition, to avoid the over regularization offect induced by the prior p ( z ) , one can utilize a flexible prior such as multi-modal distributions [12, 74], hierarchical forms [66, 38], or simply reweighing the KL divergence term in the ELBO [24].

B Details on Blurring Mollification

Recently, [61, 27, 11] have proposed appproaches to destroy information of images using blurring operations for diffusion-type generative models. Their approach involves stochastically reversing the heat equation, which is a PDE that can be used to erase fine-scale information when applied locally to the 2D plane of an image. In particular, the Laplace PDE for heat diffusions is as follows:

where we consider the initial state of the system to be x , the true image data. This PDE can be effectively solved by employing a diagonal matrix within the frequency domain of the cosine transform, provided that the signal is discretized onto a grid. The solution to this equation at time t can be effectively computed by:

Here, V glyph[latticetop] and V denote the discrete cosine transformation (DCT) and inverse DCT, respectively; the diagonal matrix D t is the exponent of a weighting matrix for frequencies Λ so that D t = exp( Λ t ) . We refer the reader to Appendix A of [61] for the specific definition of Λ . We can evaluate Eq. 13 in the Fourier domain, which is fast to compute, as the DCT and inverse DCT require O ( N log N ) operations. The equivalent form of Eq. 13 in the Fourier domain is as follows:

where u = V glyph[latticetop] x = DCT ( x ) . As Λ is a diagonal matrix, the above Fourier-space model is fast to evaluate. A Python implementation of this blurring mollification is presented in Algorithm 2.

We follow [61] to set the schedule for the blurring mollification. In particular, we use a logarithmic spacing for the time steps t k , where t 0 = σ 2 B, max / 2 and t T = σ 2 B, min / 2 = 0 . 5 2 / 2 , corresponding to sub-pixel-size blurring. Here, σ 2 B, max is the effective lengthscale-scale of blurring at the beginning of the mollification process. Following [61], we set this to half the width of the image.

Algorithm 2: Python code for blurring mollification

C Implementation of Noise Schedules

Algorithm 3 shows the Python code for the noise schedules used in this work. For the sigmoid schedule, following [7], we set the default values of start and end to 0 and 3 , respectively.

Algorithm 3: Python code for noise schedules

D Experimental Details

D.1 Data sets

Synthetic data sets.

Mixture of Gaussians : We consider a mixture of two Gaussians with means µ k = (2 sin( πk ) , 2 cos( πk )) and covariance matrices Σ k = σ 2 I , where σ = 2 3 sin( π/ 2) . We generate 10K samples for training and 10K samples for testing from this distribution.

Von Mises distribution: We use a von Mises distribution with parameters κ = 1 , and then transform to Cartesian coordinates to obtain a distribution on the unit circle in R 2 . We generate 10K training samples and 10K testing from this distribution.

Image data sets. We consider two image data sets including CIFAR10 [40] and CELEBA [41]. These data sets are publicly available and widely used in the literature of generative models. We use the official train/val/test splits for both data sets. The resolution of CIFAR10 is 3 × 32 × 32 . For CELEBA, we pre-process images by first taking a 148 × 148 center crop and then resizing to the 3 × 64 × 64 resolution.

UCI data sets. We consider four data sets in the UCI repository [17]: RED-WINE, WHITE-WINE, PARKINSONS, and MINIBOONE. 10 %of the data is set aside as a test set, and an additional 10 % of the remaining data is used for validation. To standardize the features, we subtract the sample mean from each data point and divide by the sample standard deviation.

D.2 Software and Computational Resources

We use NVIDIA P100 and A100 GPUs for the experiments, with 16GB and 80GB of memory respectively. All models are trained on a single GPU except for the experiments with NVAE model [78], where we employ two A100 GPUs. We use PyTorch [54] for the implementation of the models and the experiments. Our experiments with VAEs and NFs are relied on the pythae [6] and normflows [71] libraries, respectively.

D.3 Training Details

D.3.1 Toy examples.

In the experiments on synthetic data sets, we use a REAL-NVP flow [14] with 5 affine coupling layers consisting of 2 hidden layers of 64 units each. We train the model for 20000 itereations using an Adam optimizer [33] with a learning rate of 5 · 10 -4 and a mini-batch size of 256 .

D.3.2 Imaging experiments.

REAL-NVP. We use the multi-scale architecture with deep convolutional residual networks in the coupling layers as described in [14]. For the CIFAR10 data set, we use 4 residual blocks with 32 hidden feature maps for the first coupling layers with checkerboard masking. For the CELEBA data set, 2 resdiual blocks are employed. We use an Adam optimizer [33] with a learning rate of 10 -3 and a mini-batch size of 64 . We train the model for 100 and 80 epochs on the CIFAR10 and CELEBA data sets, respectively. For the mollification training, we perturb the data for 50 and 40 epochs for CIFAR10 and CELEBA, respectively.

GLOW. We use a multi-scale architecture as described in [34]. The architecture has a depth level of K = 20 , and a number of levels L = 3 . We use the AdaMax [33] optimizer with a learning rate of 3 · 10 -4 and a mini-batch size of 64 . We train the model for 80 and 40 epochs on the CIFAR10 and CELEBA data sets, respectively. For the mollification training, we perturb the data for 50 and 20 epochs for CIFAR10 and CELEBA, respectively.

CIFAR10

CELEBA

ENCODER:

x

∈

R

3

×

32

×

32

→

CONV

(128

,

4

,

2)

→

BN

→

RELU

→

CONV

(256

,

4

,

2)

→

BN

→

RELU

→

CONV

(512

,

4

,

2)

→

BN

→

RELU

→

CONV

(1024

,

4

,

2)

→

BN

→

RELU

→

FLATTEN

→

FC

256

×

2

x

∈

R

3

×

64

×

64

→

CONV

(128

,

4

,

2)

→

BN

→

RELU

→

CONV

(256

,

4

,

2)

→

BN

→

RELU

→

CONV

(512

,

4

,

2)

→

BN

→

RELU

→

CONV

(1024

,

4

,

2)

→

BN

→

RELU

→

FLATTEN

→

FC

256

×

2

DECODER:

z

∈

R

256

→

FC

8

×

8

×

1024

→

CONVT

(512

,

4

,

2)

→

BN

→

RELU

→

CONVT

(256

,

4

,

2)

→

BN

→

RELU

→

CONVT

(3

,

4

,

1)

z

∈

R

256

→

FC

8

×

8

×

1024

→

CONVT

(512

,

5

,

2)

→

BN

→

RELU

→

CONVT

(256

,

5

,

2)

→

BN

→

RELU

→

CONVT

(128

,

5

,

2)

→

BN

→

RELU

→

CONVT

(3

,

4

,

1)

VAEs. We use convolutional networks for both the encoder and decoder of VAEs [36, 60]. Table 5 shows the details of the network architectures. We use an Adam optimizer [33] with a learning rate of 3 · 10 -4 and a mini-batch size of 128 . We train the model for 200 and 100 epochs on the CIFAR10 and CELEBA data sets, respectively. For the mollification training, we perturb the data for 100 and 50 epochs for CIFAR10 and CELEBA, respectively. The addtional details of the variants of VAEs are as follows:

VAE-IAF [35]: We use a 3 -layer MADE [20] with 128 hidden units and RELU activation for each layer and stack 2 blocks of Masked Autoregressive Flow to create the flow for approximating the posterior.

β -VAE [24]: We use a coefficient of β = 0 . 1 for the Kullback-Leibler divergence (KL) term in the ELBO objective.

IWAE [4]: We use a number of importance samples of K = 5 .

HVAE [5]: We set the number of leapfrog steps to used in the integrator to 1 . The leapfrog step size is adaptive with an initial value of 0 . 001

NVAE. We use the default network architecture as described in [78]. We train the model on the CIFAR10 for 300 epochs with an AdaMax optimizer [33] with a learning rate of 10 -3 and a mini-batch size of 200 . For the mollification training, we perturb the data for first 150 epochs.

D.4 UCI experiments

For MAF models [53], we employ 5 autoregressive layers, each composed of a feedforward neural network utilizing masked weight matrices [20]. This neural networks consists of 512 hidden units and employ the tanh activation function. For REAL-NVP [14] and GLOW [34] models, we implement 5 coupling layers, each comprising two feedforward neural networks with 512 hidden units to handle the scaling and shift functions. The first neural network utilizes the tanh activation function, while the latter employs a rectified linear activation function. We introduce batch normalization [29] after each coupling layer in REAL-NVP and after each autoregressive layer in MAF. All models are trained with the Adam optimizer [33] for 150 epochs with a learning rate of 10 -4 and a mini-batch size of 100 . For mollification training, we employ Gaussian mollification during the entire training process.

D.5 Evaluation Metrics

Maximum Mean Discrepancy. The MMD between two distributions p data and p θ is defined as follows [21]:

where H denotes a reproducing kernel Hilbert space (RKHS) induced by a characteristic kernel K . The MMD has the closed form:

which can be estimated by using samples from p data and p θ . In our experiments, we use 10K test samples from the true data distribution p data and 10K samples from the model distribution p θ for estimating the MMD score.

We employ the popular RBF kernel for the MMD, which is defined as follows:

with a lengthscale l = 1 and variance σ 2 = 10 -4 .

FID score. To assess the quality of the generated images, we employed the widely used Fr´ echet Inception Distance [23]. The FID measures the Fr´ echet distance between two multivariate Gaussian distributions, one representing the generated samples and the other representing the real data samples. By comparing their distribution statistics, we can assess the similarity between the generated and real data distributions. The FID score is defined as follows:

The distribution statistics are obtained from the 2048 -dimensional activations of the pool3 layer of an Inception-v3 network. We use the pytorch-fid 3 library for calculating the FID score in our experiments.

E Addtional Results

Table 6 and Table 7 illustrate uncurated samples from the trained models. Fig. 9 and Fig. 10 show the progression of FID scores during training on the CIFAR10 and CELEBA datasets, respectively.

VANILLA

GAUSS. MOLLIFICATION

BLUR. MOLLIFICATION

REAL-NVP

GLOW

VAE

VAE-IAF

IWAE

β

-VAE

HVAE

VANILLA

GAUSS. MOLLIFICATION

BLUR. MOLLIFICATION

REAL-NVP

GLOW

VAE

VAE-IAF

IWAE

β -VAE

HVAE

0

50

100

150

200

5

.

07

5

.

53

5

.

99

Epoch

FID [log] (

)

VAE

0

50

100

150

200

4

.

61

5

.

07

5

.

53

5

.

99

Epoch

FID [log] (

)

β

-VAE

0

50

100

150

200

5

.

07

5

.

53

5

.

99

Epoch

FID [log] (

)

IWAE

0

50

100

150

200

5

.

07

5

.

53

5

.

99

Epoch

FID [log] (

)

VAE-IAF

0

50

100

150

200

5

.

07

5

.

53

5

.

99

Epoch

FID [log] (

)

HVAE

0

20

40

60

80

4

.

61

5

.

76

Epoch

FID [log] (

)

GLOW

20

40

60

80

100

4

.

61

5

.

76

Epoch

FID [log] (

)

VAE

20

40

60

80

100

4

.

61

5

.

76

Epoch

FID [log] (

)

β

-VAE

20

40

60

80

100

4

.

61

5

.

76

Epoch

FID [log] (

)

IWAE

20

40

60

80

100

4

.

61

5

.

76

Epoch

FID [log] (

)

VAE-IAF

20

40

60

80

100

4

.

61

5

.

76

Epoch

FID [log] (

)

HVAE

10

20

30

40

4

.

61

5

.

76

Epoch

FID [log] (

)

GLOW

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the use of data mollification to improve sample quality from likelihood-based generative models. The authors observe that likelihood-based models are worse-performing than score-based models due to manifold overfitting (i.e., learning the manifold but not the distribution on it), and poor density estimation in low-density regions (demonstrated with toy examples). Data mollification is proposed as a solution to these issues and demonstrated visually with toy mathematical constructions. On CIFAR10 and CelebA (64x64), the methodology is shown to improve FID scores and there is a visible qualitative improvement in the model samples. Different choices of mollification processes and annealing schedules are explored and compared.\n\nSTRENGTHS:\n- Main results show significant improvements in FID score on multiple image datasets by introducing a technique that entails no additional computational cost. These hold across different image datasets. Samples are not cherrypicked.\n- Results are positive on two families of generative models, normalizing flows and variational autoencoders.\n- The considered hypotheses suggesting why likelihood-based models suffer from sample quality are backed with theoretical work and experiments with toy examples, where visualization of these phenomena can be confirmed.\n- Various hyperparameters (e.g., the choices of mollification process and schedule) are compared, and the main results are ablated against the same model without the proposed methodology. This gives confidence in their methodology truly demonstrating improvement on the datasets and models considered.\n- The paper is very well written, has clear and easy-to-follow language, and the literature review is more than satisfactory. There are a few issues on presentation (see questions section in my review).\n\nWEAKNESSES:\n- My biggest concern is that the image datasets considered might be too simple to conclude that these techniques will materialize into improvement for larger-scale datasets and models. One potential way to do this is to study, e.g., ImageNet 64x64; usually, positive results there are much more likely to hold in larger scales due to the large scale and difficulty of ImageNet itself. Conversely, results in datasets with very narrow distributions (low entropy) like CIFAR10 and CelebA might not transfer to very diverse, more challenging datasets in practice. Studying the effect of the methodology on more challenging datasets will make the results much more valuable and significant.\n\n- The methodology is not very substantial due to its extreme simplicity, nor is mollification new to the literature as acknowledged by the authors themselves. Arguably, however, applying mollification to the data in a schedule for generative models is not something done in practice for generative models, and contrary to score-based models, it does not require learning all noise levels, so it could be seen as a clever and creative way of applying similar ideas from the literature. Simplicity can be positive, and perception of novelty can be extremely subjective, so I am not too concerned about this. Still, due to how short the contribution is, exploring more models would been very welcome since the paper is generally about likelihood-based generative models.\n\n- Minor: while there is enough visual and empirical evidence, there is little discussion for *why* data mollification alleviates the problems of manifold overfitting and accurate density estimation in low-density regions. Since there is a lot of discussion and even theorems from prior work in the body of the paper, a theoretical lens, or at least hypotheses or discussion around this question feel lacking and would be very welcome.\n\nQUESTIONS:\n- The single most important detail: can the authors confirm that the *targets* of the loss are mollified? I am assuming that this is indeed the case as they mention that mollification == smoothening the loss landscape, but this is not explicitly stated in the main body of the paper, and a reader might interpet the methodology as a certain kind of data augmentation where only the input is degraded via noise/blurring/etc in a schedule. I strongly suggest reflecting this in Algorithm 1.\n- Are models trained with the vanilla approach trained for the same amount of steps as those with data mollification? It might be possible that the models have different convergence speeds because the data distribution changes. It would be nice to show that the strongest baselines are considered in terms of training both approaches with the best possible number of steps.\n- Can we include samples for at least one image dataset in the body of the paper? All metrics are flawed, and visualizing the samples themselves is key to observing what the better numbers actually translate to (also, FID scores are not comparable between datasets, so it is hard to judge what the quantiative improvements entail without looking at samples).\n- Nit: can we drop the use of \"ablation\" for *comparisons* of different schedules, and refer to the vanilla models as an ablation study? Ablation == removing a component.\n\nLIMITATIONS:\nAs mentioned in the weaknesses section, the biggest limitation is that the datasets considered might be too narrow, and there is uncertainty on the methodology being helpful for larger / higher entropy / more diverse datasets . Utilizing larger-scale data essential for larger scale models to attain good results, and it has been the case in many prior work that, e.g., results in CIFAR10 can differ vastly to more realistic image datasets. In my opinion, addressing this successfully should flip this paper to acceptance.\n\nAnother limitation is that, as the authors note, score-based models (which are also likelihood-based generative models-- they can be interpreted as hierarchical VAEs with a fixed approximate posterior) are state-of-the-art image generators but are not considered in experiments. For smaller-scale tasks (e.g. CIFAR10) or the toy examples, this could have been possible without requiring significant computational resources. It is possible that naive gaussian mollification of the data might interfere with a score model's ability to learn the noise level adequately, so there is some uncertainty that it will simply work as intended on this model family which is dominating the field. Moreover, score-based models have an extremely similar coarse-to-fine inductive bias: it is possible that the same benefit provided by data mollification is already provided (albeit at sampling time) and, consequently, the methodology might not improve score-based models at all. Including these results could increase the value of this work significantly if there are positive results, and make (1) the impact of the work more explicit and (2) the exploration thorough and up to date with the field, even if results there are not good. Flows and VAEs are both very relevant to the field and there is a demonstrated contribution in these; more than one model family, and the existing findings are still potentially valuable to the community.\n\nAs noted by the authors, experiments also are limited to mathematical constructions and to image datasets. Focusing a paper on the image modality is reasonable, so I am not concerned about this. Addressing the other main limitations in my opinion will increase the value of the contributions sufficiently for this paper to merit acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address the limitation of likelihood-based Generative Models (GMs) in achieving high-quality samples compared to state-of-the-art score-based Diffusion Models (DMs). They propose a novel approach to enhance the generation quality of GMs by incorporating data mollification, a technique used in score-based DMs for accurate density estimation in low-density regions and to mitigate manifold overfitting.\n\nThe authors connect data mollification with Gaussian homotopy, a well-known optimization technique, by adding Gaussian noise to the data. This simple addition can be implemented with just one line of code in the optimization loop of likelihood-based GMs. Through experiments on popular image datasets using various likelihood-based GMs, such as variational autoencoders and normalizing flows, the authors demonstrate improvements in generation quality, as measured by the FID score.\n\nNotably, this approach provides a boost in generation quality without incurring additional computational overheads. This work could potentially be a step towards bridging the gap between likelihood-based GMs and score-based DMs, offering new possibilities for generating high-quality samples in various domains.\n\nSTRENGTHS:\nThe approach presented is both straightforward and seamlessly integrable into the training process of various generative models. The authors provide lucid instructions and accompanying pseudo code for its implementation. Notably, the results displayed in table 1 exhibit a remarkable enhancement compared to the baseline performance.\n\nWEAKNESSES:\nThe approach described in the paper lacks explicit discussion regarding its relationship to noise augmentation. However, towards the end of section 3, it appears that the authors have chosen a temperature factor of 0.7, which leads to a substantial mollification towards the end of the training process. Additionally, their results are comparable to what is typically observed with conventional data augmentation techniques.\n\nGiven that the evaluation of the presented method in this paper is entirely based on empirical analysis, it is crucial for the authors to empirically demonstrate that their approach surpasses alternative methods. Replacing the sigmoid noise schedule with a uniform/constant function would be the equivalent to simple data augmentation with Gaussian noise. Since data augmentation using Gaussian noise is a common practice in the training of many generative models, it becomes essential to provide empirical evidence showcasing that the proposed noise schedule yields results that go beyond what can be achieved with a uniform noise schedule. The authors should provide results from experiments with different amounts of noise and show that their noise schedule outperforms all of these.\n\nThe authors present little theoretical evidence for the choice of noise function. They justify their choice of the sigmoid function by referencing Jabri et al., who demonstrated improved stability with the sigmoid function compared to the cosine function through empirical results. However, the empirical findings displayed in table 2 contradict this justification, indicating that a cosine noise schedule performs better than the sigmoid noise schedule in all cases except GLOW.\n\nQUESTIONS:\n1. What is the relationship between the described approach and noise augmentation?\n2. How does applying different amounts of gaussian noise continuously (uniform noise schedule) to the data compare to the sigmoid function?\n3. How does the presented method outperform alternative approaches that address manifold overfitting, excluding data augmentation?\n4. Why was the sigmoid function selected over the cosine function given the results shown in table 2?\nIt is not clear if the procedure mentioned on line 258 is applied to all training or only certain models or dataset. What is the role of this on the presented FID numbers in table 1?\n\nLIMITATIONS:\n1. Lack of explicit discussion on the relationship to noise augmentation: The paper does not provide a thorough examination or explanation of how the approach relates to noise augmentation. While a temperature factor of 0.7 is mentioned, there is no detailed discussion on the significance of this factor or its impact.\n2. Exclucive reliance on empirical analysis: The evaluation of the presented method relies solely on empirical analysis. To establish the superiority of their approach, the authors should conduct comparative experiments with alternative (much simpler) methods and demonstrate that their approach consistently outperforms them. Specifically, they should compare their noise schedule against different amounts of constant noise and show its superiority over a set of variations.\n3. Limited theoretical justification for the choice of noise function: The paper lacks substantial theoretical evidence supporting the selection of the sigmoid noise function. Although the authors refer to Jabri et al.'s empirical results to justify their choice, the findings presented in table 2 contradict this justification by showing that a cosine noise schedule performs better in all cases except GLOW.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an optimization method that improves the training of likelihood-based generative models. The central idea is that in the early phases of the optimization, the learner tries to model a smoothed version of the likelihood function. As the optimization proceeds, the smoothening decreases (and the difficulty of the optimization problem increases) with the end goal of learning the likelihood of the true distribution. The authors show that this simple idea (inspired by the diffusion models literature) gives consistent improvements in the performance of likelihood-based generative models.\n\nSTRENGTHS:\nThe authors borrow an idea from the diffusion literature (the mollification of the data distribution at many levels) and they apply it in an interesting way to likelihood-based models such as VAEs and Normalizing Flows. Interestingly, the modeling target changes during the optimization (starting from easy distributions and slowly going to harder ones). Instead, diffusion models try to model all the distributions simultaneously. As far as I know, this method has not been proposed in previous works and it seems to be effective in improving likelihood-based models.\n\nThe paper is clearly written and the experimental methodology seems valid. \n\nThe authors showcase the effectiveness of their method starting from informative toy examples and moving to real datasets (CIFAR-10 and CelebA). The boost in performance is consistent across the experiments.\n\nWEAKNESSES:\nEven though the method increases the effectiveness of likelihood-based models, the performance is still very weak. The authors acknowledge this limitation. This is not necessarily an important weakness since future work might further improve the performance of likelihood-based models to the point they become competitive with diffusion models. However, I do not see how this particular optimization method could be improved to achieve these results.\n\nI do not see why the noise schedule has to be similar to the noise schedule diffusion models use. In the design process of a diffusion model, we typically select the corruption scheduling such that the difficulty of transitioning from one corruption level to the next one during sampling is roughly the same across all corruption levels. Why does this intuition transfer here on how we select the scheduling?\n\nI am also puzzled why there is no catastrophic forgetting happening. The authors mention that for the last 50% of the optimization, they stop using their method of mollification and they train for the true distribution.\n\nQUESTIONS:\nI would love to hear what the authors think about directions for future work to improve the method and further enhance the performance of likelihood-based models. \n\nRelated question, do the authors see any way to trade sampling speed for performance using some modification of their method?\n\nFor the scheduling, would it be possible to move to the next level once the loss flattens? This seems to me a reasonable thing to do in order to move from easier optimization problems to hard ones, using the solution of the previous problem as a warm start. I have seen this trick in the Intermediate Layer Optimization work in which the goal is again to move gradually from easier optimization problems to harder ones.\n\nI think I missed the point about blurring in the paper. Intuitively, it seems to me that both blurring and the addition of noise have the same goal of making the distribution \"easier\". Diffusion models are based on the idea of sampling from hard distributions starting with a warm start from an easier distribution. Is it possible that some other scheduling of the blurring kernel would lead to better results? How about a combination of blurring and noise as in Soft Diffusion?\n\nLIMITATIONS:\nIn my opinion, the main limitations of the paper are: i) the poor understanding of why catastrophic forgetting is not happening and how to select the scheduling of the corruption, ii) the absence of a way to trade sampling speed with quality. For ii) the FID numbers right now are very weak. This is not necessarily a limitation specific only to this work, but a limitation of one-step sampling models in general.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a idea to borrow one of the strengths of score-based DMs, which is the ability to perform accurate density estimation in low-density regions and to address manifold overfitting by means of data mollification. We connect data mollification through the addition of Gaussian noise to Gaussian homotopy, which is a well-known technique to improve optimization. Data mollification can be implemented by adding one line of code in the optimization loop, and the authors demonstrate that this provides a boost in generation quality of likelihood-based GMs, without computational overheads.\n\nSTRENGTHS:\nThe paper presents a fairly easy to use data mollification techniques to help vae and flow model to generate better samples, especially for low data density region.\n\nWEAKNESSES:\n1.The first concern of this paper is its novelty and it is very critical for conference like Neurips. It basically states the same thing as in \"Generative modeling by estimating gradients of the data distribution\", a NeurIPS 2019 oral paper. Due to the identical proposition and analysis of generative model, the novel element in this paper is very limited. The difference is instead of inference the score, the paper just let the generative networks to train under the mollified data.\n\n2. The description of the model is vague, except for algorithm1, there is no clear presentation of how to use the model. Although in line 235, it says \"This process uses the solution from one level of mollification as a means to guiding optimization for the next.\", do the authors suggest, in a generative task (lets say image generation) a sample is generated from white noise by following the gradient of each gaussian level? The connection between langevine dynamics, SDE, diffusion models are very strong to this kind of sampling.\n\nQUESTIONS:\n1. What's the difference in terms of motivation between the famous \"Generative modeling by estimating gradients of the data distribution\" paper and this one?\n\n2. What differentiates this paper from SDE score-matching and langevine dynamics generative models?\n\n3. How to combine this model with VAE? Any clear description or examples?\n\nLIMITATIONS:\n1. The model as described, seems to make the training process very heavy and slow, similar to diffusion models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}