One-Line-of-Code Data Mollification Improves Optimization of Likelihood-based Generative Models
Authors: Ba-Hien Tran
Giulio Franzese
Pietro Michiardi
Maurizio Filippone
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Models
Normalizing Flows
Variational Autoencoders

Abstract: Generative Models (GMs) have attracted considerable attention due to their tremendous success in various domains, such as computer vision where they are capable to generate impressive realistic-looking images. Likelihood-based GMs are attractive due to the possibility to generate new data by a single model evaluation. However, they typically achieve lower sample quality compared to state-of-the-art score-based Diffusion Models (DMs). This paper provides a significant step in the direction of addressing this limitation. The idea is to borrow one of the strengths of score-based DMs, which is the ability to perform accurate density estimation in low-density regions and to address manifold overfitting by means of data mollification. We propose a view of data mollification within likelihood-based GMs as a continuation method, whereby the optimization objective smoothly transitions from simple-to-optimize to the original target. Crucially, data mollification can be implemented by adding one line of code in the optimization loop, and we demonstrate that this provides a boost in generation quality of likelihood-based GMs, without computational overheads. We report results on real-world image data sets and UCI benchmarks with popular likelihood-based GMs, including variants of variational autoencoders and normalizing flows, showing large improvements in FID score and density estimation.

One-Line-of-Code Data Mollification Improves Optimization of Likelihood-based Generative Models
Ba-Hien Tran Department of Data Science EURECOM, France
Giulio Franzese Department of Data Science
EURECOM, France
ba-hien. tran@eurecom. fr
giulio fwanzesedeurecoo..c...m fr
Pietro Michiardi Department of Data Science
Maurizio Filippone
Department Data Science
EURECOM, France
EURECOM, France maurizio. maurizio. htt.pioo..i..pppeeeeecccoo.....
pietro. pnstro.michiaee
fr
Abstract
Generative Models (GMs) have attracted considerable attention due to their tremen- dous success various domains, such as computer vision where they are capable GMS
generate impressive realistic-looking images. Likelihood--ol tractive due to the possibility to generate new data by single model evaluation. However, they typically achieve lower sample quality compared to waaeeoff-the-ar score-based Diffusion Models (DMS). This paper provides a significant step in the direction of addressing this limitation. The idea is to borrow one of the strengths of basee DMS, which is the ability to perform accurate density estimation in low-density regions and address manifold overfitting by means of data mollifi- cation. We propose view of data mollification within likelihood- weeebeooddbbaed GMs continuation method, whereby the optimization objective smoothly transitions from simple-to-optmizze to the original target. Crucially, data mollification can be implemented by adding one line of code in the optimization loop, and we demon- strate that this provides boost in generation quality of ikkelihood-based GMs, without computational overheads. We report results on real-world image data sets and UCI benchmarks with popular likelihood-baat GMs, including variants of variational autoencoders and normalizing flows, showing large improvements FID score and density estimation.
Introduction
Generative Models (GMS) have attracted considerable attention recently due to their tremendous success in various domains, such computer vision, graph generation, physics and reinforcement learning [see 37, 39, 75, 76, and references therein]. Given set of data points, GMS attempt to characterize the distribution of such data so that it is then possible to draw new samples from this. Popular approaches include Variational Autoencoders (VADs), Normalizing Flows (NTS), Generative Adversarial Networks (GANs), and score- score-based Diffusion Models (DMs).
In general, the goal of any GMS is similar to that of density estimation with the additional aim to do so by constructing parametric mapping between easy-to-sample-fron distribution p. and the desired data distribution Pdata- While different GMS approaches greatly differ in their optimization strategy and formulation, the underlying objectives share some similarity due their relation to the optimal transport problem, defined as arg min, f ||x d belong to the set of joint distributions with marginals Pdata- respectively [19, 42]. This unified
dx y). Here æ is constrained to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

perspective is explicitly investigated for GANS and VAES [19] for example, whereas other works study NES [50]. Similarly, DMS can be connected Schrodinger Bridges [8], which solve the problem of regulaz-re optimal transport [55]. Given that extensions of the regularized optimal transport case are available also for other generative models [65, 58], we should expect that, in principle, any technique should allow generation of samples with similar quality, provided it is properly tuned. However, this is not true in practice, The different formulations lead to a variety of properties associated with GMS, and pros and cons of each formulation can be understood through the so-called GM tri-lemma [84]. The three desirable properties of GMS are high sample quality, mode coverage, and fast sampling, and it has been argued that such goals are difficult to be satisfied simultaneously [84].
The -fate-of-the-ar is currently dominated by score-based DMS, due to their ability to achieve high sample quality and good mode coverage. However, generating new samples is computationally expensive due to the need to simulate stochastic differential equations. Likelihood- based GMs are complementary, in that they achieve lower sample quality, but sampling requires one model evaluation per sample and it is therefore extremely fast. While some attempts have been made to bridge the gap by combining GANS with DMs [84] or training GANS with diffusions [82], these still require careful engineering of architectures and training schedules. The observation that all GMs share common underlying objective indicates that we should look at what makes DMS successful at optimizing their objective. Then, the question we address in this paper is: can we borrow the strengths of score-based DMS to improve likelihood- GMS, without paying the price of costly sample generation?
One distinctive element of score-based DMS is data mollification, which is typically achieved by adding Gaussian noise [69] or, the context of image data sets, by blurring [61]. A large body of evidence points to the manifold hypothesis [63], which states that the intrinsic dimensionality of image data sets is much lower than the dimensionality of their input. Density estimation in this context is particularly difficult because of the degeneracy of the likelihood for any density concentrated on the manifold where data lies [43]. Under the manifold hypothesis, or even when the target density is multi-modal, the Lipschitz constant of GMs has to be large, but regularization, which is necessary for robustness, is antagonist to this objective [64, 9]. As we will study in detail in this paper, the process of data mollification gracefully guides the optimization mitigating manifold overfitting and enabling a desirable tail behavior, yielding accurate density estimation in low-density regions. In likelihood-based GMS, data mollification corresponds to some form of simplification of the optimization objective. This type of approach, where the level of data mollification is annealed throughout training, can be seen as continuation method [83, 47], which is popular technique in the optimization literature to reach better optima.
Strictly speaking, data mollification in score-based DMs and likelihood based GMS are slightly different. In the latter, the amount of noise injected in the data is continuously annealed throughout training. At the beginning, the equivalent loss landscape seen by the optimizer is much smoother, due to the heavy perturbation of the data, and a continuous reduction of the noise level allows optimization to be gracefully guided until the point where the level of noise is zero [83, 47J. DMs, instead, are trained at each step of the optimization process by considering all noise levels simultaneously, where complex amortization procedures, such self-attention [70], allow the model to efficiently share parameters across different perturbation levels. is also worth mentioning that score- score-based DMS possess another distinctive feature in that they perform gradient-based density estimation [69, 28]. It has been conjectured that this can be helpful to avoid manifold overfitting by allowing for the modeling of complex densities while keeping the Lipschitz constant of score networks low [64]. In this work, we attempt to verify the hypothesis that data mollification is heavily responsible for the success of score-based DMs. We do so by proposing data mollification for Eikelihood-basee GMs, and provide theoretical arguments and experimental evidence that data mollification consistently improves their optimization. Crucially, this strategy yields better sample quality and it is extremely easy to implement, as it requires adding very little code to any existing optimization loop.
We consider a large set of experiments involving VAES and NFs and some popular image data sets. These provide a challenging test for likelihood-based GMS due to the large dimensionality of the input space and to the fact that density estimation needs deal with data lying on manifolds. The results show systematic, and in some cases dramatic, improvements in sample quality, indicating that this is simple and effective strategy improve optimization of likelihood-based GMS models. The paper is organized as follows: in $ 2 we illustrate the challenges associated with generative modeling when data points lie on a manifold, particularly with density estimation in low-density regions and
2

---

manifold overfitting; in 3 we propose data mollification to address these challenges; & 4 reports the experiments with discussion of the limitations and the broader impact, while ğ presents related works, and 3 6 concludes the paper.
2 Challenges in Training Deep Generative Models
We are interested in unsupervised learning, and in particular on the task of density estimation. Given a dataset D consisting of N i..d samples D _ {x, } with x; R?, we aim to estimate the unknown continuous generating distribution Pdata (x). In order to do so, we introduce a model pe(x) with parameters 0 and attempt to estimate 0 based the dataset D. A common approach to estimate O is to maximize the likelihood of the data, which is equivalent to minimizing the following objective:
( - (x)) [log po
There are several approaches to parameterize the generative model pe (x). In this work, we focus on two widely used likelihood-basse Generative Models (GMs), which are Normalizing Flows (NFs) [52, 39] and Variational Autoencoders (VAES) [36, 59]. Although NFS and VAES are among the most popular deep GMS, they are characterized by a lower sample quality compared to GANS and score-based DMs. In this section, we present two major reasons behind this issue by relying on the manifold hypothesis.
2.1
The Manifold Hypothesis and Density Estimation in Low-Density Regions
The manifold hypothesis is a fundamental concept in manifold learning [63, 73, 1] stating that real-world high-dimensional data tend to lie on manifold .M characterized by much lower dimensionality compared to the one of the input space (ambient dimensionality) [48]. This has been verified theoretically and empirically for many applications and datasets [51, 48, 57, 72]. For example, [57] report extensive evidence that natural image datasets have indeed very low intrinsic dimension relative to the high number of pixels in the images.
The manifold hypothesis suggests that density estima- tion in the input space is challenging and ill- ill-posed. In particular, data points on the manifold should be associ- ated with high density, while points outside the manifold should be considered as lying in regions of nearly zero density [45]. This implies that the target density in the input space should be characterized by high Lipschitz constants. The fact that data is searce in regions of low density makes it difficult to expect that models can yield accurate density estimation around the tails. These pose significant challenges for the training of deep GMs [9, 45, 69]. Recently, diffusion models [69, 25, 70] have demonstrated the ability to mitigate this problem by gradually transforming a Gaussian distribution, whose support spans the full input space, into the data distribu- tion. This observation induces us to hypothesize that the data mollification mechanism in score-based DMs is responsible for superior density estimation in low-density regions.
Data distribution
Estimated distribution
Figure 1: Left: Histogram of samples from data distribution Peau and its true scores Vx log Pem (x); Right: Histogram of of sam- ples from the estimated distribution po (x) and its scores Vx log Po (x). In the low density re- gions, the model is unable to capture the true density and scores.
To demonstrate the challenges associated with accurate estimation in low-density regions, we consider a toy experiment where we use a REAL-NVP flow [14] to model two-dimensional mixture of Gaussians, which is a difficult test for NFS in general. Details on this experiment are provided in the Appendix D. Fig. 1 depicts the true and estimated densities, and their corresponding scores, which are the gradient of the log-density function with respect to the data [28]. Note that the use of "score" here is slightly different from that from traditional statisties where score usually refers to the gradient of the log- likelioel with respect to model parameters. As it can be seen in the figure, in regions of low data density, po (x) is completely unable to model the true density and scores. This problem due to the lack of data samples in these regions and may be more problematic under the manifold hypothesis and for dimenssinnii data such as images. & 4, we will demonstrate how possible to considerably mitigate this issuc by means of data mollification.
3

---

(g) 2.2 Manifold Overfitting
The manifold hypothesis suggests that overfitting on a manifold can occur when the model po assigns an arbitrarily large likelihood in the vicinity of the manifold, even if the distribution does
not accurately capture the true distribution Piuta (x) [10, 43]. This issue is illustrated in Fig. 2 of [43] and it will be highlighted in our experiment (5 4.1), where the true data distribution Pasta (x) supported on one-dimensional curve manifold M in two-dimensional space TR? Even when the model distribution (x()) poorly approximates Piata (x), it may reach high likelihood value by concentrating the density around the correct manifold M. If pe (x) is flexible enough, any density defined on MM may achieve infinite likelihood and this might be obstacle for retrieving Pdata (x).
theoretical formalization of the problem of manifold overfitting appears in [43] and is based on the concept of Riemannian measure [56]. The Riemannian measure on manifolds holds an analogous role to that of the Lebesgue measure on Euclidean spaces. To begin, we establish the concept of smoothness for probability measure on manifold.
Definition 1. Let M be a dimee--imeesional manifold, and p be a probability measure on M. Let g (g) be a Riemannian metric on M and PM the corresponding Riemannian measure. We say that is smooth p < M
(0) and it admits a continuous density p : .M -> T&yo with respect to J'M
We now report Theorem from [43] followed by a discussion on its implications for our work. Loaiza-Ganem embedded submanifold of IR with d < D, uD is the Lebesgue measure on Rd, and p a smooth (1)
Theorem 1. (Gabriel Loaiza- Ganem et al. [43]). Let M C RD be an analytic d-dimensional
probability measure on M. Then there exists a sequence of probability measures {Pa ) on RDD such that:
1. Pg (1)
p² as 1
2. Vt 2 0.P0 (t) < PD and pe admits a density Po (t)
.R.. + Ry0 with respect to j&D such
that:
(a) c Po (t) (x) = cc, Vx E M.
(b) lim Pe (x) O. Vx & cl(M), where el(-) denotes closure in R.P
(1)
Theorem I holds for any smooth probability measure supported in M. This is an important point because this includes the desired Pdata provided that this is smooth too. The key message in [43] is that, a-priori, there is no reason to expect that for a likelihood-basee model to converge to Panta out of all the possible p!. Their proof is based on convolving pl with Gaussian kernel with variance of that decreases to zero as & -> co, and then verify that the stated properties of Po hold. Our analysis, while relying on the same technical tools, is instead constructive in explaining why the proposed data mollification allows us to avoid manifold overfitting. The idea is as follows: at time step 0, we select the desired Pdata convolved with Gaussian kernel with a large, but finite, variance (0) as the target distribution for the optimization. Optimization is performed and po targets this distribution, without any manifold overfitting issues, since the dimensionality of the corrupted data non-degenerate. At the second step, the target distribution is obtained by convolving Pdata with the kernel with variance o² (I) o²(0), and again manifold overlitting is avoided. By iteratively repeating this procedure, we can reach the point where we are matching a distribution ved with an arbitrarily small variance (t), without ever experiencing manifold overfitting. When removing the last bit of perturbation we fall back to the case where we experience manifold overfitting. However, when we operate in stochastic setting, which is the typical scenario for the GMS considered here, we avoid ending up in solutions for which the density is degenerate and with support which is exactly the data manifold. Another way to avoid instabilities is to adopt gradient clipping. Note that, as mentioned in [43] and verified by ourselves in earlier investigations, a small constant amount of noise does not provide any partieular benefits over the original scheme, whereas gradually reducing the level of data mollification improves optimization dramatically.
2.3 Data Mollification as a Continuation Method
We can view the proposed data mollification approach continuation method [83, 47]. Starting from the target objective function, which in our case L(0) in Eq. (or a lower bound in the case of VAES), construct a family of functions H(,, 7) parameterized by an auxiliary variable

---

E [0. 1] so that 0)) = (()). The objective functions 7((0,) are defined so that the higher the easier is to perform optimization. In our case, when ? 1 we operate under a simple regime where we target a Gaussian distribution, and likelihood-based GMs can model these rather easily. By annealing from 1 to with given schedule, the sequence of optimization problems with objective H((,)) is increasinly more complex to the point where we target ((0). In essence, the proposed data mollification approach can seen as good initialization method, as the annealing procedure introduces memory effect in the optimization process, which is beneficial in order to obtain better optima.
Generative Models with Data Mollification
Motivated by the aforementioned problems with density estimation in low-density regions and manifold overfitting, we propose simple yet effective approach to improve likelihood-basee GMs. Our method involves mollifying data using Gaussian noise, gradually reducing its variance, until recovering the original data distribution Pamaa (x). This mollification procedure is similar to the reverse process diffusion models, where prior noise distribution is smoothly transformed into the data distribution [69, 25, 70J. As already mentioned, data mollification alleviates the problem of manifold overfitting and it induces a memory effect in the optimization which improves density estimation in regions of low density.
X
XT
Algorithm I: Gaussian mollification
for tort- +- 1,2, I.......' do
X Phaal (x) // Sample training data
X, dre // Mollify data with or. (t/T) and (0,
Truining Iteration
Illustration of Gaussian mollification. where &, is the mollified data at iteration I. 0₂ O. // Train model
Figure 2:
Gaussian Mollification. Given that train the model (x for T iterations, we can create sequence of progressively less smoothed versions of the original data x, which we refer to as mollified data X, Here, ranges from 0 (the most mollified) to (the least mollified). For any t [0, T), the distribution of the mollified data X,, conditioned on is given as follows:
(xc x) = (Kx axx, of1).
(2)
where ar and of are are positive scalar-valued functions of t. In addition, we define the signal-to- noise ratio (NR( aj /o/ and we assume that it monotonically increases with i.e., SNR(1) < SNR(L+1) for all / E [0, - 1]. In other words, the mollified data x, is progressively less smoothed as increases. In this work, we adopt the variance- prrseric- formulation used for diffusion models [67, 25, 32], where 0; /1 of and of = (t/T). Here, 7(-) is a monotonically decreasing function from 1 to 0 that controls the rate of mollification. Intuitively, this procedure involves gradually transforming an identity-covariance Gaussian distribution into the distribution of the data. Algorithm I summarizes the proposed Gaussian mollification procedure, where the red line indicates a simple additional step required to mollify data compared with vanilla training using the true data distribution.
Noise schedule. The choice of the noise schedule 7(-) has an impact on the performance of the final model. In this work, we follow common practice in designing the noise schedule based on literature of acore-based DMs [49, 26, 7]. In particular, we adopt sigmoid schedule [31] which has recently been shown be more effective practice compared to other choices such linear [25] or cosine schedules [49].
5

---

Turget
Estimated distribution
Iteration
MMD2 [log]
4.61
2.3
Training loss [log]
Truc scores
1.38
1.15-
5000 10000 15000 20000
lteration
Figure 4: The first column shows the target distribution and the true scores. The second column depicts the estimated distributions of the Gaussian Mixture Model (GMM), which yield MMD² 15.5 and 2.5 for the vanilla (top) and mollification (bottom) training, respectively. The remaining columns show histogram of samples from the true (top row) and mollified data (bottom row), and estimated scores.
Vanilla
Mollfication
Figure The learning curves the GMM exper- iments.
The sigmoid schedule r(t/T) [31] is defined through the sigmoid function:
Toge logi (SNN) R)
t/T
sigmoid
% 0.5
where is temperature hyper-parameter, This func- tion is then scaled and shifted to ensure that 7(0) = and (1) = O. We encourage the reader to check the implementation of this schedule, available in Ap- pendix C. Fig. illustrates the sigmoid schedule and the corresponding log(SNR) with different values of T. We default temperature of 0.7 it demon- strates consistently good results in our experiments.
0.5
0.5 t/T
Figure 3: Illustration sigmoid schedule and corresponding log( (SNR.. The temperature values from 0.2 to 0.9 are progressively shaded, with the lighter shade corresponding to lower temperatures.
4 Experiments
In this section, we demonstrate emprically the effectiveness of our proposal through wide range of experiments on synthetic data, and some popular real-world tabular and image data sets. Appendix contains detailed description of each experiment to guarantee reproducibility.
4.1 2D Synthetic Data Sets
We begin our experimental campaign with two 2D synthetic data sets. The two-dimensional nature of these data sets allows to demonstrate the effectiveness of Gaussian mollification in mitigating the challenges associated with density estimation in low-density regions and manifold overfitting. Here, consider po (x) to be REAL-NVP flow [14], which comprises five coupling bijections, each consisting of two-hidden layer multilayer perceptron (MLP). To assess the capability of use Maximum Mean Discrepancy (MMD) [21] with
(x) to recover the true data distribution,
radial basis function (RBF) kernel on a held-out set. In these experiments, we employ the Gaussian mollification strategy presented in the previous section and compare the estimated density with the vanilla approach where we use the original training data without any mollification.
Mixture of Gaussians. First, consider target distribution that mixture of Gaussians, as depicted in Fig. 4. As discussed in 8 2.1, the vanilla training procedure fails to accurately estimate the true data distribution and scores, particularly the low-density regions. The estimated densities and the mollified data during the training are depicted in Fig. 4. Initially, the mollification process the target density, which easy model. This demonstrated by the low training loss at the beginning of the optimization, as depicted in Fig. 5. Subsequently, the method gradually reduces the level of noise allowing for progressive refinement of the estimated versions of the target density. This process uses the solution from one level of
considers simpler coarse-grained version

---

mollification means to guiding optimization for the next. As a result, Gaussian mollification facilitates the recovery of the modes and enables effective density estimation in low-density regions. The vanilla training procedure, instead, produces a poor estimate of the target density, as evidenced by the trace-plot of the MMD metric in Fig. and the visualization of the scores in Fig. 4.
Von Mises distribution. We proceed with an investigation of the von Mises distribution the unit circle, as depicted in Fig. 6, with the aim of highlighting the issue of manifold overfitting [43]. In this experiment, the data lies a one-dimensional manifold embedded in a two-dimensional space. The vanilla training procedure fails to approximate the target density effectively, as evidenced by the qualitative results and the substantially high value of MMD (= 383.44) shown in Fig. 6. In contrast, Gaussian mollification gradually guides the estimated density towards the target, as depicted in Fig. leading to significantly lower MMD (== 6.13). Additionally, the mollification approach enables the estimated model not only to accurately learn the manifold but also to capture the mode of the density correctly.
4.2 Image Experiments
Table 1: FID score on CIFAR 10 and CELEBA dataset (lower is better). The small colored numbers indicate improvement or degration of the mollification training compared to the vanilla training.
MODEL
CIFARIO
CELEBA
GAUSSIAN MOLLIPCATION
BLURRING MOLLIFICATION
GAUSSIAN MOLLIFICATION
BLURRING MOLLIFICATOIN
VANILLA
VANILLA
REAL-XVP [14]
131.15
121.5
77.9%
120.88 J 7.855
81.25
79.68
1.98
85.40
5.
GLOW [34]
74.62
64.87
13.07%
66.70
10.018
97.59
70.91
4 27.334
74.74
23.419
[3][36]
191.98
155.13
175.40
8.644
80.19
72.97
1.0.0%
77.29
5.62%
VAE-IAF [35]
193.58
156.39
19.21%
162.27
16.177%
80.34
73.56
8.44%
75.67
5.818
TWAE [4]
183.04
146.70
19.85%
163.79
10.522%
78.25
71.38
8.768
76.45
2,300
Ø-VAE [24]
112.42
93.90
16.47%
101.30
9.89%
67.78
64.59
4.71%
67.08
J. 1.08%
HVAE [5]
172.47
137. 84
20. 081
147.15
14.08%
74.10
72.28
2.45
77.54
4.6.4
Setup. We evaluate method image generation tasks CIFAR 10 [40] and CELEBA 64 [41] datasets, using diverse set of likelihood-basee GMs. The evaluated models include the vanilla VAB [36], the 3-VAE [24], and the VAE-IAF [35] which employs an expressive inverse autoregressive flow for the approximate posterior. To further obtain flexible approximations of the posterior of latent variables well tight evidence lower bound (ELBO), also select the Hamiltonian-Val VAE (HVAD) [5] and the importance weighted VAE (IWAE) [4]. For flow-based models, we consider the REAL-NVP [14] and GLOW [34] models in our benchmark. We found that further training the model on the original data after the mollification procedure leads to better performance. Hence, in our approach we apply data mollification during the first half of the optimization phase, and we continue optimize the model using the original data in the second half. Nevertheless, to ensure fair comparison, we adopt identical settings for the vanilla and the proposed approaches, including random seed, optimizer, and the total number of iterations.
t 100
500
1000
1200
18000
20000
Target distribution
Figure 6: The progression of the estimated densities for the von Mises distribution from the vanilla (bottom row) and our mollification (top row) approaches.
7

---

Blurring mollification. Even though Gaussian mollification is motivated by the manifold hypoth- esis, it is not the only way mollify the data. Indeed, Gaussian mollification does not take into account certain inductive biases that are inherent in natural images, including their multi-scale nature. Recently, [61, 27, 11] have proposed methods that incorporate these biases diffusion-type genera- tive models. Their approach involves stochastically reversing the heat equation, which is partial differential equation (PDE) that can be used to erase fine-scale information when applied locally to the 2D plane of image. During training, the model first learns the coarse-scale structure of the data, which easier to learn, and then gradually learns the finer details. It is therefore interesting to assess whether this form of data mollification is effective in the context of this work compared to the addition of Gaussian noise. Note, however, that under the manifold hypothesis, this type of mollification produces the opposite effect to the addition of Gaussian noise in that at time 0 mollified images lie on a ID manifold and they are gradually transformed to span the dimension of the data manifold; more details on blurring mollification can be found in Appendix B.
Image generation. We evaluate the quality of the generated images using the popular Fréchet Inception Distance (FID) score [22] computed on 50K samples from the trained model using the pytorch-fid library. The results, reported in Table 1, indicate that the proposed data mollification consistently improves model performance compared vanilla training across all datasets and models. Additionally, mollification through blurring, which is in line with recent results from diffusion models [61], is less effective than Gaussian mollification, although it still enhances the vanilla training in most cases. We also show intermediate samples in Fig. 8 illustrating the progression of samples from pure random noise or completely blurred images high- nigh-quaitty images. Furthermore, we observe that Gaussian mollification leads to faster convergence of the FID score for VAE-based models, as shown in Fig. 7. We provide additional results in Appendix E. As a final experiment, we consider a recent large-scale VAE model for the CIFAR 10 data set, which is deep hierarchical VAE (N VAE) [78]. By applying Gaussian mollification without introducing any additional complexity, e.g., step-size annealing, we improve the FID score from 53.64 to 52.26.
VAE- FID [log]
5.99
Epoch 0
Epoch 10
Epoch 20
Epoch 80
5.53
5.07
0
50
100
150
200
Epoch
GLOW FID (log)
5.76-
0
20
Epoch
Vanilla
Gauss.
Blurring
Intermediate samples generated from REAL-NVP flows [14]. which are trained on CELEBA dataset employed with Gaussian (top row) and blurring mollification (bottom row).
Figure 8:
Figure 7: The progression of FID
on CIFARI 10 dataset.
Choice of noise schedule. We ablate on the choice of noise schedule for Gaussian mollification. Along with the sigmoid schedule, we also consider the linear [25] and cosine [49] schedules, which are also popu- lar for diffusion models. As shown in Table 2, our method consistently outperforms the vanilla approach under all noise schedules. We also observe that the sigmoid schedule consistently produced good results. Therefore, we chose to use the sigmoid schedule all experiments.
Table 2: FID CIFARIO w.r.t. different
choices of noise schedule.
MODEL
VANILLA
SIGMOID COSINE LINEAR
REAL-NVP
191.98
121.75
118.71
123.98
GLOW
74.62
61.87
71.90
74.36
VAE
191.98
155.13
154.71
156.45
8-VAE
112.42
93.90
92.86
93.14
183.04
146.770
146.449
149.16
8

---

Comparisons with the two-step approach in [43]. Manifold overfitting in likelihood- -ased GMS has been recently analyzed in [43], which provides a two-step procedure to mitigate the issue. The first step maps inputs into a low-dimensional space to handle the intrinsic low dimensionality of the data. This step is then followed by likelihood based density estimation on the resulting lower-dimensional representation. This is achieved by means of a generalized autoencoder, which relies on a certain set of explicit deep GMS, such as VAES. Here, we compare our proposal with this two-step approach; results are reported in Table 3. To ensure fair comparison, we use the same network architecture for our VAE and their generalized autoencoder, and we rely on their official implementation 2. Following [43], we consider variety of density estimators in the cow-dimensional space such NT', Autoregressive Models (ARMS) [77], Adversarial Varitional Bayes (AVB) [46] and Energy- -Based Models (EBMs) [16]. We observe that Gaussian mollification is better or comparable with these variants. In addition, our method is extremely simple and readily applicable to any likelihood-basee GMS without any extra auxilary models the need modify training procedures.
Table 3: Comparisons of FID scores CIFARIO between mollitication and step methods.
MOLLIFICATION
TWO-STEP
VANILLA VAE
VAR+GAUSSIAN VAE + BLURRING
VAE+NF
VAE+EBM
VAE+AVB
VAE+ARM
191.98
155.13
175.40
208.80
166.20
153.72
203.32
4.3
Density Estimation on UCI Data Sets
We further evaluate our proposed method in the context of density estimation tasks using UCI data sets [17], following [68]. We consider three different types of normalizing flows: masked autorgressive flows (MAF) [53], REAL-NVP [14] and GLOW [34]. To ensure a fair comparison, we apply the same experimental configurations for both the vanilla and the proposed method, including random seeds, network architectures, optimizer, and the total number of iterations. Table shows the average log- log-likeling likelihood (the higher the better) on the test data. Error bars correspond to the standard deviation computed over 4 runs. As it can be seen, our proposed Gaussian mollification approach consistently and significantly outperforms vanilla training across all models and all datasets.
Table 4: The average test log- likelihood (higher is better) on the UCI data sets. Error bars correspond to the standard deviation over 4 runs
MAF [53]
REAL-NVP [14]
GLOW [34]
DATASET
VANILLA
MOLLIFICATION
VANILLA
MOLLIFICATION
VANILLA
MOLLIFICATION
WINE
16.32 + 1.88
-11.51 + 0.44
-27.83 = 2.56
-12.51 # 0.40
-18.21 1.14
-12.37 + 0.33
WHITE-WINE
-14.87 + 0.24
-11.96 0.17
-18.34 2.77
-12.30
0.16
-15.24 0.69
-12.44 + 0.36
PARKINSONS
-8.27 0.24
-6.17 0.11
-14.21 = 0.97
-7.74
0.27
-8.29 4 1.18
-6.90 +: 0.24
MINIBOONE
-13.03 à: 0.04
-11.65 0.09
-20.01
0.22
-13.96
0.12
-14.48 0.10
-13.88 4: 0.08
Limitations. One limitation is that this work focuses exclusively on likelihood- GMs. On image data, the improvements in FID score indicate that the performance boost is generally substantial, but still far from being comparable with sfate-of-the-a -ftt DMs. While this may give an impression of a low impact, we believe that this work is important in pointing to one of the successful aspects characterizing DMs and show how this can be easily integrated in the optimization GMs. A second limitation is that, in line with the literature on GMs for image data, where models are extremely costly to train and evaluate, we did not provide error bars on the results reported in the tables in the experimental section. Having said that, the improvements reported in the experiments have been shown on a variety of models and on two popular image data sets. Furthermore, the results supported theretically and experimentally by a large literature continuation methods for optimization.
likelihood-
This work provides an efficient way to improve class of GMs. While we focused mostly on images, the proposed method can be applied other types of data as shown the density
Broader impact.
Phttps://gitaa http:::://ggtttt..o.aaaaaaaaaaaa

---

estimation experiments on the UCI datasets. Like other works in this literature, the proposed method can have both positive (e.g.. synthesizing new data automatically or anomaly detection) and negative (e.g., deep fakes) impacts on society depending on the application.
5 Related work
Our work is positioned within the context of improving GMS through the introduction of noise the data. One popular approach is the use of denoising autoencoders [81] which are trained to reconstruct clean data from noisy samples. Building upon this, [2] proposed framework for modeling a Markov chain whose stationary distribution approximates the data distribution. In addition, [80] showed connection between denoising autoencoders and score matching, which is an objective closely related to recent diffusion models [67, 25J. More recently, [45] introduced a two-step approach to improve autoregressive generative models, where a smoothed version of the data is first modeled by adding a fixed level of noise, and then the original data distribution is recovered through an autoregressive denoising model. In a similar vein, [44] recently attempted to use Tweedie's formula [62] as a denosing step, but surprisingly found that it does not improve the performance of NFs and VAES. Our work distinct from these approaches in that Gaussian mollification guides the estimated distribution towards the true data distribution in progressive manner by means of annealing instead of fixing a noise level. Moreover, our approach does not require any explicit denoising step, and it can be applied off-the-shelf to the optimization of any fikelihood-base GMs without any modifications.
6 Conclusion
Inspired by the enormous success of score-based Diffusion Models (DMS), in this work we hypothe- sized that data mollification is partially responsible for their impressive performance in generative modeling tasks. In order test this hypothesis, we introduced data mollification within the optimiza- tion of likelihood- (NFS) and Variational Autoencoders (VAES). Data mollification is extremely easy to implement and it has nice theoretical properties due to its connection with continuation methods in the optimization literature, which are well-known techniques to improve optimization. We applied this idea to chal- lenging generative modeling tasks involving imaging data and relatively large-scale architectures as means to demonstrate systematic gains in performance various conditions and input dimensions. We measured performance in quality of generated images through the popular FID score.
Generative Models (GMS), focusing in particular on Normalizing Flows
While we are far from closing the gap with DMS in achieving competing FID score, we are confident that this work will serve as the basis for future works on performance improvements in state-of-the-an models mixing DMS and likelihood-base GMS, and alternative forms of mollification to improve optimization of state-of-thhe- art GMS. For example, it would be interesting to study how to apply data mollification to improve the training of GANS; preliminary investigations show that the strategy proposed here does not offer significant performance improvements, and we believe this is due the fact that data mollification does not help in smoothing the adversarial objective. Also, while our study shows that the data mollification schedule is not critical, it would be interesting to study whether it possible to derive optimal mollification schedules, taking inspiration, e.g., from [30]. We believe it would also be interesting to consider mixture of likelihood- to the union of manifolds hypothesis, whereby the intrinsic dimension changes over the input [3]. Finally, it would be interesting investigate other data, such as 3D point cloud data [85] and extend this work to deal with other tasks, such as supervised learning.
GMs to counter problems due
Acknowledgments and Disclosure of Funding
MF gratefully acknowledges support from the AXA Research Fund and the Agence Nationale de la Recherche (grant ANR- 100-1-8-CF6-000 and -00-000)31
References
[1] Y. Bengio, A. C. Courville, and P. Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15:1798-1828. 2012.
10

---

[2] Y. Bengio, E. Laufer, G. Alain, and J. Yosinski. Deep Generative Stochastic Networks Trainable by Backprop. In Proceedings of the 31st International Conference on Machine Learning. volume 32 of Proceedings of Machine Learning Research, pages 226-234, Bejing, China, 22-24 Jun 2014. PMLR.
[3] B. C. Brown, A. L. Caterini, B. L.. Ross, J. C. Cresswell, and G. Loaiza-Ganem. Verifying the Union of Manifolds Hypothesis for Image Data. In International Conference on Learning Representations, 2023.
[4] Y. Burda, R. Grosse, and R. Salakhutdinov. Importance Weighted Autoencoders, In International Conference on Learning Representations, 2015.
[5] A. L. Caterini, A. Doucet, and D. Sejdinovic. Hamiltonian Variational Auto-Encoder. In Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[6] C. Chadebec, L. Vincent, and S. Allassonniere. Pythae: Unifying Generative Autoencoders in Python A Benchmarking Use Case. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 21575-21589, Curran Associates, Inc., 2022.
[7] T. Chen. On the Importance of Noise Scheduling for Diffusion Models. arXiv preprint 1rX1:22301.10972, 2023.
[8] T. Chen, G.-H. Liu, and E. Theodorou. Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory. In International Conference on Learning Representations, 2022.
[9] R. Cornish, A. Caterini, G. Deligiannidis. and A. Doucet. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2133-2143, PMLR, 13-18 Jul 2020.
[10] B. Dai and D. Wipf. Diagnosing and Enhancing VAE Models. In International Conference on Learning Representations, 2019.
[11] G. Daras, M. Delbracio, H. Talebi, A. Dimakis, and P. Milanfar. Soft Diffusion: Score Matching with General Corruptions. Transactions on Machine Learning Research, 2023.
[12] N. Dilokthanakul, A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and
M. Shanahan. Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders. arXiv preprint 161iv:1611.02644 2016.
[13] L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-linear Independent Components Estimation.
In International Conference on Learning Representations, 2015.
[14] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density Estimation Using Real NVP. In International Conference on Learning Representations, 2017.
[15] H. M. Dolatabadi, S. Erfani, and C. Leckie. Invertible Generative Modeling using Linear Rational Splines. In Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 4236 4246. PMLR, 26 28 Aug 2020.
[16] Y. Du and I. Mordatch. Implicit Generation and Modeling with Energy Based Models. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[17] D. Dua and Graff. UCI Machine Learning Repository, 2017.
[18) C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural Spline Flows. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[19] A. Genevay, G. Peyr, and Cuturi. GAN and VAE from an Optimal Transport Point of View. arXiv preprint arXiv: 01800,080700000 2017.

---

[20] M. Germain, K. Gregor, I. Murray, and H. Larochelle. MADE: Masked Autoencoder for Distribution Estimation. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 881-889, Lille, France, 07-09 Jul 2015. PMLR.
[21] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A Kernel Avo-Sample Test. Journal of Machine Learning Research, 13(25):22-773, 2012.
[22] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[23] M. Heusel, Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6626-6637, 2017.
[24] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerch- ner. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In International Conference Learning Representations, 2017.
[25] J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[26] E. Hoogeboom, J. Heek, and T. Salimans. simple diffusion: End-to-end Diffusion for High Resolution Images. arXiv preprint 21Xv0 2023.
[27] E. Hoogeboom and T. Salimans. Blurring Diffusion Models. In International Conference on Learning Representationss 2023.
[28] A. Hyvärinen. Estimation of Non- Non-Normalezze Statistical Models by Score Matching. Journal of Machine Learning Research, 6(24):695 2005.
[29] S. loffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448-456, Lille, France, 07-09 Jul 2015. PMLR.
[30] H. Iwakiri, Wang. S. Ito, and A. Takeda. Single Loop Gaussian Homotopy Method for Non-convex Optimization. In Advances in Neural Information Processing Systems, volume 35, pages 7065-7076. Curran Associates, Inc., 2022.
[31] A. Jabri, D. Fleet, and Chen. Scalable Adaptive Computation for Iterative Generation. arXiv preprint arXiv:2212.11992 2022.
[32] D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational Diffusion Models. In Advances in Neural Information Processing Systems, volume 34, pages 21696-21707. Curran Associates, Inc., 2021.
[33] D. P. Kingma and Ba. Adam: A Method for Stochastic Optimization. In International Conference Learning Representations, 2015.
[34] D. P. Kingma and P. Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[35] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, Improved Variational Inference with Inverse Autoregressive Flow. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[36] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
[37] D. Kingma and M. Welling. An Introduction to Variational Autoencoders. Foundations and Trends in Machine Learning, 12(4):307-392, 2019.
12

---

[38] A. Klushyn, N. Chen, R. Kurle, B. Cseke, and P. van der Smagt. Learning Hierarchical Priors in VAEs. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[39] I. Kobyzev. S. J. D. Prince, and M. A. Brubaker. Normalizing Flows: An Introduction and Review of Current Methods. IEEE Transactions Pattern Analysis and Machine Intelligence, 43(1 ((111):33643393 3979, 2021.
[40] A. Krizhevsky and G. Hinton. Learning Multiple Layers of Features from Tiny Images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[41] Z. Liu, Luo, X. Wang, and X. Tang. Deep Learning Face Attributes in the Wild. 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 3730-3733 3738. IEEE Computer Society, 2015.
[42] A. Liutkus, U. Simsekli, S. Majewski, A. Durmus, and F.-R. Stöter. Nonparametric Generative Modeling via Optimal Transport and Diffusions. In Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 111004111 4113. PMLR, 09-15 Jun 2019.
Flows:
[43] G. Loaiza-Gann Ganem, B. L. Ross. J. C. Cresswell, and A. L. Caterini. Diagnosing and Fixing Manifold Overfitting in Deep Generative Models. Transactions on Machine Learning Research, 2022.
[44] G. Loaiza-Ganena B. L. Ross, Wu, J. P. Cunningham, J. C. Cresswell, and A. L. Caterini. Denoising Deep Generative Models. In Can't I Believe It's 's Not Better Workshop at NeurIPS 2022, 2022.
[45] C. Meng, J. Song, Y. Song, S. Zhao, and S. Ermon. Improved Autoregressive Modeling with Distribution Smoothing. In International Conference on Learning Representations, 2021.
[46] L. Mescheder, Nowozin, and A. Geiger. Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1391-2400. PMLR, 06-11 Aug 2017.
[47] H. Mobahi and J. W. Fisher. A theoretical analysis of optimization by gaussian continuation. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI' 15, page 12051211. AAAI Press, 2015.
[48] H. Narayanan and S. Mitter. Sample Complexity of Testing the Manifold Hypothesis. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[49] A. Q. Nichol and P. Dhariwal. Improved Denoising Diffusion Probabilistic Models. In Proceed- ings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMLR, 18-24 Jul 2021.
[50] D. Onken, S. W. Fung, X. Li, and L. Ruthotto. OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9223-9232, 2021.
[51] A. Ozakin and A. Gray. Submanifold Density Estimation. In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
[52] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normal- izing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research, 22(57): 2021.
[53] G. Papamakarios, T. Pavlakou, and I. Murray. Masked Autoregressive Flow for Density Estima- tion. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
13

---

[54] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Desmaison, Kopf, E. Yang, Z. De Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang. J. Bai, and S. Chintala. PyTorch: An Imperative Style, Pigh-Peeffrmance Deep Learning Library. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[55] M. Pavon, G. Trigila, and E. G. Tabak. The Data-Driven Schrödinger Bridge. Communications on Pure and Applied Mathematies, 74(7):1545-1573 2021.
[56] X. Pennec. Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measure- ments. Journal of Mathematical Imaging and Vision, 25:127-154, 2006.
[57] Pope, C. Zhu, A. Abdelkader, M. Goldblum, and T. Goldstein. The Intrinsic Dimension of Images and Its Impact on Learning. In International Conference Learning Representations, 2021.
(58) D. Reshetova, Y. Bai, X. Wu, and A. Özgür. Understanding Entropic Regularization in GANs. In 2021 IEEE International Symposium on Information Theory (ISIT), pages 025-830. IEEE, 2021.
[59] D. Rezende and Mohamed. Variational Inference with Normalizing Flows. Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
[60] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference deep generative models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1278-1286. JoLR.org. 2014.
[61] S. Rissanen, M. Heinonen, and A. Solin. Generative Modelling with Inverse Heat Dissipation. In International Conference on Learning Representations, 2023.
[62] H. Robbins. An Empirical Bayes Approach to Statistics. In Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probability, 1956, volume I, pages 157-163, 1956.
[63] S. T. Roweis and L. K. Saul. Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science, 290(5500):2323- 2326,, 2000.
[64] A. Salmona, V. De Bortoli, J. Delon, and A. Desolneux. Can Push-forward Generative Mod- els Fit Multimodal Distributions? In Advances in Neural Information Processing Systems, volume 35, pages 10766-10779. Curran Associates, Inc., 2022.
[65] M. Sanjabi, J. Ba, M. Razaviyayn, and J. D. Lee. On the Convergence and Robustness of Training GANs with Regularized Optimal Transport. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[66] C. K. Sø nderby, T. Raiko, L. Maalø e, S. r. K. Sø nderby, and O. Winther. Ladder Variational Autoencoders. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[67] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256- 2265, Lille, France, 07-09 Jul 2015. PMLR.
[68] Song and S. Ermon. Bridging the Gap Between f-GANs and Wasserstein GANs. In Proceed- ings of the 37th International Conference Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 9078-9087. PMLR, 13-18 Jul 2020.
[69] Y. Song and S. Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
14

---

[70] Y. Song, J. Sohl-Dickstein D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Base Generative Modeling through Stochastic Differential Equations. International Conference on Learning Representations, 2021.
[71] V. Stimper, D. Liu, A. Campbell, V. Berenz, L. Ryll, B. Schölkopf, and J. M. Hermández-Lobato. normflows: A PyTorch Package for Normalizing Flows. arXiv preprint 2rXiv:2302.12011 2023.
[72] P. Tempezyk, R. Michaluk, L. Garncarek, P. Spurek, J. Tabor, and A. Golinski. LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood. In Proceedings of the 39th In- lernational Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21205-21231. PMLR, 17-23 Jul 2022.
[73] J. B. Tenenbaum, V. d. Silva, and J. C. Langford. A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science, 290(5500):2319 2323, 2000.
[74] Tomezak and M. Welling. VAE with a VampPrior. In Proceedings of the Twenty- First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 1214-1223, PMLR, 09-11 Apr 2018.
[75] B.-H. Tran, S. Rossi, D. Milios, P. Michiardi, E. V. Bonilla, and M. Filippone. Model Selection for Bayesian Autoencoders. In Advances in Neural Information Processing Systems, volume 34, pages 19730-19742. Curran Associates, Inc., 2021.
[76] B.-H. Tran, B. Shahbaba, S. Mandt, and M. Filippone. Fully Bayesian Autoencoders with I Latent Sparse Gaussian Processes. In Proceedings of the 40th International Conference on Machine Learming, volume 202 of Proceedings of Machine Learning Research, pages 34409-34430. PMLR, 23-29 Jul 2023.
[77] B. Uria, Murray, and H. Larochelle. RNADE: The Real-valued Neural autoregressive Density-estimator. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[78] A. Vahdat and J. Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. In Advances in Neural Information Processing Systems, volume 33, pages 19667-19679, Curran Associates, Inc., 2020.
[79] R. van den Berg, L. Hasenclever, J. M. Tomczak, and M. Welling. Sylvester Normalizing Flows for Variational Inference. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 60, 2018, pages 393-402. AUAI Press, 2018
[80] Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23: 1ee1661-16777 2011.
[81] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pages (096-1103) 2008
[82] Z. Wang, H. Zheng. P. He, W. Chen, and M. Zhou. Diffusion-GAN: Training GANs with Diffusion. In International Conference on Learning Representations, 2023.
[83] A. Witkin, D. Terzopoulos, and M. Kass. Signal matching through scale space. In Proceedings of the Fifth AAAI National Conference on Artificial Intelligence. AAAI 86, page 714719. AAAI Press, 1986.
[84] Z. Xiao, K. Kreis, and A. Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In International Conference on Learning Representations, 2022.
[85] G. Yang. X. Huang, Z. Hao, M. Liu, J. Belongie, and B. Hariharan. PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 4540 4549. IEEE, 2019.
15

---

A A Primer on Normalizing Flows and VAEs
Given a dataset consisting of N iidd samples D a {x. LN with x; E RD, we aim at estimating the unknown continuous generating distribution Pdata (x). In order to do so, introduce model po(x) with parameters and attempt to estimate 0 based on the dataset D. A common approach to
estimate f is to maximize the likelihood of the data, which is equivalent to minimizing the following objective:
( (xxa(()) [log (x)
Optimization for this objective can be done through a stochastic gradient descent algorithm using minibatches of samples from Pdata (x).
A.1 Normalizing Flows
In flow-based generative models [52, 39], the generative process is defined as:
(5) where z E DR is a latent variable, and (z) is a tractable base distribution with parameters Φ,, such as an isotropic multivariate Gaussian. The function f. RR. + RD is invertible, such that given any input vector x we have z = r (x). A Normalizing Flow (NF) [59] defines a sequence of invertible transformations = f1 f2 o ... fk. such that the relationship between x and z can be written as:
~ (z)(z)) x = (z),
x =+ h1 f2, h2 h........ [
where h2 = - (hk-1;; and u are the parameters of the transformation fg. For the sake of simplicity, we define ho è x and hx - z. The likelihood of the model given a datapoint can be computed analytically using the change of variables follows:
log pe (x)
log Po (z) + log |det (dz 00x)
K
log P (z)
+ Σ
log
(8)
where log (Ohx (0h 1]] is the logarithm absolute value the determinant of the Jacobian matrix Ohg/ This term accounts for the change of measure when going from h hę using the transformation fg. The resulting NF model is then characterized by the set of parameters = {Φ} U { which can be estimated using the maximum likelihood estimation (MLE) objective Eq. 4.
Though NPs allow for exact likelihood computation, they require fü to be invertible and to have tractable inverse and Jacobian determinant. This restricts the flexibility to certain transformations that can be used within NFs [see e.g., 52, 39, and references therein], such as affine coupling [13, 14], invertible convolution [34], spline [18, 15], or inverse autoregressive transformations [35].
A.2 Variational Autoencoders
Variational Autoencoders (VAES) [36, 59] introduce low-dimensionall latent variable z E Rº, with P & D, the generative process as follows:
99) Here, p(z) is a tractable prior distribution over the latent variables z, and (x z). which is also known as decoder, is usually implemented by a flexible neural network parameterized by 9. Different from NFs, VAES employ stochastic transformation [x z) to map
z z~p(z)) X~ (x z).
x. Indeed, NFS can be viewed as VAES where the decoder and encoder are modelled by Dirac deltas po(x 2) = 5 (fo((x)
and 9 (z) x) 8(f (x)) respectively, using restricted family of transformations fe. The marginal likelihood VAES is intractable and given by:
(x = po (x| 12)((2)de
(10)
16

---

A variational lower bound on ther marginal likelihood can be obtained by introducing a variational distribution (z x). with parameters Φ. which acts as an approximation to the the unknown postenor p(z) x): 1x) 2 (a (o po po (x 2)] - KI. (2 x) p())
(11)
(0.0)
where, (O, Φ) is known as the ELBO, and the expectation can be approximated by using Monte Carlo samples from d x). This objective can be optimized with stochastic optimization w.r.t. parameters 0 and & in place of Eq. 4.
To tighten the gap between the ELBO and the true marginal likelihood. VAES be employed with an expressive form of the approximate posterior qs (z x) such as importance weighted sampling [4] normalizing flows [35, 79]. In addition, to avoid the over regularization offect induced by the prior p(z), one can utilize flexible prior such as multi-modal distributions [12, 74), hierarchical forms [66, 38), simply reweighing the KI. divergence term in the ELBO [24].
B Details on Blurring Mollification
Recently, [61, 27, 11] have proposed appproaches to destroy information of images using blurring operations for diffusion-type typoontttype-eeo generative models. Their approach involves stochastically reversing the heat equation, which is a PDE that can be used to erase fine-scale information when applied locally the 2D plane of an image. In particular, the Laplace PDE for heat diffusions is as follows:
Ot x(i,j.t) Ax(i.j,t)
(12)
where we consider the initial state of the system to be x, the true image data. This PDE can be effectively solved by employing diagonal matrix within the frequeney domain of the cosine transform, provided that the signal is discretized onto a grid. The solution to this equation at time can be effectively computed by: = VD.V x,
(13)
Here, V and V denote the discrete cosine transformation (DCT) and inverse DCT, respectively; the diagonal matrix D, is the exponent of a weighting matrix for frequencies A so that D, exp(At). A. We can evaluate Eq. 13
We refer the reader to Appendix of [61] for the specific definition in the Fourier domain, which is fast to compute, as the DCT and inverse DCT require O log N) operations. The equivalent form of Eq. 13 the Fourier domain is as follows:
ú, (t),,
(14)
where u VT DCT(x). As A is a diagonal matrix, the above Fourier-space model is fast to evaluate. A Python implementation of this blurring mollification is presented in Algorithm 2.
We follow [61] to set the schedule for the blurring mollification. In particular,
use a logarithmic spacing for the time steps tk. where to = oB B/2/// and tr = 22.0//2 = 0.5² /2, corresponding to sub- bibellxe--size blurring. Here, B max is the effective lengthscale-scte of blurring at the beginning of the mollification process. Following [61], we set this to half the width the image.
Algorithm 2: Python code for blurring mollification
import numpy up from scipy. rftpack import dct, idet
def blurring.nollity (x, t):
# Assuning the image u is an (KxK) numpy array
s.ape fregs wtp.piiap. K-1,
7
freqe[: None]] freqs [Kone =:]+2 x_proj det(u, axis=0, norm='ortho
Irequencies_equaree.
10
x.proj (x.pox_ axis-l, norm='ortho")
11
x_proj ap.axp(-treqqunctee.ee
x.proj
12
x_molliiied = idct(x__proj. axis=0, nora= (othoo)
I_nollified (det(i.noliiiee, axis=1, oortho
14
raturn x_mollified
17

---

C Implementation of Noise Schedules
Algorithm 3 shows the Python code for the noise schedules used in this work. For the sigmoid schedule, following [7], we set the default values of start and end to 0 and 3, respectively.
Algorithm 3: Python code for noise schedules
import nшпру up
def signoid (x): Signoid function.
raturn (1 epxxxp(-x))
sigmoid_schedulat т, tau=0.7, start=D, and=3, :isp_nia=le-9): # scheduling function based on signoid function with a temperature tau. y_start signoid tau)
10
v_end signoid(end tau)
(v_end signoid (and start) start) tau)) / (v_and v_atart)
12
13 def linear scnnnr-sceddllet. T):
scheduling return 1 t/7
function.
15
17 def cosine_schodulete Т, m000. ke=0.00026):
18
# scheduling function based cosine function,
19
return ap.coa((((1)) ns) (1 de)) up up-pi p1 a+2
D Experimental Details
D.1 Data sets
Synthetic data sets.
Mixture of Gaussians: We consider mixture of two Gaussians with means 14 (2iin(k)), 2 cos(Tt)) and covariance matrices Σg = I²I, where a = ssin(z/2) We generate 10K samples for training and 10K samples for testing from this distribution.
Mises distribution: We use von Mises distribution with parameters k 1, and then transform to Cartesian coordinates to obtain distribution on the unit circle in IR.. We generate 10K training samples and 10K testing from this distribution.
Image data sets. We consider two image data sets including CIFARIO [40] and CELEBA [41]. These data sets publicly available and widely used in the literature of generative models. We use the official train/val/test splits for both data sets. The resolution of CIFAR 10 x 32 32. For CELEBA, we pre-pree pre-peccess images by first taking 148 x 148 center crop and then resizing to the 3 x 64 x 64 resolution.
UCI data sets. We consider four data sets in the UCI repository [17]: RED-WINE, WHITE-WINE, PARKINSONS, and MINIBOONE, 10% of the data is aside as test set, and additional 10% of the remaining data is used for validation. To standardize the features, we subtract the sample mean from each data point and divide by the sample standard deviation.
D.2 Software and Computational Resources
We use NVIDIA P100 and A100 GPUs for the experiments, with 16GB and 80GB of memory respectively. All models are trained on a single GPU except for the experiments with NVAE model [78], where we employ two A100 GPUs. We use PyTorch [54] for the implementation of the models and the experiments. Our experiments with VAES and NFS are relied on the pythae (6] and normflows [71] libraries, respectively.
18

---

D.3
Training Details
D.3.1
Toy examples.
In the experiments synthetic data sets, layers consisting of 2 hidden layers of 64 units each. We train the model for 20000 itereations using an Adam optimizer [33] with learning rate of 5- 10-1 and mini-batch size of 256.
REAL-NVP flow [14] with 5 affine coupling
D.3.2 Imaging experiments.
REAL-NVP. We use the multi-scale architecture with deep convolutional residual networks in the coupling layers as described in [14]. For the CIFAR1O data set, we use 4 residual blocks with 32 hidden feature maps for the first coupling layers with checkerboard masking. For the CELEBA data set, resdiual blocks are employed. We use an Adam optimizer [33] with a learning rate of 10-3 and mini-batch size of 64. We train the model for 100 and 80 epochs the CIFARIO 10 and CELEBA data sets, respectively. For the mollification training, we perturb the data for 50 and 40 epochs for CIPARIO and CELEBA, respectively.
GLOW. We use a multi-scale architecture as described in [34]. The architecture has a depth level of K = 20, and number of levels L = 3. We use the AdaMax [33] optimizer with learning rate 3 10 and a mini-batch size of 64. We train the model for 80 and 40 epochs on the CIFARIO 10 and CELEBA data sets, respectively. For the mollification training, we perturb the data for 50 and 20 epochs for CIFARIO 10 and CELEBA, respectively.
Table 5: Neural network architectures used for VAES in experiments. Here, CONV (x. and CONVT (n...))) respectively denotes convolutional layer and transposed convolutional layers with filters, stride of and padding p. whereas FC., represents fully-connected layer with n units, and BN denotes batch-normalization layer.
CIFARIO
ENCODER:
+ CONV - BN + RELU
- CONY + BN - RELU
CONV BN RELU
(256.220 BN RELU
4 CONV/912.422 + BN + RELU
(ONV(1)2242)) + BN + RELU
++ CoNV
CONV BN
++ FLATTEN - FC
4 FLATTEN - FCC26x2
DECODER:
F 2,200 FCa*sxi024
2 PCax8x1024
+ CONVT(312.4. + BN RELU
+ CONNTTOzZ..21 + BN RGLU
CONVT (256.4.9) BN RELU
CONVT1220.500 BN RELU
+ CONVT
- CONY CoNYTu28.5.))
T(129.5.3) + BN + RELU
CONVT
VAES. We use convolutional networks for both the encoder and decoder of VAES [36, 60]. Table shows the details of the network architectures. We use an Adam optimizer [33] with learning rate of 3 10 and a mini-batch size of 128. We train the model for 200 and 100 epochs on the CIFARIO and CELEBA data sets, respectively. For the mollification training, we perturb the data for 100 and 50 epochs for CIFAR10 and CELEBA, respectively. The addtional details of the variants of VAES are as follows:
VAE-IAF [35]: We use a 3-layer MADE [20] with 128 hidden units and RELU activation for each layer and stack 2 blocks of Masked Autoregressive Flow to create the flow for approximating the posterior.
3-VAB [24]: We use a coefficient of 3 = 0.1 for the Kullhack-Leibler divergence (KI.) term in the ELBO objective.
IWAE [4]: We use a number of importance samples of K = 5.
HVAE [5]: We set the number of leapfrog steps to used in the integrator to The leapfrog
step size is adaptive with initial value of 0.001
19

---

NVAE. We use the default network architecture as described in [78]. We train the model on the CIFARIO for 300 epochs with an AdaMax optimizer [33] with learning rate of 10-3 and mini-batch size of 200. For the mollification training, we perturb the data for first 150 epochs.
D.4 UCI experiments
For MAF models [53], employ autoregressive layers, each composed of a feedforward neural network utilizing masked weight matrices [20]. This neural networks consists of 512 hidden units and employ the tanh activation function. For REAL-NVP [14] and GLOW [34] models, we implement coupling layers, each comprising two feedforward neural networks with 512 hidden units to handle the scaling and shift functions. The first neural network utilizes the tanh activation function, while the latter employs a rectified linear activation function. We introduce batch normalization [29] after each coupling layer REAL-NVP and after each autoregressive layer in MAP. All models are trained with the Adam optimizer [33] for 150 epochs with learning rate of 10-4 and mini-batch size of 100. For mollification training, we employ Gaussian mollification during the entire training process.
D.5 Evaluation Metrics
Maximum Mean Discrepancy. The MMD between two distributions Pdata and pe is defined as follows [21]:
MMD Po) sup |
Epen [h] - Epo [hl]
(15)
where denotes a reproducing kernel Hilbert space (RKHS) induced by characteristic kernel K. The MMD has the closed form:
MMD3 Po) xx (xx, x') Ex,x'=po [K(x, x') - ([l -pp [x(x, x'), (16)
which can be estimated by using samples from Pdata and pg. In our experiments, we use 10K test samples from the true data distribution Pdata and 10K samples from the model distribution pe for estimating the MMD score.
We employ the popular RBF kernel for the MMD, which is defined as follows:
||x x' 12
K(x,x') = 0² exp
(17)
21
with lengthscale / and variance o² 10-4
FID score. To assess the quality of the generated images, we employed the widely used Fréchet Inception Distance [23]. The FID measures the Fréchet distance between multivariate Gaussian distributions, one representing the generated samples and the other representing the real data samples. By comparing their distribution statistics, we can assess the similarity between the generated and real data distributions. The FID score is defined as follows:
FID
Preal
/ggel' ||2
Tr(Eneal
+ E gen
2 Exeal
Pagen
(18)
The distribution statistics are obtained from the 2048-dimensional activations of the poo13 layer of an Inception-v3 network. We use the pytorch-fid library for calculating the FID score in our experiments.
E
Addtional Results
Table 6 and Table 7 illustrate uncurated samples from the trained models. Fig. and Fig. 10 show the progression of FID scores during training on the CIFAR10 10 and CELEBA datasets, respectively.
20

---

Table 6: Uncurated samples from the models trained the CIFARIO dataset.
VANILLA
GAUSS. MOLLIFICATION
BLUR. MOLLIFICATION
REAL-NVP
GLOW
VAE
VAE-IAF
IWAE
B-VAB
HVAE
21

---

Table 7: Uncurated samples from the models trained on the CELEBA dataset.
VANILLA
GAUSS. MOLLIFICATION
BLUR. MOLLIFICATION
REAL-NVP
GLOW
VAE
VAE-IAF
IWAE
B-VAB
HVAE
22

---

8-VAE
IWAE
5.99
5.99
FID (log] (c-)
FID [log] (+-)
5.53
FID [log]((+)
5.53
5.53-
5.07
5.07
4.61
100
100
200
Epoch
Epoch
Epoch
VAE-IAF
HVAE
GLOW
5.90
5.99
FID (logl (4-)
FID (log] (4-)
FID [log] (-)
5.53-
4.61
5.07
5.07
50
100
150
200
50
40
Epoch
Epoch
Epoch
Figure The progression FID scores during training on the CIFAR10 dataset.
VAE
3-VAE
IWAE
5.76
5.76
FID (log] (<-)
FID [log] (t-)
FID [log] (+)
4.61
80
100
20
60
80
Epoch
Epoch
Epoch
VAE-IAF
HVAE
GLOW
5.76
FID [log] (c)
FID (log] (-)
[log] (+)
4.61
40
100
20
80
10
30
Epoch
Epoch
Epoch
Figure 10: The progression scores during training on the CELEBA dataset.
23


Reviews:
[{"review_id": "plYofdC7Ax", "reviewer": "Reviewer_S7wY", "summary": "The paper proposes a idea to borrow one of the strengths of score-based DMs, which is the ability to perform accurate density estimation in low-density regions and to address manifold overfitting by means of data mollification. We connect data mollification through the addition of Gaussian noise to Gaussian homotopy, which is a well-known technique to improve optimization. Data mollification can be implemented by adding one line of code in the optimization loop, and the authors demonstrate that this provides a boost in generation quality of likelihood-based GMs, without computational overheads.", "questions": "1. What's the difference in terms of motivation between the famous \"Generative modeling by estimating gradients of the data distribution\" paper and this one?\n\n2. What differentiates this paper from SDE score-matching and langevine dynamics generative models?\n\n3. How to combine this model with VAE? Any clear description or examples?", "limitations": "1. The model as described, seems to make the training process very heavy and slow, similar to diffusion models.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper presents a fairly easy to use data mollification techniques to help vae and flow model to generate better samples, especially for low data density region.", "weaknesses": "1.The first concern of this paper is its novelty and it is very critical for conference like Neurips. It basically states the same thing as in \"Generative modeling by estimating gradients of the data distribution\", a NeurIPS 2019 oral paper. Due to the identical proposition and analysis of generative model, the novel element in this paper is very limited. The difference is instead of inference the score, the paper just let the generative networks to train under the mollified data.\n\n2. The description of the model is vague, except for algorithm1, there is no clear presentation of how to use the model. Although in line 235, it says \"This process uses the solution from one level of mollification as a means to guiding optimization for the next.\", do the authors suggest, in a generative task (lets say image generation) a sample is generated from white noise by following the gradient of each gaussian level? The connection between langevine dynamics, SDE, diffusion models are very strong to this kind of sampling.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The first concern of this paper is its novelty and it is very critical for conference like Neurips. It basically states the same thing as in \"Generative modeling by estimating gradients of the data distribution\", a NeurIPS 2019 oral paper. Due to the identical proposition and analysis of generative model, the novel element in this paper is very limited. The difference is instead of inference the score, the paper just let the generative networks to train under the mollified data … What's the difference in terms of motivation between the famous \"Generative modeling by estimating gradients of the data distribution\" paper and this one?**\n\nThanks for your comment. Indeed, our work is motivated by the success of score-based models and inspired by Song et al. 2019.  We emphasize this multiple times in the paper, for example from line 118 to line 128. In this work, we identify that one of the key elements making score-based diffusion models so performant is the data mollification mechanism as mentioned by Song et al. 2019. However, other likelihood-based models are not equipped with this mechanism. The key research question we aim to address in this paper is whether or not the data mollification mechanism can also be helpful for other likelihood-based generative models and how to implement this procedure effectively. Our work is theoretically backed and provides extensive empirical evidence. Other Reviewers acknowledge this novelty. For example, Reviewer sZKb commented that *“applying mollification to the data in a schedule for generative models is not something done in practice for generative models, and contrary to score-based models, it does not require learning all noise levels, so it could be seen as a clever and creative way of applying similar ideas from the literature”*. We respect the position of the Reviewer on the novelty of our work; however, we believe that science progresses through the integration of existing knowledge. There are many great works built up or combined from existing works. We believe that our results will be valuable to the scientific community and practitioners.\n\n**Q2. The description of the model is vague, except for algorithm1, there is no clear presentation of how to use the model. Although in line 235, it says \"This process uses the solution from one level of mollification as a means to guiding optimization for the next.\", do the authors suggest, in a generative task (lets say image generation) a sample is generated from white noise by following the gradient of each gaussian level? The connection between langevine dynamics, SDE, diffusion models are very strong to this kind of sampling…. What differentiates this paper from SDE score-matching and langevine dynamics generative models?**\n\nThanks for your comment. We strongly believe that there is a fundamental misunderstanding here. Indeed, in this work, we do not propose any model but a method to mollify data during the training to improve likelihood-based generative models. The key difference between this work and “SDE score-matching and Langevin dynamics generative models” is that we do not consider these models. Instead, we borrow one of the key elements of these models, which is the data mollification mechanism, to apply other likelihood-based models, such as VAEs and Normalizing Flows. Notice that these models are one-step sampling models, and we do not change the nature of these models but only change the training procedure as described clearly from line 195 to line 208, and Algorithm 1. In addition, we provide a theoretical justification and extensive empirical evidence to demonstrate the effectiveness of our proposed method.   \n\n**Q3. The model as described, seems to make the training process very heavy and slow, similar to diffusion models…..How to combine this model with VAE? Any clear description or examples?**\n\nThanks for your comment. We respectfully disagree with this point as our method does not introduce any additional computational overheads to the training pipeline of likelihood-based models. This is one of the main advantages of our method and is acknowledged by other Reviewers. This comment makes us believe that there is a fundamental misunderstanding of our paper by the Reviewer. Algorithm 1 clearly describes how to integrate our mollification procedure into any training loop of likelihood-based models like VAEs and normalizing flows. Basically, we only introduce a simple additional step into the training loop to mollify the data according to a noise schedule. Appendix C clearly provides the Python code for how to implement this noise schedule.\n"}, {"review_id": "ABn23gQmUI", "reviewer": "Reviewer_yxxe", "summary": "The paper proposes an optimization method that improves the training of likelihood-based generative models. The central idea is that in the early phases of the optimization, the learner tries to model a smoothed version of the likelihood function. As the optimization proceeds, the smoothening decreases (and the difficulty of the optimization problem increases) with the end goal of learning the likelihood of the true distribution. The authors show that this simple idea (inspired by the diffusion models literature) gives consistent improvements in the performance of likelihood-based generative models.", "questions": "I would love to hear what the authors think about directions for future work to improve the method and further enhance the performance of likelihood-based models. \n\nRelated question, do the authors see any way to trade sampling speed for performance using some modification of their method?\n\nFor the scheduling, would it be possible to move to the next level once the loss flattens? This seems to me a reasonable thing to do in order to move from easier optimization problems to hard ones, using the solution of the previous problem as a warm start. I have seen this trick in the Intermediate Layer Optimization work in which the goal is again to move gradually from easier optimization problems to harder ones.\n\nI think I missed the point about blurring in the paper. Intuitively, it seems to me that both blurring and the addition of noise have the same goal of making the distribution \"easier\". Diffusion models are based on the idea of sampling from hard distributions starting with a warm start from an easier distribution. Is it possible that some other scheduling of the blurring kernel would lead to better results? How about a combination of blurring and noise as in Soft Diffusion?\n", "limitations": "In my opinion, the main limitations of the paper are: i) the poor understanding of why catastrophic forgetting is not happening and how to select the scheduling of the corruption, ii) the absence of a way to trade sampling speed with quality. For ii) the FID numbers right now are very weak. This is not necessarily a limitation specific only to this work, but a limitation of one-step sampling models in general.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The authors borrow an idea from the diffusion literature (the mollification of the data distribution at many levels) and they apply it in an interesting way to likelihood-based models such as VAEs and Normalizing Flows. Interestingly, the modeling target changes during the optimization (starting from easy distributions and slowly going to harder ones). Instead, diffusion models try to model all the distributions simultaneously. As far as I know, this method has not been proposed in previous works and it seems to be effective in improving likelihood-based models.\n\nThe paper is clearly written and the experimental methodology seems valid. \n\nThe authors showcase the effectiveness of their method starting from informative toy examples and moving to real datasets (CIFAR-10 and CelebA). The boost in performance is consistent across the experiments.\n", "weaknesses": "Even though the method increases the effectiveness of likelihood-based models, the performance is still very weak. The authors acknowledge this limitation. This is not necessarily an important weakness since future work might further improve the performance of likelihood-based models to the point they become competitive with diffusion models. However, I do not see how this particular optimization method could be improved to achieve these results.\n\nI do not see why the noise schedule has to be similar to the noise schedule diffusion models use. In the design process of a diffusion model, we typically select the corruption scheduling such that the difficulty of transitioning from one corruption level to the next one during sampling is roughly the same across all corruption levels. Why does this intuition transfer here on how we select the scheduling?\n\nI am also puzzled why there is no catastrophic forgetting happening. The authors mention that for the last 50% of the optimization, they stop using their method of mollification and they train for the true distribution.\n\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q4. ... I do not see how this particular optimization method could be improved to achieve these results.**\n\nWe agree with Reviewer that our method does not necessarily help likelihood-based generative models **(GMs)** reach the performance level of diffusion models **(DMs)**, as this heavily relies on the nature of these models. However, our results clearly demonstrates that our method consistently enhances the performance of likelihood-based GMs. The key advantage lies in the simplicity and ease of applicability of our approach, which can be readily integrated into any likelihood-based GMs without incurring additional costs or necessitating model modifications. As a result, our work holds valuable practical implications for practitioners and may open up new avenues for exploring and enhancing likelihood-based GMs (see Q7).\n\n**Q5. I do not see why the noise schedule has to be similar to the noise schedule diffusion models use...**\n\nOur noise schedule choice is guided by Theorem 1, advocating diminishing Gaussian noise. Any monotonically decreasing function from 1 to 0 is a vaid noise schedule. This property is similar to noise schedules used for DMs. However, we agree that the noise schedule for our method is not necessarily the same as that for DMs. Notice that the choice of noise schedule for DMs is also mainly heuristic and empirical-based [39,20,6].  We drew inspiration from noise schedules in the diffusion literature and carefully evaluated them. The sigmoid schedule showed remarkable results across all our experiments. This aligns with recent results for DMs showing that the noise distribution skewed towards noisier levels is more helpful [6,20]. This is reasonable for our method as we expect more iterations for lower noise to capture finer data distribution details, fewer for higher noise with predominantly Gaussian mollified data for easier learning.\n\n**Q6. ... why there is no catastrophic forgetting happening...**\n\nWe hypothesize that our data-mollification procedure gradually steers the model optimization towards global optima regions, as illustrated in Figure 6. Consequently, we observed that for the image experiments, conducting further training of the models using the true distribution resulted in enhanced convergence to global optima, leading to improvement in image quality. To establish a relevant connection, we associate this procedure with Gaussian homotopy, a widely recognized technique for non-convex optimization. For an example and further insights, you can refer to [16], particularly Figure 3 in that reference, which provides a helpful illustration.\n\n**Q7. ... what the authors think about directions for future work ...**\n\nIn Conclusion, we outlined several directions for future work aimed at enhancing our method and improving the performance of likelihood-based models. Specifically, in line 349, we mentioned that *“...derive optimal mollification schedules ... consider mixtures of likelihood-based GMs to counter problems due to the union of manifolds hypothesis...”*. Additionally, we believe it would be worthwhile to consider an extension of our method that accommodates multiple noise levels simultaneously, similar to the concept seen in DMs. Recent work on Gaussian homotopy for optimization (Lin et al., 2023) could serve as a source of inspiration for exploring this direction and potentially yielding even better results.\n\nRef: Lin et al. Continuation Path Learning for Homotopy Optimization. ICML 2023 Oral. \n\n**Q8. ... Do the authors see any way to trade sampling speed for performance using some modification of their method?...**\n\nWe believe that there is a possible misunderstanding. Indeed, our method does not require modifying the nature of likelihood-based GMs, but only requires a small modification in the training loop as shown in Algorithm 1. Throughout our experiments, we have focused on NFs and VAEs, which are one-step sampling GMs. As we mention in the response to Q7, an interesting direction is to extend our method to consider multiple levels of noise taking inspiration from a recent work on continual path learning for homotopy optimization (Lin et al., 2023).  By adopting this approach, we can efficiently trade training time to achieve a more optimal solution.\n\n**Q9. Would it be possible to move to the next level once the loss flattens?...**\n\nWe agree with Reviewer that designing an adaptive noise schedule using training information might be efficient. While we have not yet explored this approach because our proposed method operates under stochastic optimization, we acknowledge its potential and find it promising for enhancing our method's effectiveness further. We will include this direction in the Conclusion.\n\n**Q10. I think I missed the point about blurring...**\n\nWe explored blurring as an alternative method to perturb image data, starting from an easier distribution. However, our experiments showed that Gaussian mollification outperforms blurring and, in some cases, blurring even harms the generative performance when compared to vanilla training (Table 1). One of the main motivations behind our proposed Gaussian mollification is to address manifold overfitting. By adding Gaussian noise to the data, we effectively increase the intrinsic dimension of the high-dimensional data [34,36]. Thus, this mechanism helps prevent manifold overfitting. Meanwhile, the blurring process averages out the images in the dataset, resulting in a contraction of the data into an even lower-dimensional subspace [48]. For example, at the first iteration, the effective dimensionality is reduced to one. This contraction could potentially magnify manifold overfitting, explaining why Gaussian mollification performs better than blurring. These findings are consistent with results observed in DMs [48]. Given these insights, we believe that a combination of blurring and noise, as seen in Soft Diffusion, could lead to improved results compared to solely relying on blurring. "}, {"review_id": "MIj3WnhTLR", "reviewer": "Reviewer_qdL5", "summary": "In this paper, the authors address the limitation of likelihood-based Generative Models (GMs) in achieving high-quality samples compared to state-of-the-art score-based Diffusion Models (DMs). They propose a novel approach to enhance the generation quality of GMs by incorporating data mollification, a technique used in score-based DMs for accurate density estimation in low-density regions and to mitigate manifold overfitting.\n\nThe authors connect data mollification with Gaussian homotopy, a well-known optimization technique, by adding Gaussian noise to the data. This simple addition can be implemented with just one line of code in the optimization loop of likelihood-based GMs. Through experiments on popular image datasets using various likelihood-based GMs, such as variational autoencoders and normalizing flows, the authors demonstrate improvements in generation quality, as measured by the FID score.\n\nNotably, this approach provides a boost in generation quality without incurring additional computational overheads. This work could potentially be a step towards bridging the gap between likelihood-based GMs and score-based DMs, offering new possibilities for generating high-quality samples in various domains.", "questions": "1. What is the relationship between the described approach and noise augmentation?\n2. How does applying different amounts of gaussian noise continuously (uniform noise schedule) to the data compare to the sigmoid function?\n3. How does the presented method outperform alternative approaches that address manifold overfitting, excluding data augmentation?\n4. Why was the sigmoid function selected over the cosine function given the results shown in table 2?\nIt is not clear if the procedure mentioned on line 258 is applied to all training or only certain models or dataset. What is the role of this on the presented FID numbers in table 1?", "limitations": "1. Lack of explicit discussion on the relationship to noise augmentation: The paper does not provide a thorough examination or explanation of how the approach relates to noise augmentation. While a temperature factor of 0.7 is mentioned, there is no detailed discussion on the significance of this factor or its impact.\n2. Exclucive reliance on empirical analysis: The evaluation of the presented method relies solely on empirical analysis. To establish the superiority of their approach, the authors should conduct comparative experiments with alternative (much simpler) methods and demonstrate that their approach consistently outperforms them. Specifically, they should compare their noise schedule against different amounts of constant noise and show its superiority over a set of variations.\n3. Limited theoretical justification for the choice of noise function: The paper lacks substantial theoretical evidence supporting the selection of the sigmoid noise function. Although the authors refer to Jabri et al.'s empirical results to justify their choice, the findings presented in table 2 contradict this justification by showing that a cosine noise schedule performs better in all cases except GLOW.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The approach presented is both straightforward and seamlessly integrable into the training process of various generative models. The authors provide lucid instructions and accompanying pseudo code for its implementation. Notably, the results displayed in table 1 exhibit a remarkable enhancement compared to the baseline performance.", "weaknesses": "The approach described in the paper lacks explicit discussion regarding its relationship to noise augmentation. However, towards the end of section 3, it appears that the authors have chosen a temperature factor of 0.7, which leads to a substantial mollification towards the end of the training process. Additionally, their results are comparable to what is typically observed with conventional data augmentation techniques.\n\nGiven that the evaluation of the presented method in this paper is entirely based on empirical analysis, it is crucial for the authors to empirically demonstrate that their approach surpasses alternative methods. Replacing the sigmoid noise schedule with a uniform/constant function would be the equivalent to simple data augmentation with Gaussian noise. Since data augmentation using Gaussian noise is a common practice in the training of many generative models, it becomes essential to provide empirical evidence showcasing that the proposed noise schedule yields results that go beyond what can be achieved with a uniform noise schedule. The authors should provide results from experiments with different amounts of noise and show that their noise schedule outperforms all of these.\n\nThe authors present little theoretical evidence for the choice of noise function. They justify their choice of the sigmoid function by referencing Jabri et al., who demonstrated improved stability with the sigmoid function compared to the cosine function through empirical results. However, the empirical findings displayed in table 2 contradict this justification, indicating that a cosine noise schedule performs better than the sigmoid noise schedule in all cases except GLOW.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q11. The approach described in the paper lacks explicit discussion regarding its relationship to noise augmentation. However, towards the end of section 3, it appears that the authors have chosen a temperature factor of 0.7, which leads to a substantial mollification towards the end of the training process.** \n\nWe believe that there is a fundamental misunderstanding here. We explicitly explained in line 195 that our approach involves creating a sequence of progressively less smoothed versions of the original data, rather than applying substantial mollification towards the end of the training process. The visualization in Figure 3 of the main paper provides a clear illustration of the sigmoid schedule with different temperatures. As depicted, the noise variance $\\sigma^2_t$ gradually decreases from 1 to 0 as the iteration $t$ increases, across all values of the temperature hyper-parameter.\n\n**Q12. Given that the evaluation of the presented method in this paper is entirely based on empirical analysis, it is crucial for the authors to empirically demonstrate that their approach surpasses alternative methods. Replacing the sigmoid noise schedule with a uniform/constant function would be the equivalent to simple data augmentation with Gaussian noise... it becomes essential to provide empirical evidence showcasing that the proposed noise schedule yields results that go beyond what can be achieved with a uniform noise schedule. The authors should provide results from experiments with different amounts of noise and show that their noise schedule outperforms all of these ...**\n\nThanks for your comment. Indeed, we already discussed carefully the approach of adding noise into the data for generative models in the related work section, from line 317: *“Our work is positioned within the context of improving GMs through the introduction of noise to the data…”*. Recent works [34,36,35] showed that naively adding a fixed amount of noise to the data can degrade the quality of generated samples as the model is completely trained on the modified data, not with the true data distribution. Our work is distinct from these approaches in that Gaussian mollification guides the estimated distribution towards the true data distribution in a progressive manner by means of annealing instead of fixing a noise level. This is inspired by the reverse process of diffusion models. As the Reviewer suggested, we have performed a new set of experiments with the suggested data augmentation using Gaussian noise (uniform noise schedule). We follow the suggestion in [36] to set the noise variance $\\sigma^2=0.3$ and also consider a smaller noise, i.e. $\\sigma^2=0.1$. The results of the synthetic mixture of Gaussians and von Mise distribution and on CIFAR-10 are attached in the general response. We can see that the uniform noise schedule consistently performs worse than our proposed approach of annealing noise using the sigmoid function.\n\n**Q13. The authors present little theoretical evidence for the choice of noise function. They justify their choice of the sigmoid function by referencing Jabri et al., who demonstrated improved stability with the sigmoid function compared to the cosine function through empirical results. However, the empirical findings displayed in table 2 contradict this justification, indicating that a cosine noise schedule performs better than the sigmoid noise schedule in all cases except GLOW ... Why was the sigmoid function selected over the cosine function  ... It is not clear if the procedure mentioned on line 258 is applied to all training or only certain models or dataset. What is the role of this on the presented FID numbers in table 1?.... While a temperature factor of 0.7 is mentioned, there is no detailed discussion on the significance of this factor or its impact.**\n\nThanks for your comment. In response to Q5 of Reviewer yxxe's, we thoroughly discussed both the theoretical motivation and empirical evidence guiding our choice of noise schedule.\n\n- As presented in Table 2, the sigmoid function demonstrates comparable performance to the cosine function in most cases, with only marginal differences favoring the cosine function. However, notably, the sigmoid function outperforms the cosine function significantly for the Glow model. Based on these observations, we have decided to adopt the sigmoid function as the default noise schedule for our experiments.\n\n- The temperature of the sigmoid function is a hyper-parameter, and ideally should be chosen for different datasets [6].  This hyper-parameter governs the rate of data mollification, controlling how rapidly the noise variance diminishes. After extensive experimentation, we found that a temperature factor of 0.7 consistently yields favorable outcomes across all experiments, making it a sensible default value.\n\n- We confirm that the procedure mentioned in line 258 is consistently applied to all experiments with image data. We also further elaborate on this in our response to Reviewer yxxe's question (Q6).\n\n**Q14. How does the presented method outperform alternative approaches that address manifold overfitting, excluding data augmentation? … Exclucive reliance on empirical analysis: The evaluation of the presented method relies solely on empirical analysis. To establish the superiority of their approach, the authors should conduct comparative experiments with alternative (much simpler) methods and demonstrate that their approach consistently outperforms them.**\n\nThanks for your comment. We believe that the reviewer potentially missed our experiments to compare our approach with a two-stage procedure proposed by [34] in mitigating manifold overfitting. We found that our proposed Gaussian mollification is better or comparable with this approach.  In addition, our method is extremely simple and readily applicable to any likelihood-based generative models without any extra auxiliary models or the need to heavily modify training procedures."}, {"review_id": "AxX2yVhngl", "reviewer": "Reviewer_sZKb", "summary": "This paper proposes the use of data mollification to improve sample quality from likelihood-based generative models. The authors observe that likelihood-based models are worse-performing than score-based models due to manifold overfitting (i.e., learning the manifold but not the distribution on it), and poor density estimation in low-density regions (demonstrated with toy examples). Data mollification is proposed as a solution to these issues and demonstrated visually with toy mathematical constructions. On CIFAR10 and CelebA (64x64), the methodology is shown to improve FID scores and there is a visible qualitative improvement in the model samples. Different choices of mollification processes and annealing schedules are explored and compared.", "questions": "- The single most important detail: can the authors confirm that the *targets* of the loss are mollified? I am assuming that this is indeed the case as they mention that mollification == smoothening the loss landscape, but this is not explicitly stated in the main body of the paper, and a reader might interpet the methodology as a certain kind of data augmentation where only the input is degraded via noise/blurring/etc in a schedule. I strongly suggest reflecting this in Algorithm 1.\n- Are models trained with the vanilla approach trained for the same amount of steps as those with data mollification? It might be possible that the models have different convergence speeds because the data distribution changes. It would be nice to show that the strongest baselines are considered in terms of training both approaches with the best possible number of steps.\n- Can we include samples for at least one image dataset in the body of the paper? All metrics are flawed, and visualizing the samples themselves is key to observing what the better numbers actually translate to (also, FID scores are not comparable between datasets, so it is hard to judge what the quantiative improvements entail without looking at samples).\n- Nit: can we drop the use of \"ablation\" for *comparisons* of different schedules, and refer to the vanilla models as an ablation study? Ablation == removing a component.", "limitations": "As mentioned in the weaknesses section, the biggest limitation is that the datasets considered might be too narrow, and there is uncertainty on the methodology being helpful for larger / higher entropy / more diverse datasets . Utilizing larger-scale data essential for larger scale models to attain good results, and it has been the case in many prior work that, e.g., results in CIFAR10 can differ vastly to more realistic image datasets. In my opinion, addressing this successfully should flip this paper to acceptance.\n\nAnother limitation is that, as the authors note, score-based models (which are also likelihood-based generative models-- they can be interpreted as hierarchical VAEs with a fixed approximate posterior) are state-of-the-art image generators but are not considered in experiments. For smaller-scale tasks (e.g. CIFAR10) or the toy examples, this could have been possible without requiring significant computational resources. It is possible that naive gaussian mollification of the data might interfere with a score model's ability to learn the noise level adequately, so there is some uncertainty that it will simply work as intended on this model family which is dominating the field. Moreover, score-based models have an extremely similar coarse-to-fine inductive bias: it is possible that the same benefit provided by data mollification is already provided (albeit at sampling time) and, consequently, the methodology might not improve score-based models at all. Including these results could increase the value of this work significantly if there are positive results, and make (1) the impact of the work more explicit and (2) the exploration thorough and up to date with the field, even if results there are not good. Flows and VAEs are both very relevant to the field and there is a demonstrated contribution in these; more than one model family, and the existing findings are still potentially valuable to the community.\n\nAs noted by the authors, experiments also are limited to mathematical constructions and to image datasets. Focusing a paper on the image modality is reasonable, so I am not concerned about this. Addressing the other main limitations in my opinion will increase the value of the contributions sufficiently for this paper to merit acceptance.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Main results show significant improvements in FID score on multiple image datasets by introducing a technique that entails no additional computational cost. These hold across different image datasets. Samples are not cherrypicked.\n- Results are positive on two families of generative models, normalizing flows and variational autoencoders.\n- The considered hypotheses suggesting why likelihood-based models suffer from sample quality are backed with theoretical work and experiments with toy examples, where visualization of these phenomena can be confirmed.\n- Various hyperparameters (e.g., the choices of mollification process and schedule) are compared, and the main results are ablated against the same model without the proposed methodology. This gives confidence in their methodology truly demonstrating improvement on the datasets and models considered.\n- The paper is very well written, has clear and easy-to-follow language, and the literature review is more than satisfactory. There are a few issues on presentation (see questions section in my review).", "weaknesses": "- My biggest concern is that the image datasets considered might be too simple to conclude that these techniques will materialize into improvement for larger-scale datasets and models. One potential way to do this is to study, e.g., ImageNet 64x64; usually, positive results there are much more likely to hold in larger scales due to the large scale and difficulty of ImageNet itself. Conversely, results in datasets with very narrow distributions (low entropy) like CIFAR10 and CelebA might not transfer to very diverse, more challenging datasets in practice. Studying the effect of the methodology on more challenging datasets will make the results much more valuable and significant.\n\n- The methodology is not very substantial due to its extreme simplicity, nor is mollification new to the literature as acknowledged by the authors themselves. Arguably, however, applying mollification to the data in a schedule for generative models is not something done in practice for generative models, and contrary to score-based models, it does not require learning all noise levels, so it could be seen as a clever and creative way of applying similar ideas from the literature. Simplicity can be positive, and perception of novelty can be extremely subjective, so I am not too concerned about this. Still, due to how short the contribution is, exploring more models would been very welcome since the paper is generally about likelihood-based generative models.\n\n- Minor: while there is enough visual and empirical evidence, there is little discussion for *why* data mollification alleviates the problems of manifold overfitting and accurate density estimation in low-density regions. Since there is a lot of discussion and even theorems from prior work in the body of the paper, a theoretical lens, or at least hypotheses or discussion around this question feel lacking and would be very welcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q15. My biggest concern is that the image datasets considered might be too simple ... One potential way to do this is to study, e.g., ImageNet 64x64...**\n\nThank you for your comment. Unfortunately, prior to submission, we faced limitations in compute capacity, preventing us from exploring such a dataset that demands significant training efforts. In addition, due to the restricted time frame of the rebuttal phase, we are unable to conduct experiments on the ImageNet 64x64 dataset. However, we respectfully disagree that the CIFAR10 and CelebA datasets are too simple. These datasets are widely recognized as standard benchmarks for evaluating generative models. It is worth noting that recent works addressing manifold-overfitting challenges in generative models have also utilized these datasets. For instance, Loaiza-Ganem et al. (2022) employed much simpler datasets such as MNIST, FMNIST, SVHN. Additionally, Meng et al. (2021) considered the same datasets but focused solely on a single autoregressive model, PixelCNN++. In contrast, our approach involved rigorous testing across a diverse set of VAEs and normalizing flows, showcasing consistent and significant improvements in all experiments. Therefore, we believe that our empirical evidence is strong enough to demonstrate the effectiveness of our proposed data mollification method.\n\nRef:\n\n-  Loaiza-Ganem et al. Diagnosing and Fixing Manifold Overfitting in Deep Generative Models. TMLR 2022.\n- Meng et al. Improved Autoregressive Modeling with Distribution Smoothing. ICLR 2021 Oral.\n\n\n**Q16. Minor: while there is enough visual and empirical evidence, there is little discussion for why data mollification alleviates the problems of manifold overfitting and accurate density estimation in low-density regions...**\n\nThank you for your comment. Our mollification approach is theoretically motivated by Theorem 1. Indeed, we also further discuss this theorem from line 166 to line 185. \n\n**Q17. Can the authors confirm that the targets of the loss are mollified? I am assuming that this is indeed the case as they mention that mollification == smoothening the loss landscape, but this is not explicitly stated in the main body of the paper ... I strongly suggest reflecting this in Algorithm 1.**\n\nThanks for your comment. We confirm that the targets of the loss are mollified. In fact, in the Introduction (line 66) we clarified that *“Strictly speaking, data mollification in score-based DMs and likelihood-based GMs are slightly different. In the latter, the amount of noise injected in the data is continuously annealed throughout training. At the beginning, the equivalent loss landscape seen by the optimizer is much smoother, due to the heavy perturbation of the data, and a continuous reduction of the noise level allows optimization to be gracefully guided until the point where the level of noise is zero”*. We appreciate the reviewer's suggestion and we will further elaborate on this when discussing Algorithm 1 in the camera-ready.\n\n**Q18. Are models trained with the vanilla approach trained for the same amount of steps as those with data mollification? It might be possible that the models have different convergence speeds because the data distribution changes...**\n\nThanks for your comment. Indeed, we explicitly stated in line 259 that *“Nevertheless, to ensure a  fair comparison, we adopt identical settings for the vanilla and the proposed approaches, including random seed, optimizer, and the total number of iterations.”* Furthermore, we also report the progression of FID scores during training in Figure 2 of the Appendix.\n\n**Q19. Can we include samples for at least one image dataset in the body of the paper? All metrics are flawed ... so it is hard to judge what the quantiative improvements entail without looking at samples).**\n\nThanks for your comment. Although the metrics we used may be flawed, they are widely accepted as standard ways to assess performance of generative models in the literature. However, we agree with your point and will move some qualitative samples from the Appendix to the main paper of the camera ready. \n\n**Q20. Nit: can we drop the use of \"ablation\" for comparisons of different schedules, and refer to the vanilla models as an ablation study?...**\n\nThanks for your comment. We will drop the use of “ablation” for comparisons of different schedules in the camera ready.\n\n**Q21. Another limitation is that, as the authors note, score-based models ... are state-of-the-art image generators but are not considered in experiments ... Moreover, score-based models have an extremely similar coarse-to-fine inductive bias: it is possible that the same benefit provided by data mollification is already provided (albeit at sampling time) and, consequently, the methodology might not improve score-based models ...**\n\nThanks for your very interesting comment. As the Reviewer mentioned, we believe that our data mollification might not improve score-based models as these models are already equipped with a data mollification mechanism. Our primary objective is not to improve score-based diffusion models. Instead, as stated in line 47, our paper aims to address the question: *Can we leverage the strengths of score-based diffusion models to enhance likelihood-based generative models without incurring the high cost of sample generation?*. Furthermore, our work proves beneficial in untangling the diverse factors contributing to the exceptional performance of score-based diffusion models. Specifically, we hypothesize that data mollification serves as a distinctive element that sets these models apart from other likelihood-based models, such as VAEs and normalizing flows. \n"}]

Author rebuttal:
Dear Reviewers,

We thank the Reviewers for careful reviews of our paper and for insightful comments, which have helped us improve the paper significantly. We are encouraged by the endorsements in the initial reviews that: 1) **the paper is very well written** (Reviewer sZKb and yxxe); 2) **the proposed method is novel, straightforward and clever, and has not been proposed in previous works and it seems to be effective in improving likelihood-based models** (Reviewer sZKb, qdL5 and yxxe); 3) **the experimental methodology is valid** (Reviewer yxxe) and **the results are positive** (Reviewer sZKb, qdL5); 4) **the literature review is more than satisfactory** (Reviewer sZKb).

However, there are potential misunderstandings and concerns from the reviewers, particularly Reviewer S7wY and qdL5. We believe that we have been able to address the Reviewers' comments by clarifying certain sections of the paper and incorporating some additional experiments. Details on these changes can be found in the response to individual comments by the Reviewers. The attached file contains additional results used to address Q12 of Reviewer qdL5.


Meta-review: The reviewers identified some issues and areas of confusion.  During the discussion phase the authors provided clarifications to these issues which led to reviewers increasing their scores.  As a result and after having read the reviews, discussion and paper I recommend acceptance.
Decision: Accept (poster)