Can Push-forward Generative Models Fit Multimodal Distributions?
Authors: Antoine Salmona
Valentin De Bortoli
Julie Delon
Agnès Desolneux
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative Models
GAN
VAE
Diffusion Models
Score-based Models
Expressivity
Lipschitz Mappings

Abstract: Many generative models synthesize data by transforming a standard Gaussian random variable using a deterministic neural network. Among these models are the Variational Autoencoders and the Generative Adversarial Networks. In this work, we call them "push-forward" models and study their expressivity. We formally demonstrate that the Lipschitz constant of these generative networks has to be large in order to fit multimodal distributions. More precisely, we show that the total variation distance and the Kullback-Leibler divergence between the generated 
and the data distribution are bounded from below by a constant depending on the mode separation and the Lipschitz constant. Since constraining the Lipschitz constants of neural networks is a common way to stabilize generative models, there is a provable trade-off between the ability of push-forward models to approximate multimodal distributions and the stability of their training. We validate our findings on one-dimensional and image datasets and empirically show that the recently introduced diffusion models do not suffer of such limitation.

Can Push-forward Generative Models Fit Multimodal Distributions?
Antoine Salmona
Agnès Desolneux Centre Borelli, CNRS ENS Paris Saclay, France
Julie Delon
Centre Borelli,
MAP5, Université Paris Cité, France Institut Universitaire de France (IUF)
ENS Paris Saclay, France
Valentin De Bortoli
Center for Sciences of Data, CNRS ENS Ulm, France
Abstract
Many generative models synthesize data by transforming a standard Gaussian random variable using deterministic neural network. Among these models are the Variational Autoencoders and the Generative Adversarial Networks. In this work, we call them "push-forward" models and study their expressivity. We formally demonstrate that the Lipschitz constant of these generative networks has to be large in order to fit multimodal distributions. More precisely, show that the total variation distance and the Kullback-Leible divergence between the generated bounded from below by constant depending on
and the data distribution the mode separation and the Lipschitz constant. Since constraining the Lipschitz constants of neural networks is common way stabilize generative models, there is a provable trade-off between the ability push-forward models to approximate multimodal distributions and the stability of their training. We validate our findings on one-dimensional and image datasets and empirically show that the recently introduced diffusion models do suffer of such limitation.
Introduction
Generative modeling has become over the last years one of the most popular research topics in machine learning and computer vision. From mathematical perspective, the goal of generative modeling can be seen as predicting new synthetic samples from an unknown probability distribution I&d given the information of m true samples x; drawn from (the data distribution). A general approach to solve this problem is to define a parametric family of probability distributions (vg (vg)ege and solve the problem
mingge D 1 Σ, vo),
where D is similarity measure between probability distributions and 8, is the delta distribution at v. Beside their direct application (Sandfort et al., 2019; Antoniou al., 2018), generative models have been used in numerous applications in various machine learning subfields, such as solving inverse problems (Ravuri al., 2021; Ledig et al., 2017) machine translation (Isola al., 2017; Yang et al., 2018). However, most generative modeling methods still lack theoretical understanding and it remains often unclear whether the method approaches correctly the probability distribution or only generates samples that appear to have been drawn from /' without fully recovering the underlying structure of the distribution. In this work, we focus on the particular class of push-forwan generative models. Those models have in common that for any 0 E θ, the parametric distribution approaching v is of the form
V8 g8
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

where Ae - N(O,Id,) is the Gaussian standard distribution in dimension # is the push-forward operator and go Rº + R is deterministic neural network of parameter O. This class includes two of the most popular generative models: the Variational Anto-Encoders (VAEs) (Kingma and Welling, 2014) and the Generative Adversarial Networks (GANs) (Goodfellow et al., 2014). It also includes other models such as most of normalizing flows (Rezende and Mohamed, 2015).
Deep neural networks are most of the time Lipschitz mappings by design, since their activation functions are generally Lipschitz. In the literature, constraining the Lipschitz constant of neural network is widely used as a way to increase its robustness (Virmaux and Scaman, 2018; Fazlyab et al., 2019), in particular to adversarial attacks (Goodfellow et al., 2015). Common approaches to bound Lipschitz constants of neural networks are spectral normalization (Miyato et al.. 2018), adding a gradient penalization in the loss (Gulrajani al., 2017; Mohajerin Esfahani and Kuhn, 2018), or Jacobian regularization (Pennington et al., 2017). These approaches have been widely used to stabilize the training of GANs, where Lipschitz constraints have been first imposed on discrimators (Arjovsky et al., 2017: Kodali et al., 2017; Fedus et al., 2018), while recent state-of-the-- art architectures such as BigGAN (Brock et al., 2018), SAGAN (Zhang et al., 2019) or StyleGAN2 (Karras et al., 2020) also impose similar constraints on the generators through spectral normalization (Brock et al., 2018; Zhang et al., 2019), or Jacobian regularization (Karras et al., 2020). contrast to GANs, the recent study of Kumar and Poole (2020) shows that the decoder Jacobian in VAFs is implicitly regularized, which limits its Lipschitz constant.
Recently, Dhariwal and Nichol (2021) trained an unconditional Score-based Generative Model (SGM) (Song and Ermon, 2019; Ho al., 2020) on ImageNet (Russakovsky et al., 2015) and achieved state-of-the-ar generation. To the best of our knowledge, there is no push-forward generative model capable of reaching this kind of performance on such a complex dataset without explicitly adding any conditional label information in the model, see (Brock et al., 2018) for instance. SGMs (also known as diffusion models) proceed as follows: first, noise is progressively added to the data distribution until we reach standard Gaussian distribution. Then this forward dynamics is reversed leveraging recent advances in deep learning and tools from score- mate-nn (Hyvärinen, 2005; Vincent, 2011). We refer to Song et al. (2020) for an introduction on SGMs. In those models, the parametric distribution is also of the form uy = 9e#/Up Hpp where ge is the whole reverse diffusion dynamic (which can be seen as a composition of deterministic Lipschitz mappings) and p d(N + 1) with N being the total number of steps in the dynamic (More details can be found in Appendix S2). However, those models are not mush-forward generative models in the strict sense of the term, since the forwad mapping is not a simple neural network anymore. An important difference is that optimization is not directly auxiliary function (the score). We therefore
performed on the push-forwer mapping itself but categorize them as indirect push-forward generalive models in this work.
Contributions of the paper. In this paper, we study the expressivity of direct and indirect push- forward generative models in relation to the Lipschitz constant of the push-forward mapping they learn. More precisely, in Section 3, for a Lipschitz function g and given multimodal probability distribution v, we formally demonstrate that the Lipschitz constant of must necessarily be large in order for 9 (pp to approximate v correctly, as it has been already intuitively observed in the literature (Lu et al., 2020; Luise et al., 2020; Khayatkhoei et al., 2018). As a direct consequence, we exhibit lower bounds on (gg 2/), where D is the total variation distance or the Kellback-Leiblee divergence, with an explict dependence on the Lipschitz constant Lip(g) of g, which highlights that there is a fundamental trade-off for (direct) push-forward generative models between expressivity and stability of training. In Section 4, we illustrate these theoretical results on several experiments, showing the difficulties of GANs and VAEs to simulate multimodal distributions. We compare these models with SGMs and show experimentally that SGMs seem to be able to generate correctly multimodal distributions while keeping the Lipschitz constant of the score network relatively small, suggesting that these models do not suffer of such previously mentioned limitations. All the proofs are postponed to the appendix.
'If j is a measure on R and ff is mapping from TR. to Rd, the push-forward measure f is the measure on de such that for all Borel set A of ued f* = (A)).
II this work the VAE model considered is the Gaussian- VAE since the data are real-valued. See Appendix $2 for details on why the generated distribution is of the form fo# up in this model.
2

---

2
Related Works
Assessing the efficiency of push-forward models is a recurrent and important question in the literature. Sajjadi et al. (2018) 8) and Kynkäänniemi et al. (2019) propase Precison and Recall metrics to assess GANs, aiming to measure simultaneously the mode collapse and the proportion of off-manifold generated samples. Using similar metrics, Tanielian et al. (2020) prove an upper bound on the precision of vanilla GANs (the proportion of generated samples which could have been generated by the target distribution). To overcome this limitation, they simply propose to reject samples associated with large values of the generator Jacobian. The intuition behind this idea is that those samples lie in regions of the space where the discontinuous optimal generator would "jump" between modes and so are off- offffonifodd.
In the context of normalizing flows, it has been shown that the invertibility constraint limits the expressivity of the model. Indeed, Cormish et al. (2020) show that distributions generated by invertible normalizing flows have a support which is necessarily homeomorphic to the support of the latent distribution. As an outcome, the Lipschitz constant of the inverse flow has approach infinity to correctly approximate distributions lying on disconnected manifolds (Cornish et al., 2020; Hagemann and Neumayer, 2021; Behrmann et al., 2021). To improve the expressivity of normalizing flows, has been proposed in Cornish et al. (2020) and Wu et al. (2020) to inject stochasticity in the model.
Another line of research focuses on the fact that the model has access only the empirical distribution Pa A Σ. dz. and not to the true target distribution. For instance Nagarajan et al. (2018) 8) study to what extend GANs only memorize the data. Gulrajani et al. (2018) highlight the fact that common GAN benchmarks prefer training set memorization to a model which imperfectly fits the true distribution but covers more of its support. Related to this, Stéphanovitch specifically the Wasserstein GAN case, where the latent distribution is uniform and construct an optimal generator which minimizes the Wasserstein distance of order between the push-forwar measure and the empirical distribution, thus deriving lower bound on the 1 1-Wasserstein distance. In the same paper, and more related to our work, the authors study the asymptotic case of an infinite number of data and show that most of the time the minimal 1 1-Wasserstein distance between the push-forwan measure and the target distribution remains strictly positive.
al. (2022) study
Push-forward measure and Lipschitz mappings
In this section, we study the properties of the puwh-forward measure I# when Hp = N(O,, Idp) is the standard Gaussian distribution in dimension p and g is a Lipschitz mapping. First, for any probability measure 7 on Idd and any Borel set A of RRd, we define the 7-surface area of A by
>- (OA) = lim inf (A(( - ((A)//E.
where Ag fxr E Rd there exists a E A, || all S. }} is the s-extension of A and OA is the boundary of A. The 7 surface area can be interpreted as the mass of y on the hypersurface dA. Note that the support of y and A can be sets of intrinsic dimension smaller than d. which is most of the time the case when working with real data which are likely to live on low dimensional manifolds (Pope et al., 2020). The main theoretical result of this paper establishes some properties of push-forward measures depending on the regularity of the fush-forward mapping.
Theorem 1. Let g: RP Rd be a Lipschitz fipschitzfuncttoon with Lipschitz constant Lip(g). Then for any Borel set A e B.
Lip(o)(*#un)) (aA) 2 2 (0=)()p(())))
(1)
where xpx)= = (2T) 1/2 expl-x2/2 and ((x) = -_ (f)dtt. In addition, we have that for any r 1 0
9 (*H)(A.)) 2 + (r/Lip(g) + + ((**H((A)))
(2)
Sketch of proof. The proof of this result consists in establishing lower-bounds on (0# ) ) +(A)
and (A,))(,)) which can be expressed as Gaussian integrals. We conclude upon combining this result and the Gaussian isoperimetric inequality, see Sudakov and Tsirelson (1978) 14 (dA) 2
p (A),
3

---

Note that (2) implies (1) upon remarking that (2) is an equality for r = O, dividing by and letting - 0. Theorem recovers the Gaussian inequality in the case where g is the identity map- ping and extends it all Lipschitz mappings. As the Gaussian inequality, Theorem is dimension free, in the sense that neither d, nor p. nor the intrinsic dimension g(RP) play role in the lower bounds. In the following section, we are going to use Theorem to (i) give a lower bound on the Lipschitz constant so that push- forwad generative models exactly match the data distribution, (ii) give a lower bound on the total variation and the Kullback-Leible divergence between the push-forwan and data distributions which depends on the Lipschitz constant of the model.
04-
0.3
002
011
00
0.0
02
04
0.
06
Figure 1: Graph of yp Φ
3.1 Lower bounding the Lipschitz constant of push-forward mappings
Equation (1) implies that the Lipschitz constant of g must necessarily be large for I#/pp to be multimodal. It provides indeed a lower bound on the Lipschitz constant of the mappings which push Mp into given measure In the extreme case where the support of is composed of disconnected manifolds, we retrieve that there doesn't I exist any Lipschitz mapping which pushes Hp into since it can be found Borel sets A with null -surface area but such that the right-hand term of (1) is strictly positive (which occurs when 0 < (AA) < 1). In the intermediate case where the support of is connected but is multimodal, the less mass has between modes, the larger must be the Lipschitz constant of the mappings which push Ml into v. Indeed, if y has little mass between its modes, one can find sets A with small v-surface area and such that v(A) < As a toy example, we get an explicit bound on the Lipschitz constant of the mappings which push Hp into a mixture of two isotropic Gaussians.
Corollary 2. Let v XN(m,, o² Ida) (1 A)N(m2, o² Ida) with mi, m2 Rd., > and À E (0, 1). Assume that there exists IP R Lipschitz such that I# Then
Lip(g) 2 a exp ||m - m1ll mmlll//8002) (A)1(/)))
Sketch of proof. The proof of this result consists into applying Inequality (1) of Theorem on the half-space H such that ØH the equidistant line of the means of the two Gaussians, and then to explicit the value of ++ (8H). See Figure S1 for visualization of ÖH in the univariate case.
Note that assuming there exists g RP + Rd such that I# = implies 2 d since covers the whole ambient space and so y must be a surjective mapping. This bound is maximal in the balanced case when À 1/2 since &- 1 (A) 0 in that case. Otherwise, the more unbalanced the modes the smaller the bound since the two terms in the exponential compensate each other more and more. Extending this corollary to mixtures of more than two Gaussians with different covariance matrices is technically difficult but we could expect a similar exponential growth in the square distance between modes since it depends mainly the order of magnitude of the local minima of the distribution density. As a by product of Theorem 1, we also get the following result which shows that (in the one-dim one-dimensional case) the optimal transport map for the & cost minimizes the Lipschitz constant of the push-forward mapping.
Corollary 3. Let v probability measure on R with density w.r.f. the Lesbesgue measure and such that supp(v) R. Assume that there exists g R. + R Lipschitz such that 94Hp. Let us denote Tor = - t the Monge map between μr and v, where t is the cumulative distribution function of v. Then we have Lip(g) Z Lip(Tor).
To the best of our knowledge. extending this proposition to the case where d > 1 remains open problem. We show now that Equation (2) of Theorem allows to derive lower bounds on similarity measures between the push-forward measure and the target distribution.
3.2 Lower bounds on similarity measures between probability distributions
Equation (2) provides bound on the minimal mass the push-forward measure g# can have on given set when g is fixed with Lipschitz constant Tip(g). As a consequence, if is a distribution

---

such that there exists sets on which V/ has less mass than the minimal quantity that I#H4p can reach on those sets given the value of Lip(g), then Hy cannot be equal to v, implying that most of similarity measures between Pp and v will be automatically strictly positive. In the following, we consider that g and v are fixed and we derive lower bounds on the total variation distance and the Kullback-Leible divergence between P and v. We recall that the total variation distance between two probability measures on R. Mo, V1 is given by
d v1) (A) in (A): A E ((RR))
Similarly, define the Kullback-Leibler divergence between two probability measures on R Y0, V1, using the Donsker- Donsker-Varadhan representation (Dupuis and Ellis, 2011, Lemma 1.4.3a):
(KL(vol = sup f(z)du((z) log (Jz² explf(=)]ddz(z)) : (BRRRR)
. where 2 (IR., IR) denotes the set of all bounded mappings from Rd to R. In the following, we will denote for any A E ((Rd) and r > 0,
ag (A, r) = 1b (r/Lip(g) + & T (*#Hp(A)))
(A) (,, - (A))
where ag(A,,) r) and (A. are the lower bounds of (p/p)(Ap) and Ppp A) provided by Theorem 1. We start by proving lower bounds on the total variation distance.
Theorem 4. Let v be a probability measure on Ed and let g R2 + R2 be a Lipschitz function. Then,
(Tv(##1p: v) 2 (up/a,(A,r) min(g v(A) - v(A, AA) A A 0} (3)
Sketch of Proof. The proof of this result consists in bounding from below the total variation distance on one hand by 19* \ A) ,(A, A)] and the other hand by 19 (A,) v(A,) for given A B(RR)) and given 0, and then applying Theorem 1.
Observe that (3) always holds but the right- hand term may become negative if the Lipschitz constant of g is large enough. The main idea behind this bound is to find a set A and real >> 0 such that v has a lot of mass on A but almost no mass on A, \ A. For instance, if y is a distribution on two disconnected manifolds M₁ and M2, an optimal choice for A would either be M₁ or M2 and the optimal r would be the distance between the two manifolds. Using Theorem 4, one can derive smaller but more explicit lower bounds only depending on and the Lipschitz constant of g. As first example, we derive an explicit lower bound in the case where v is a bi-modal distribution on two disconnected manifolds.
Corollary 5. Let v be a measure on RRd on two disconnected manifolds Mi and M2 such that v(M1) X and v(M2)= 1 with [1/2,1] 1), and let g R2 + Rd be a Lipschitz function. Then,
dTv(0 v) dM M2) Maa/ (())dtt
where d(Mj, M2) = inf[[lm - -221 mj E M1, E M2).
As second examplc, we also get an explicit lower bound in the case where v is mixture of two isotropic Gaussians. For simplicity we stick to the balanced case.
Corollary 6. Let v (N()) o² Ida) + N(m2, a² Ida)] with mi, 1222 E Rd and a 2 O. Let
: ::P + Rd be a copschitz,uucttin Then,
maa- Pp, v) 2 mal/om-ll//o ((())l - (1/2) [ [|mm-m 1120-1)/442 (p)))d
In both corollaries, the lower bound tends 1/2 when the distance between the modes tends to infinity, meaning that I#Hpp is far from well approaching v. Note that the lower bound exhibited in Corollary 5 is always strictly positive regardless of the value of the Lipschitz constant of g. One can also observe that this latter bound is maximal in the balanced case, when À = 1/2, since the standard normal distribution concentrates its mass around 0. Finally, we end this section by deriving a similar lower bound on the Kullback-Leibles divergence between I# ('p and v. We consider the Kullback-Leine divergence since this is measure of similarity between measures which is bounded and is very sensitive to the mismatch of supports between the generated and the data distributions.
5

---

Theorem 7. Let v be a probability measure on B. and let g R + R be Lipschitz function. Then,
dkI(g=Hpllp) 2 sup(
(g(A, log ((A,(A)) : A C BBBR) )r 0}
((A,
Sketch of Proof. The proof of this result consists in setting f (XA.IA with >> 0 and where XA the characteristic function of the set and plugging it in the Donsker- Varadhan representation in order to get lower bound depending on the probability g (A, \ AA for given A, a given r and given Ç. Then we apply Theorem and we derive the optimal value of Ç.
As above, this bound always holds but the right-hand term becomes negative if Lip(g) is large enough. As for Theorem 4, the main idea is to find set A and real r such that v has a lot of mass on A, but has almost mass A, \ A. Observe that if v/A,'' \ A) tends to 0, the left-hand term of the bound tends to infinity. This is coherent with the behavior of the Kullback-Leiblee divergence. Similarly to Corollary 6, we also get explicit lower bound in the case where is a mixture of two isotropic Gaussians. As for Corollary 6, we stick to the balanced case for simplicity.
ma E R2d and 0. Let Corollary 8. Let v = (1/2) [N(m1, o² Ida) N(m2, o² Ida)] with m1,
9: TRP g:R + Rdd be a Lipschitz function.. We denote
A P# (4(m2 (1(m2-m))* (x (m2 m2)/2) 0 ER)
and we suppose without loss of generality, that À E 1/2]. Then.
dkL. 4kt.99#1p: v) 2 log 4 + (1 A) log
1-B
where
(1-x) (tt)dt, and B = (1/2) I m 1)/0- (o(t) A A (1-
llm (20 12
Observe that this time, Lip(g) is no longer the only dependency in g since the bound also depends on the proportion of the modes of 9# /tp However, should be noted that when I# Ppp approximates correctly v, À automatically close to 1/2 and so - A) small in that case. To conclude, this section, we highlight the fact that, if our results are dimension free in theory, the dimension might be hidden in the distances between modes and the Lipschitz constant of g when working with real datasets. Indeed, the order of magnitude of the Euclidean distance between two samples 2, is likely to increases with the dimension d. As an outcome, the orders of magnitude of the distance between modes and so the Lipschitz constant that g must reach for correct generation probably increase with d also.
4 Experiments
In what follows, we illustrate the pratical implications of our results by training GANs, VAEs and SGMs on simple bi-modal distributions. More precisely, we show on one hand that generating multimodal distributions with GANs and VAEs is difficult since for those models, good generation necessarily involves generative networks with large Lipschitz constants. On the other hand, we show that SGMs seem to be able generate multimodal distributions while keeping the Lipschitz constant of the score network relatively small and thus do not suffer of the same limitation. First, we focus on the univariate case where we illustrate our results in higher dimensions by training the three models on datasets derived from MNIST (LeCun et al., 1998). In all our experiments, decoder and the GAN generator in order to offer rigorous comparisons of the different models. For score-based modeling, on the experiments and architecture the networks can be found in Appendix S5.
easily assess the Lipschitz constants of the networks. Then
the same architecture for the VAE
architectures with similar numbers of learnable parameters. All details
4.1 Univariate case
First, we train a VAE and GAN with one-dimensional latent spaces 50000 independent samples drawn from balanced mixture of two univariate Gaussians v = (1/2) (2()NN(m,))) 1) + N(m, 1)) for different values of m > 0. We also train SGM on the same samples.

---

Histograms of generated distributions. Figure 2 shows histograms of generated distributions for m 10 with the three different models. VAE models seem generate Gaussians modes but interpolate significantly between them, while GANs do not interpolate but fail to retrieve the structure of the target distribution and forget parts of their support, which known as mode collapse and is a common pitfall of such models (Arjovsky and Bottou, 2017; Metz et al., 2017). On the same task, SGMs do not suffer from such shortcomings. deerpolation/modecclllaap properties of these models with their Lipschitz constants.
the following section, we will link the
Figure 2: Histograms of distributions generated with VAE (left, in orange), GAN (middle, in green), and with SGM (right, in purple) for 10. The data distribution densities are plotted in blue.
Lipschitz constant and mass between modes. In Figure 3 (right), we observe that the GAN generator reaches much larger Lipschitz constants than the VAE decoder. This explains the difference of behaviors between GAN and VAE observed in Figure 2, as the mapping learned by the VAE is not stiff enough to concentrate the push forward measure on the two modes. One possible explanation for the interpolating behavior of the VAE is that the Euclidean norm of the Jacobian of the VAE decoder is implicitly regularized during training, as has been demonstrated in Kumar and Poole (2020). Both GAN and VAE saturate the constraint [#Hpp[[mm 2 m 2]) provided by Theorem 1, meaning that the generative networks minimize the amount of mass between modes as much as their Lipschitz constants allow it. Finally, we observe that the score network is able to keep relatively small Lipschitz constant compared to the GAN, while managing to interpolate less than the latter. A probable explanation for this follows from the fact that the score network is used multiple time during inference. Hence, the Lipschitz constant of the push-forwane mapping (the whole generation dynamic) likely much larger than the Lipschitz constant of the neural network itself, and so the model is able to push-forward a Gaussian distribution into a multimodal distribution keeping relatively small Lipschitz constant of the score network. Finally, in Figure 3 (left), we observe that when m increases, the Lipschitz constant of the VAE decoder and the GAN generator becomes rapidly much smaller than the value of the lower bound provided by Corollary This means that for m large enough is not possible to close the gap between the data distribution and the push-forward distribution. We highlight that this observation does not apply to SGMs since in this setting the network is applied multiple times.
Lipschitz constunts
between modes
VAE
0.12
VAlI GAN
mas probutllty
GAN
heara Conoliry
bound VAE
Score
hoard CAN
DiC-m/2.m//1
0.08
0.004
10 0.00
0
3
6
10
Figure 3: Left: evolution of the Lipschitz constants of the three different generative models trained on 50000 samples of (1/2)[[(-m, + N(m, 1)], in function of m. Right: evolution of the proportion of samples generated by the three models on the interval [-m/2. m/2). We also show on this graph the lower bounds predicted by Theorem for the VAE and the GAN, as well as the true probability u(l-m/2, m/2)). Experiments are averaged over 10 runs and the colored bands correspond to +/- the standard deviation.
7

---

Stability of GAN and mode collapse. Odena et al. (2018) suggested that the magnitude of the norm of the generator jacobian may be causally related instability and mode collapse. This is why many state-of-the-art GANs apply spectral normalization (Miyato et al., 2018) on their generators. In Figure 4 (left), we show that this technique cannot be used when training GANs on multimodal distributions: since spectral normalization constraints the Lipschitz constant of the generator to be smaller than 1, the GAN is trained towards concentrating in one of the modes of the distribution over interpolating massively between them. This has been referred to mode dropping by Khayatkhoei et al. (2018). To complete this analysis, we also train the GAN adding an additional gradient penalty term 10/L² max Id), V (z) 222 L) in the generator loss function, similarly to WP-GAN (Gulrajani et al., 2017), where L is an hyperparameter corresponding to the targeted Lipschitz constant. As expected, to generate both modes but becomes also more and more prone to mode collapse. This illustrates the fundamental trade-off between expressivity and robustness in push-forwart generative models.
observe in Figure 4 (right), that when Lip(g) increases, the GAN begin
Figure 4: Histograms of distributions generated with GANs with spectral normalization applied on the generator (left), and with gradient penalty (right) for Lip(g) L 5. Lip(g) L = 15 and Lip(g)= = L = 25. The data distribution densities are plotted in blue.
Influences of generator depth and time of training. In Figure 5, we study the effect of increasing the number of layers of the generative network as well as increasing the training time on the value of the Lipschitz constant of the VAE decoder and the GAN generator. In the VAE setting, the Lipschitz constant increases linearly with the depth of the decoder. This is not the case in the GAN setting, where increasing the size of the model seems to dramatically affect its stability. For both models, the Lipschitz constants of the generative network grow with the number of epochs. Yet this growth seems to be logarithmic for the VAE and the GAN seems to becomes more unstable as the number of epochs increases.
150-
VAE
VAE
GAN
GAN
100
150
50
2
0
200
400
500
1000
4_
8_spocts
Figure 5: Evolution of the Lipschitz constant of the generative network with respect its number of layers (left) and of the Lipschitz constant in function of the numbers of epochs (right). The experiments are averaged over 10 runs and the colored bands correspond +/- the standard deviation.
Inffuence of generator architecture. Finally, we study in Figure 6 the impact of the architecture of the generative network (i.e. the VAF decoder and the GAN generator) on its Lipschitz constant as well as on the training stability of the model by comparing three different architectures: first, we use simple feed feed-forward network as precedently, then we add additive -kip-connections of type "resnet" (He et al., 2016) the previous backbone, and last we add concatenation skip-connections of type "densenet" (Huang et al., 2017) instead of additive skip- conpectenns.... For both models, it seems that more expressive decoder architectures do not help to reach larger values of Lipschitz constant. However, one can observe that in the GAN setting, even if the model remains certainly too unstable for correct distribution generation, adding additive skip-connections seems to stabilize the training a little since the colored bands are narrower than for the two other models. This suggests that some generator architectures may be better than others at learning mappings with large Lipschitz constants while staying stable.
8

---

VAB
GAN
70-
m-lip
moxe dips
Teapet cepo
dessnn skts
desuzee
30-
602
40-
1/((0)
400
20-
200
10-
0
10
10
Figure 6: Evolution of the Lipschitz constant of the VAE decoder (left) and the GAN generator (right) trained on 50000 samples of (1/2)[[(-m,)
N(m, 1)] for 3 different architectures of the generative network: simple feed-forward backbone, backbone with skip-connections of type "resnet", and backbone with skip-connections of type "densenet". Experiments are averaged over 5 runs and the colored bands correspond to +/- the standard deviation.
4.2 Experiments on MNIST
We train VAE, a GAN and SGM on two datasets derived from MNIST (LeCun et al., 1998):
first, two images of two different digits (3 and 7) are chosen and 10000 noisy versions of theses images are drawn with a noise amount of = 0.15, forming a dataset of n 20002 independent samples drawn from a balanced mixture of two Gaussian distributions in dimension 784 = 28 x 28. Second, we train the models on the subset of all and of MNIST. We emphasize that our goal is not reach state-of-the-an performance on this problem but rather to illustrate our theoretical results in a moderate dimensional setting.
VAE
GAN
SGM
Data
J.
2.
e
f.2
Y
37
3 7 3 2
3 7
3 7
3 7
3
7
d
3 7 3 3 7 3 7 37 > 7 3
7 7 1
Figure 7: mixture of Gaussians (top): histograms of projections on the line passing through the mean of each Gaussian. Subset of MNIST (bottom): histograms of projections on the line passing through the barycenters of all the and in the deep Wasserstein embedding space. Bins of data are colored in blue if they are classified as 3, in green if classified as 7, and in red if classified as another digit.
Mixture of Gaussians. For this experiment, we set the dimension of the latent space in the GAN and the VAE to 784 = 28 x 28 since it is the intrinsic dimension of the support of the data distribution. In order visualize the interpolation between modes, we project the data on the line passing through the mean of each Gaussian, iee. the two original clean images, and we plot histograms of the one- dimensional projections. order to understand which bins of data in the histograms correspond to which digit, we train classifier and we assign a color in function of which digit the data have been classified as. Results can found in Figure top. Moreover, GAN and VAE both fail to generate noisy versions of the images. As in the univariate case, the SGM is able to not interpolate between

---

modes and seem to retrieve the Gaussian structure of the modes. This suggests that while direct push-forward models fail at representing multimodal distributions, considering stacked models with noise input at each step (as in SGM) might help to close the gap between the generated and the data distributions. However SGM does not manage to retrieve the right modes proportions. This is a well-known shortcoming of score-base based models which has been studied in (Wenliang and Kanagawa, 2020).
Subset of MNIST. Finally we train the three different models on the subset of MNIST composed of all 3 and 7 (no Gaussian noise was added). We choose a latent dimension of 20 for the VAF and the GAN. Since the Fuclidean distance is not meaningful metric compare the different digits of MNIST, we use the deep Wasserstein embedding proposed by Courty et al. (2018): an autoencoder is learned in a supervised fashion such that the Euclidean distance in the latent space approximates the Wasserstein distance between pairs of images of MNIST. In the learned Wasserstein space, we project data on the line passing through the Euclidean barycenters of all 3 and 7 and plot histograms of projections, using the same classifier as before. Results can be found in Figure 7 (bottom). Note that the distribution does not exhibit strong multimodality features contrary to the mixture of Gaussians settings, see Figure 7. As before, the VAE interpolate between modes, the GAN manages to not interpolate but generate a narrower histogram, and the score-based model does not interpolate and seems to recover the structure of the distribution, but doesn't t retrieve the right modes proportions. However, we emphasize that all these models seem to perform better than on the previous dataset. A possible explanation of this is that the modes are less separated than in the Gaussian mixtures and therefore the model is easier to train.
5 Discussion
In this work, given a Lipschitz mapping 9 and a measure v, we derived lower bounds on the total variation distance and the Kullback-Leibler divergence between the push-forward measure g# Up and v depending on the Lipschitz constant of the mapping y. These bounds indicate how the
mass between the modes of the push-forward measure depends on the Lipschitz constant of the push-forward mapping. They highlight the trade-off between the ability of VAEs and GANs to fit multimodal distributions and the stability of their training.
common assumption in the imaging literature, validated empirically by Pope al. (2020), is that distributions of natural images live on low dimensional manifolds. Understanding whether these distributions are composed of separated modes or not remains, to the best of our knowledge, an open problem. To that extent, the fact that unsupervised push-forward generative models perform well on datasets such as CelebA (Liu et al., 2015) 5) could possibly be, in regard of our work, an indicator that the data distributions of those datasets are unimodal, or at least not composed of well separated modes.
Several techniques have been proposed in the literature to fit data distributions on disconnected manifolds. Most of them consist in overparametrizing the model, either by using stacked generative networks (Khayatkhoei et al., 2018; Mehr et al., 2019) or by learning a more complex latent distribu- tion than the standard Gaussian (Gurumurthy et al., 2017; Rezende and Mohamed, 2015; Kingma et al., 2016; Luise et al., 2020). Other methods consist in rejecting a posteriori samples associated to large values of the Jacobian generator (Tanielian al., 2020; Issenhuth et al., 2020). In this work, we empirically showed that score-based models seemed to be able to fit separated manifolds without model overparametrization or additional posterior sample rejection scheme. This suggests that the structure of the generation dynamic in these models is particularly adapted to (indirectly) learn mappings with large Lipschitz constants. Their good performance on multimodal distributions might follow from the fact that these models do not optimize directly the push-forward mapping itself and/or that noise is injected at each step during the generation process. Hence, a future perspective of work would be to study what are the structural aspects of diffusion models that play a significant role in their expressivity.
A possible limitation of this work is that the bounds derived on the Kullback-Leible divergence and total variation distance are not tight (see Appendix S6), mainly because they take no account of the fact that when interpolating, I# My has automatically less mass than v on the modes since a significant amount of its total mass is between them. In future work, we plan to tighten the gap between our bounds and the true distance.
10

---

References
Antoniou, A., Storkey, A., and Edwards, H. (2018). Augmenting image classifiers using data augmentation generative adversarial networks. In International Conference Artificial Neural Networks, pages 194-603. Springer.
Arjovsky, M. and Bottou, I.. (2017). Towards principled methods for training generative adversarial networks. stat, 1050:17.
Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214-223. PMLR.
Behrmann, J., Vicol, P., Wang, K.-C,, Grosse, R., and Jacobsen, J.-H. (2021). Understanding and mitigating exploding inverses in invertible neural networks. In International Conference on Artificial Intelligence and Statistics, pages 1792-18000 PMLR.
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017). Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518)8859-8 877.
Brock, A., Donahue, J., and Simonyan, (2018). Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations.
Cornish, R., Caterini, A., Deligiannidis, G., and Doucet, A. (2020), Relaxing bijectivity constraints with continuously indexed normalising flows. In International Conference on Machine Learning, pages 2133-2143. PMLR.
Courty, N., Flamary, R., and Ducoffe, M. (2018). Learning wasserstein embeddings. In International Conference on Learning Representations.
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34.
Dupuis, P. and Ellis, R. S. (2011). weak convergence approach the theory of large deviations. John Wiley & Sons.
Elfwing, S., Uchibe, E., and Doya, K. (2018). Sigmoid linear units for neural network function approximation in reinforcement learning. Neural Nerworks, 107:3-11.
Fazlyab, M., Robey, A., Hassani, H., Morari, M., and Pappas, (2019). Efficient and accurate estimation of Lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32.
Fedus, W., Rosca, M., Lakshminarayanan, B., Dai, A. M., Mohamed, S., and Goodfellow, (2018).
Many paths to equilibrium: Gans do not need to decrease a divergence every step. In International Conference on Learning Representations.
Goodfellow, I., Poudet--bbdic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, 27.
Goodfellow, J., Shlens, J., and Szegedy, C. (2015). Explaining and harnessing adversarial examples. Stat, 1050:20.
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. (2017). Improved training of wasserstein gans. Advances in Neural Information Processing Systems, 30.
Gulrajani, I., Raffel, C., and Metz, L. (2018). Towards gan benchmarks which require generalization. In International Conference on Learning Representations.
Gurumurthy, S., Kiran Sarvadevabhatla, R., and Venkatesh Babu, R. (2017). Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE Conference On Computer Vision and Pattern Recognition, pages 166-174.
Hagemann, P. and Neumayer, S. (2021). Stabilizing invertible neural networks using mixture models. Inverse Problems, 388800000

---

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances Neural Information Processing Systems, 33:6840-6851.
Huang, G., Liu, Z.., Van Der Maaten, L.., and Weinberger, K. Q. (2017). Densely connected con- volutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700-4708.
Hyvärinen, A. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4).
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1125-1 1334.
Issenhuth, T., Tanielian, U., Picard, D., and Mary. J. (2020). Learning disconnected manifolds: Avoiding the gan's land by latent rejection.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020). Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8110-8119.
Khayatkhoei, M., Elgammal. A., and Singh, M. (2018). Disconnected manifold learning for generative adversarial networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7364.
Kingma, D. P. and Ba, (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations.
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. (2016). Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29.
Kingma, D. P. and Welling, M. (2014). Auto-Encoding Variational Bayes. Stat, 1050:1.
Kodali, N., Abernethy. J., Hays, J., and Kira, Z. (2017), On convergence and stability of gans. arXiv preprint arXiv: 100.:0775...25..
Kumar, A. and Poole, B. (2020). On implicit regularization in 3-vaes. International Conference Machine Learning, pages -4800 PMLR.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. (2019). Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):278--2324
Ledig, C., Theis, I.., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., al. (2017). Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4681-4690.
Lim, H. and Ye, C. (2017). Geometric gan. arXiv preprint arXiv:1705.02880
Liu, Z., Luo, P., Wang, X., and Tang. X. (2015). Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pages 3730-3738.
Lu, C., Chen, J., Li, C., Wang, Q., and Zhu, J. (2020). Implicit normalizing flows. In International Conference on Learning Representations.
Luise, G., Pontil, M., and Ciliberto, C. (2020). Generalization properties of optimal transport gans with latent distribution learning. arXiv preprint 2rXiv:2007.14644
12

---

Maas, A. L., Hannun, A. Y., Ng, A. Y., et al. (2013). Rectifier nonlinearities improve neural network acoustic models. International Conference on Machine Learning. volume 30, page 3. PMLR.
Mehr, E., Jourdan, A., Thome, N., Cord, M., and Guitteny, V. (2019). Disconet: Shapes learning on disconnected manifolds for 3d editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3474 -483..
Metz, L.., Poole, B., Pfau, D., and Sohl-Dickstein, (2017). Unrolled generative adversarial networks. In International Conference on Learning Representations.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. International Conference on Learning Reprexentations,
Mohajerin Esfahani, and Kuhn, D. (2018). Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming, 17 171(1):115-16,
Nagarajan, V., Raffel, C., and Goodfellow, I. J. (2018). Theoretical insights into memorization in gans. In Neural Information Processing Systems Workshop, volume 1.
Odena, A., Buckman, J., Olsson, C., Brown, T., Olah, C., Raffel, C., and Goodfellow, I. (201 8). Is generator conditioning causally related to gan performance? In International Conference on Machine Learning, pages 3849-3858. PMLR.
Pennington, J., Schoenholz. S., and Ganguli, S. (2017). Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in Neural Information Processing Systems, 30.
Peyré, G. and Cuturi, M. (2019). Computational optimal transport: with applications to data science. Foundations and Trends in Machine Learning, 1165-6):355-677
Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T. (2020). The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations.
Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:151 1.16433
Ravuri, S., Lenc, K., Willson, M., Kangin, D., Lam, R., Mirowski, P., Fitzsimons, M., Athanassiadou, M., Kashem, S., Madge, S., et al. (2021). Skilful precipitation nowcasting using deep generative models of radar. Nature, 597(7878):672-677.
Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538. PMLR.
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234 1441241..5 Springer.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, 7.,, Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211
Sajjadi, M. S., Bachem, O., Lucic, M., Bousquet, O.,, and Gelly, S. (2018). Assessing generative models via precision and recall. Advances in Neural Information Processing Systems, 31.
Sandfort, V., Yan, K., Pickhardt, P. J., and Summers, R. M. (2019). Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in et segmentation tasks. Scientifiic Reports, 91):119.
Song, Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2020). Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations.
13

---

Stéphanovitch, A., Tanielian, U., Cadre, B., Klutchnikoff, N., and Biau, G. (2022). Optimal 1- wasserstein distance for wgans, arXiv preprint vrXiv:2201.0282.
Sudakov, V. N. and Tsirelson,
(1978). Extremal properties of half-spaces for spherically invariant measures. Journal of Soviet Mathematics, 91):9-18.
Tanielian, U., Issenhuth, T., Dohmatob, E.,, and Mary, J. (2020). Learning disconnected manifolds: no gan's land. In International Conference on Machine Learning, pages 9418 11889927. PMLR.
Tran, D., Ranganath, R., and Blei, D. (2017). Hierarchical implicit models and likelihood-free variational inference. Advances in Neural Information Processing Systems, 30.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing
Systems, 30.
Vincent, P. (2011). connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674.
Virmaux, A. and Scaman, K. (2018). Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31.
Wang, X., Girshick, R., Gupta, A., and He, K. (2018). Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794-7803.
Wenliang. L. K. and Kanagawa, H. (2020). Blindness of score-based methods isolated components and mixing proportions. arXiv preprint arXiv:2008. 10087.
Wu, H., Köhler, J., and Noé. F. (2020). Stochastic normalizing flows. Advances in Neural Information Processing Systems, 33:5933-5944.
Wu, Y. and He, K. (2018). Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19.
Yang. Z., Chen, W. Wang. F., and Xu, B. (2018). Improving neural machine translation with conditional sequence generative adversarial nets. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1:1346-1355.
Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. (2019). Self-attention generative adversarial networks. In International Conference on Machine Learning, pages 7354-7363. PMLR.
14


Reviews:
[{"review_id": "tyBvLIwmc8u", "reviewer": "Reviewer_JkRJ", "summary": "This paper studies the expressivity and stability of generative model that transforming a gaussian to data (such as GAN and VAE). In particular, the authors showed that there is a tradeoff between the approximation power and the stability (in terms of Lipchitz constant) of training for multimodal distribution. Experiment results are provided to validate the theoretical results.", "questions": "I would appreciate if the following questions could be answered:\n1. In line 100 it is mentioned that in most of the time ‘p is almost always smaller than d in practice’. However, for corollary 2 and 3, in line 135 it is mentioned that p >= d. I was wondering if authors could comment on this.\n2. Influences of model size and time of training paragraph in the experiment section: in line 277, I was wondering why ‘depth and the training time allow for greater expressivity’. It seems to me that only the Lipchitz constant is given in Figure 4, which I believe can be thought as a measure of the training stability. On the other hand, I do not seem to see which quantity measures the expressivity. In particular, I am not sure why longer training time could allow for better expressivity since it does not change the architecture of the network.\n", "limitations": "The limitations of the work are clearly discussed. This is a theoretical work thus I don’t see any potential negative social impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\n1. The lower bound on Lipchitz constant (Theorem 1) and TV/KL distance (Theorem 4/7) seems to be new.\n\nQuality\n1. The overall quality of this paper is good. Theoretical results are provided with proof sketch. Detailed proofs are provided in the appendix. Limitations are clearly discussed.\n\nClarity\n1. The paper is overall well-written and easy-to-follow.\n\nSignificance\n1. Studying the expressivity and stability of push forward models is an important problem.\n2. These theoretical results suggest a fundamental limitation of ‘push-forward’ generative models like GAN and VAE to achieve both good expressivity and stability (Lipschitz constant) for multimodal distribution. In particular, it shows that if the Lipchitz constant is small, then the model cannot recover the target distribution.\n3. Recently scored-based generative models/diffusion models are becoming popular. Though the paper does not study these models in the theoretical analysis, some experiments on simple datasets are provided to show the differences between diffusion models and ‘push-forward’ models.\n\nMinor\n1. Line 33, dimension d -> dimension p?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e5vJrTzG8E", "reviewer": "Reviewer_k9qb", "summary": "This paper derives a lower bound for the total deviation distance, and KL-divergence between the distribution learnt by push-forward generative models (VAEs-GANs), and the true data distribution. According to their lower bound (corollary 6), the lower bound would be larger for more multi-modal distributions, but the Lipschitz constant is also a determinining factor, in that a larger Lipschitz constant should be able to compensate for the large degree of multi-modality. \n\nThe authors also concretize their theoretical findings by providing three main experiments. \n\t\n1 - They train GAN, VAE, Score-based generative model on 1D-Mixture of Gaussian data. They find that VAEs consistently generate significant amount of data in the region in-between modes, due to its inflexibility reflected by its small Lipschitz constant. GANs are more flexible with a larger Lipschitz constant, but they fail to capture the whole distribution. Score-based models however better capture the distribution, even though they have smaller Lipschitz constant than GANs as shown in Figure 2. \n\t\t\n2 - They train the same three models, on a 784 dimensional-GMM data where means are located on two specific MNIST-digits, from the classes 3 and 7. We see that the behavior is pretty much the same as the 1D GMM data, except the score based model is not able to learn the correct proportion for each mode this time. \n\t\t\n3 - They train the same three models on the all 3-7 digits from the MNIST dataset. We now actually see that the data is less multi-modal, and therefore VAEs and GANs work better. \n\n", "questions": "As you pointed out in the discussion section, there also exists techniques that use learnable / multimodal prior distributions. And this significantly alleviates the burden on the decoder / encoder. Given this, I wonder how relevant is your lower bound for generative models that push a unimodal prior distribution through the decoder. Do you think the theory that you provide in this paper could be extended to models with multimodal priors (e.g. GMM priors)?\n\nI haven't checked all the details of the provided theorems and corollaries, but unlike corollary 6 where you provide a lower bound total variation distance, in theorem 7 the result does not seem to be final. Meaning, you still have the Borel sets in the result, and so it is not immediately clear to me what does that imply in practice. Do you think it would be possible to provide a result similar to corollary 6, in theorem 7 as well? Even considering a special case where the data distribution is Gaussian could be an improvement in my opinion. \n\n", "limitations": "I think the main limitation of the theory provided in this paper is due to that fact that it does not consider multi-modal priors. I think that multi-modal priors are crucial to mitigate the stiffness of VAEs. If however, the theory in this paper could be extended for multi-modal priors, it could be a nice first step. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\t\nThis paper provides potentially useful insights on the relationship between the level of multimodality and the expressivity of the push-forward generative models. \n\t\t\t\nI think that the numerical experiments really help to consolidate the message regarding multimodality and the behavior of the three main types of generative model learning methods. \n\n\nWeaknesses: \n\nFor a non-theory researcher working with generative models, I think that this paper could be useful to read. However, I also think that section 3 could be improved for reader-friendliness. My suggestion would be to clearly highlight the main results. (Corollary 6 and Theorem 7, right?) It might be useful to highlight the main result (the actual lower bound as opposed to alluding to it with text) in the introduction actually.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MaXTGQQCDYk", "reviewer": "Reviewer_tYcL", "summary": "The paper studies Lipschitz lower bound of push-forward mappings on expression multi-modal target distributions. A particular focus is put on two-modal mixture distributions. The major discovery is the connection between how well a push-forward mapping represents a target mixture distribution and the mapping's Lipschitz continuity. Besides push-forward distribution generation, in experiments, score-based models are tested and compared with GANs and VAEs; experimental results support theory.", "questions": "Theories in the paper appear to be sound and correct. The following questions aim for a better understanding of interpretations and implications.\n\n1. The dimension of latent distribution does not directly interact with the Lipschitz continuity of push-forward maps. Is it true that multi-dimensional latent distribution does not help in generating high-dimensional distributions? As Corollary 3 suggested, the continuity of any push-forward mapping cannot be better than the transport plan on transforming a univariate distribution. This seems discouraging.\n\n2. Is the lower bound in Theorem 1 tight?\n\n3. Can we discuss the scale of total variation lower bound and KL lower bound? For example, in Corollary 5, if the distance between the two manifold is $o(1)$, then the total variation lower bound is almost trivial, as the right-hand side is approximately 0, as long as ${\\rm Lip}(g) = O(1)$. On the other hand, when dimension $d$ is large, the distance function can easily be very large, in the order of $O(\\sqrt{d})$. This suggests the Lipschitz constant of $g$ should be at least $\\sqrt{d}$ for a successful distribution generation.\n\nSome additional questions for the experiments:\n\n1. Is there any trend that increasing the Lipschitz continuity gives rise to better distribution generation? Figure 3 demonstrates a comparison. However, it is hard to tell whether the generated distribution becomes better, especially for $L = 15$ and $L = 25$. We see a clear trend of capturing modes though.\n\n2. I feel like the experiments on score-based models does not help to support and clarify the theory, if I am not misunderstanding.", "limitations": "Authors already mentioned the tightness of total variation and KL lower bounds. I am mainly concerned with practical implications.\n\n1. There is no clear message of how the Lipschitz continuity lower bound helps to improve performance of GANs in practice.\n\n2. Theories focus on a relatively restrictive setting. In practice, different modes can overlap and there are often more than 2 modes.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written with few minor typos. The theoretical results are interesting and to the best of my knowledge, are new, at least newly applied for analyzing push-forward distribution generation. I like the presentation of the results, beginning with a useful technical theorem and then applying it to different instances. Experimental results are neatly summarized and provide support to theory.\n\nNo outstanding weaknesses as one may perceive; I do have some questions in the following section. I will also comment on limitations in the limitation section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MBf6-2YU8Gm", "reviewer": "Reviewer_K1r7", "summary": "This paper focuses on the expressivity of \"push-forward\" generative models trained on multimodal distributions. Specifically, a series of bounds are derived to relate the Lipschitz constant of a generative model to the total variation distance and the KL divergence between the generated and target distributions. Empirical analyses based on synthetic data and MNIST confirm the theoretical results.", "questions": "It would be interesting to see experiments on more realistic datasets, as well as on more diverse generative models, such as energy-based models.", "limitations": "The authors have addressed a potential limitation in their theoretical analysis.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nWhile the main claim of this paper is intuitive and somewhat expected, it is always good to formalize the intuition. Moreover, the theoretical analyses are thorough and well-presented, providing bounds for both the Lipschitz constant and the probability distance measures.\n\nWeaknesses\n\nIt seems the limited expressivity of push-forward generative models results from using simple feedforward NNs for generators, rather than from the VAE and GAN objectives. This distinction should be further discussed, and could be verified experimentally by using more expressive generator architectures.\n\nThe empirical study only uses simple synthetic data and the MNIST dataset, thus it is not clear how well the results will generalize to more realistic datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HdQbQI19kR3", "reviewer": "Reviewer_Wby3", "summary": "The paper derives a lower bound of the Lipschitz constant of the push-forward function between the standard Gaussian distribution and the data measure. Or, in a complementary way, it provides a bound on the \"distance\" between the model distribution (defined by the standard Gaussian prior and the push-forward function) and data measure.\n\nThe paper further considers a mixture of two Gaussians to calculate the bound explicitly and vadiliate the theorom. \n\nThe paper evaluates VAE, GAN and SBM on mixtures of two Gaussians to explain certain known behavoiur of such models, e.g., mode collapse and so on.", "questions": "See weakness.", "limitations": "The limitations of the paper mentioned in weakness should be discussed.\n\nThis is a theory paper and I did not see a direct negative societal impact.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I'm not that excited about the paper and I have the following concerns.\n\n1. Generally, the results of Theorem 1 are not surprising. As far as I know, the role of Lipschitz constant in the expressiveness of deep generative models has been extensively studied, e.g. [1*] and many others.\n\n2. The implications of the results are not new. Theorem 2-7 are either hard to evaluate or limited to the oversimplified mixture of two Gaussians. I did not see clear guidance on how to design a push-forward generator given practical assumptions of data. Besides, the behavior explained by this paper has been widely studied and explained from different perspectives. I did not see a particularly interesting point or obtain new insights from the theory. \n\n3. According to my understanding, VAE uses a stochastic decoder and I'm not sure whether it can be formulated as a push-forward function.\n\n4. I'm a little bit confusing about evaluating score-based models as well. Do such models fit the theory? The excellent empirical results of such models on a mixture of two Gaussians are not surprising given the fact that they are proven effective on much more complex real data. I don't think the theory of the paper explains the empirical results either.\n\n[1*]  implicit normalizing flow.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept