Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies Lipschitz lower bound of push-forward mappings on expression multi-modal target distributions. A particular focus is put on two-modal mixture distributions. The major discovery is the connection between how well a push-forward mapping represents a target mixture distribution and the mapping's Lipschitz continuity. Besides push-forward distribution generation, in experiments, score-based models are tested and compared with GANs and VAEs; experimental results support theory.
strengths_and_weaknesses: The paper is well written with few minor typos. The theoretical results are interesting and to the best of my knowledge, are new, at least newly applied for analyzing push-forward distribution generation. I like the presentation of the results, beginning with a useful technical theorem and then applying it to different instances. Experimental results are neatly summarized and provide support to theory.

No outstanding weaknesses as one may perceive; I do have some questions in the following section. I will also comment on limitations in the limitation section.
questions: Theories in the paper appear to be sound and correct. The following questions aim for a better understanding of interpretations and implications.

1. The dimension of latent distribution does not directly interact with the Lipschitz continuity of push-forward maps. Is it true that multi-dimensional latent distribution does not help in generating high-dimensional distributions? As Corollary 3 suggested, the continuity of any push-forward mapping cannot be better than the transport plan on transforming a univariate distribution. This seems discouraging.

2. Is the lower bound in Theorem 1 tight?

3. Can we discuss the scale of total variation lower bound and KL lower bound? For example, in Corollary 5, if the distance between the two manifold is $o(1)$, then the total variation lower bound is almost trivial, as the right-hand side is approximately 0, as long as ${\rm Lip}(g) = O(1)$. On the other hand, when dimension $d$ is large, the distance function can easily be very large, in the order of $O(\sqrt{d})$. This suggests the Lipschitz constant of $g$ should be at least $\sqrt{d}$ for a successful distribution generation.

Some additional questions for the experiments:

1. Is there any trend that increasing the Lipschitz continuity gives rise to better distribution generation? Figure 3 demonstrates a comparison. However, it is hard to tell whether the generated distribution becomes better, especially for $L = 15$ and $L = 25$. We see a clear trend of capturing modes though.

2. I feel like the experiments on score-based models does not help to support and clarify the theory, if I am not misunderstanding.
limitations: Authors already mentioned the tightness of total variation and KL lower bounds. I am mainly concerned with practical implications.

1. There is no clear message of how the Lipschitz continuity lower bound helps to improve performance of GANs in practice.

2. Theories focus on a relatively restrictive setting. In practice, different modes can overlap and there are often more than 2 modes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the expressivity and stability of generative model that transforming a gaussian to data (such as GAN and VAE). In particular, the authors showed that there is a tradeoff between the approximation power and the stability (in terms of Lipchitz constant) of training for multimodal distribution. Experiment results are provided to validate the theoretical results.
strengths_and_weaknesses: Originality
1. The lower bound on Lipchitz constant (Theorem 1) and TV/KL distance (Theorem 4/7) seems to be new.

Quality
1. The overall quality of this paper is good. Theoretical results are provided with proof sketch. Detailed proofs are provided in the appendix. Limitations are clearly discussed.

Clarity
1. The paper is overall well-written and easy-to-follow.

Significance
1. Studying the expressivity and stability of push forward models is an important problem.
2. These theoretical results suggest a fundamental limitation of ‘push-forward’ generative models like GAN and VAE to achieve both good expressivity and stability (Lipschitz constant) for multimodal distribution. In particular, it shows that if the Lipchitz constant is small, then the model cannot recover the target distribution.
3. Recently scored-based generative models/diffusion models are becoming popular. Though the paper does not study these models in the theoretical analysis, some experiments on simple datasets are provided to show the differences between diffusion models and ‘push-forward’ models.

Minor
1. Line 33, dimension d -> dimension p?

questions: I would appreciate if the following questions could be answered:
1. In line 100 it is mentioned that in most of the time ‘p is almost always smaller than d in practice’. However, for corollary 2 and 3, in line 135 it is mentioned that p >= d. I was wondering if authors could comment on this.
2. Influences of model size and time of training paragraph in the experiment section: in line 277, I was wondering why ‘depth and the training time allow for greater expressivity’. It seems to me that only the Lipchitz constant is given in Figure 4, which I believe can be thought as a measure of the training stability. On the other hand, I do not seem to see which quantity measures the expressivity. In particular, I am not sure why longer training time could allow for better expressivity since it does not change the architecture of the network.

limitations: The limitations of the work are clearly discussed. This is a theoretical work thus I don’t see any potential negative social impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the expressivity of "push-forward" generative models trained on multimodal distributions. Specifically, a series of bounds are derived to relate the Lipschitz constant of a generative model to the total variation distance and the KL divergence between the generated and target distributions. Empirical analyses based on synthetic data and MNIST confirm the theoretical results.
strengths_and_weaknesses: Strengths

While the main claim of this paper is intuitive and somewhat expected, it is always good to formalize the intuition. Moreover, the theoretical analyses are thorough and well-presented, providing bounds for both the Lipschitz constant and the probability distance measures.

Weaknesses

It seems the limited expressivity of push-forward generative models results from using simple feedforward NNs for generators, rather than from the VAE and GAN objectives. This distinction should be further discussed, and could be verified experimentally by using more expressive generator architectures.

The empirical study only uses simple synthetic data and the MNIST dataset, thus it is not clear how well the results will generalize to more realistic datasets.
questions: It would be interesting to see experiments on more realistic datasets, as well as on more diverse generative models, such as energy-based models.
limitations: The authors have addressed a potential limitation in their theoretical analysis.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper derives a lower bound for the total deviation distance, and KL-divergence between the distribution learnt by push-forward generative models (VAEs-GANs), and the true data distribution. According to their lower bound (corollary 6), the lower bound would be larger for more multi-modal distributions, but the Lipschitz constant is also a determinining factor, in that a larger Lipschitz constant should be able to compensate for the large degree of multi-modality. 

The authors also concretize their theoretical findings by providing three main experiments. 
	
1 - They train GAN, VAE, Score-based generative model on 1D-Mixture of Gaussian data. They find that VAEs consistently generate significant amount of data in the region in-between modes, due to its inflexibility reflected by its small Lipschitz constant. GANs are more flexible with a larger Lipschitz constant, but they fail to capture the whole distribution. Score-based models however better capture the distribution, even though they have smaller Lipschitz constant than GANs as shown in Figure 2. 
		
2 - They train the same three models, on a 784 dimensional-GMM data where means are located on two specific MNIST-digits, from the classes 3 and 7. We see that the behavior is pretty much the same as the 1D GMM data, except the score based model is not able to learn the correct proportion for each mode this time. 
		
3 - They train the same three models on the all 3-7 digits from the MNIST dataset. We now actually see that the data is less multi-modal, and therefore VAEs and GANs work better. 


strengths_and_weaknesses: Strengths: 
	
This paper provides potentially useful insights on the relationship between the level of multimodality and the expressivity of the push-forward generative models. 
			
I think that the numerical experiments really help to consolidate the message regarding multimodality and the behavior of the three main types of generative model learning methods. 


Weaknesses: 

For a non-theory researcher working with generative models, I think that this paper could be useful to read. However, I also think that section 3 could be improved for reader-friendliness. My suggestion would be to clearly highlight the main results. (Corollary 6 and Theorem 7, right?) It might be useful to highlight the main result (the actual lower bound as opposed to alluding to it with text) in the introduction actually.


questions: As you pointed out in the discussion section, there also exists techniques that use learnable / multimodal prior distributions. And this significantly alleviates the burden on the decoder / encoder. Given this, I wonder how relevant is your lower bound for generative models that push a unimodal prior distribution through the decoder. Do you think the theory that you provide in this paper could be extended to models with multimodal priors (e.g. GMM priors)?

I haven't checked all the details of the provided theorems and corollaries, but unlike corollary 6 where you provide a lower bound total variation distance, in theorem 7 the result does not seem to be final. Meaning, you still have the Borel sets in the result, and so it is not immediately clear to me what does that imply in practice. Do you think it would be possible to provide a result similar to corollary 6, in theorem 7 as well? Even considering a special case where the data distribution is Gaussian could be an improvement in my opinion. 


limitations: I think the main limitation of the theory provided in this paper is due to that fact that it does not consider multi-modal priors. I think that multi-modal priors are crucial to mitigate the stiffness of VAEs. If however, the theory in this paper could be extended for multi-modal priors, it could be a nice first step.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper derives a lower bound of the Lipschitz constant of the push-forward function between the standard Gaussian distribution and the data measure. Or, in a complementary way, it provides a bound on the "distance" between the model distribution (defined by the standard Gaussian prior and the push-forward function) and data measure.

The paper further considers a mixture of two Gaussians to calculate the bound explicitly and vadiliate the theorom. 

The paper evaluates VAE, GAN and SBM on mixtures of two Gaussians to explain certain known behavoiur of such models, e.g., mode collapse and so on.
strengths_and_weaknesses: I'm not that excited about the paper and I have the following concerns.

1. Generally, the results of Theorem 1 are not surprising. As far as I know, the role of Lipschitz constant in the expressiveness of deep generative models has been extensively studied, e.g. [1*] and many others.

2. The implications of the results are not new. Theorem 2-7 are either hard to evaluate or limited to the oversimplified mixture of two Gaussians. I did not see clear guidance on how to design a push-forward generator given practical assumptions of data. Besides, the behavior explained by this paper has been widely studied and explained from different perspectives. I did not see a particularly interesting point or obtain new insights from the theory. 

3. According to my understanding, VAE uses a stochastic decoder and I'm not sure whether it can be formulated as a push-forward function.

4. I'm a little bit confusing about evaluating score-based models as well. Do such models fit the theory? The excellent empirical results of such models on a mixture of two Gaussians are not surprising given the fact that they are proven effective on much more complex real data. I don't think the theory of the paper explains the empirical results either.

[1*]  implicit normalizing flow.




questions: See weakness.
limitations: The limitations of the paper mentioned in weakness should be discussed.

This is a theory paper and I did not see a direct negative societal impact.
review_title: null

Meta-review: metareview: This paper proposes a theory on the Lipschitz continuity of the pushforward mapping, which transforms a Gaussian distribution to a multimodal distribution. This is a borderline paper. The reviewers are generally positive about the paper and leaning towards acceptance, though there is still some notable gap between the proposed theory and empirical results. The authors are expected to further revise their paper based on the reviewers' suggestions.
confidence: Less certain
Decision: Accept