Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper analyzes the statistical properties of WAE, providing theoretical guarantees for both sample reconstruction and latent distribution tasks. To do so, the authors use the family of so-called f-Wasserstein autoencoders due to the specific choice of discrepancy selected for measuring the target and latent distribution. Here total variation distance is chosen. After setting up necessary preliminaries, and assumptions the authors prove the first of two main theorems, consistency in estimating the latent space distribution. Under suitable conditions, the second theorem provides a guarantee regarding the regeneration. Since they analysis was conducted using TV, which upper bounds the Jenson-Shannon divergence, the final corollary transfers the regeneration guarantees to WAEs. Fairly gentle assumptions and interesting theoretical results are discussed in detail. 
main_review: Originality: 
This paper offers novel analysis of the simultaneous tasks of WAE, and reveled new statistical guarantees for latent space consistency and regeneration quality. To my knowledge such an analysis has not been done. The related work seems appropriately cited. 

Quality: 
The analysis presented here doesn’t appear to have technical issues. The work is self-contained, and the authors appear to fairly describe where any limitations may lie. 
 
Clarity: 
The submission is clearly written and well organized though some polishing could be done. Line 82-84 reads awkwardly, and I’m not sure it’s fair to say VAEs haven’t had much theoretical work done. Nevertheless, the logic from lemmas to theorems is nicely built up, with discussions where needed. I enjoyed reading it.

Significance:
Given the pace at which the AI community advances, theoretical work is often playing catch up. Results like the ones presented here are a delight, and I’d imagine others may find them interesting as well.  While it’s difficult to determine whether it advances state-of-the-art (no empirical results), it principally offers statistical guarantees of WAE using VC theory. I believe the approach is of significance and that the NeurIPS community would benefit from the contributions presented in this paper. 

Typo: 
Equation 10, the last parenthesis under the square root should be on the outside.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The submission aims to provide rigorous theoretical results for the Wasserstein Autoencoder (WAE) - A model that has achieved great empirical success yet remains severely understudied. In particular, their main contribution in showing learning theoretic bounds such as complexity and convergence guarantees. As a feature of the Autoencoder set-up, they introduce the notion of statistical regeneration and go out to prove this statement for their learning setting.
main_review: The main strength of this paper is the problem they tackle which is an extremely topical issue with little theoretical studies, especially for the autoencoder settings. The contribution of the paper is original and makes important and clear references to related work when appropriate. The significance of the paper is also clear by virtue of the importance of the problem.

In particular, the authors establish a connection to the VC-dimension of the encoder distribution classes which is an interesting manifestion of complexity in the autoencoder setting. This opens up fruitful future work in the direction of studying autoencoders and generative models are large from a learning theoretic perspective. The work also establishes the notion of statistical regeneration, which although somewhat not interpretable, can be an important step for understanding the generalization performance of autoencoder models.

The clarity of the paper can definitely be improved as I found it quite challenging to follow notation and had to refer to external references to better understand some definitions. For example, in Assumption 3, I could not properly understand what is this space \mathcal{Y}( \mathscr{P}(\mathcal{C})) formally? I understand it refers to the choice of models however I would like to better understand what this function class is precisely. However, I understand that the area of generative modelling and autoencoders are quite notation dense, given the differing ways they are defined so this is not so much an issue for a paper with such results.

Some questions I had for the authors:

(1) One of the main interesting questions about WAE is how should one pick the divergence between distributions in the latent space. This paper comments on this however I was wondering if there is anything extremely actionable that you can comment? I am aware that the most common choices are MMD due to computational convenience and perhaps this has some stronger effect on the convergence?

(2) There is a common discussion in the autoencoder literature between the choice of lambda (the regularization weighting) and 'disentanglement'. While disentanglement is not properly defined, I wonder if the results of your work can establish this formally from the perspective of statistical regeneration since it seems to be intimately linked with the latent space. Do you have any comments on this?


limitations_and_societal_impact: Yes they have.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper provides a statistical guarantee of f-WAEs on learning the latent distribution by VC theory.
main_review: This paper is well-written with accurate and proper definition on related notations and probability measures which is appreciated. 

The author studied statistical properties of f-WAE on learning latent space and reconstruction. This is the first kind of such result from my knowledge. 

My main concern is about the implication of those theoretical results.

--- The current theoretical result is a bit rough and hard for the practitioner to gain some insights why WAE is better than vanilla VAE and AEs. 

--- The main theoretical results in Theorems 1 and 2 are independent of training and specific model architecture. For example, whether the VC-dimension will be related to the depth and width of the networks? In Theorem 1, how large lambda^* it is? Is that an absolute constant or can be arbitrarily large? Do you have any model assumption on the networks? For me, assumption 4 is a bit strong.  I suggest the author could discuss under what cases it can be satisfied.
limitations_and_societal_impact: It's also better to discuss if there is any technical contribution. For the proof of Theorem 1, it looks naive and just uses the basic concentration. In assumption 4, "$E_{\#}\mu$ is easy to draw sample from". This is not a rigorous statement.
review_title: null

Meta-review: All reviewers liked the paper and support acceptance. So it is a clear accept.

The reviewers provided good feedback on shortcomings in the presentation. The authors are strongly encouraged to take these comments into account for the final version of the paper.
Decision: Accept (Spotlight)