Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the robustness of deterministic autoencoders by introducing a regularization scheme to incorporate adversarially perturbed data points into the training pipeline. The proposed method increases the generation and robustness of the learned representations without increasing the computational complexity.
strengths_and_weaknesses: Strengths:

1. The writing is good, and the paper is easy to follow.
2. The experiment part demonstrates the proposed method has a large improvement in robustness of the most time.

Weaknesses:

In the paper, the author may need to show some qualitative results.
questions: About the sentence in lines 45 and 46, ``However, regularizing the VAE objective ..., Hence, ... ''. The traditional methods lead to poor reconstruction fidelity, while you aim at maintaining the generation performance? Not both the reconstruction and generation quality of the image?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a technique to improve the robustness of VAEs. It builds upon earlier work by Saseendran et al, Ghosh et al. which provide a deterministic alternative to the variational formulation of VAEs. The formulation by Saseendran et al. uses the Kolmogorov-Smirnov (KS) test to regularize the latent space of deterministic auto encoders and impose a multi modal gaussian prior. 
This paper extends this idea by imposing the same multi modal gaussian prior on the pair ($z_n$, $z^{adv}_n$) where $z^{adv}_n$ is the adversarial latent code obtained via perturbation at the input image level (added before the encoder). This is as opposed to just imposing the prior on $z_n$ like in earlier work.
This means that the regularizer goes from: the one point KS test used in Saseendran et al. to a two sample KS test between latent codes and their adversarial versions.
The models used in the paper are VAE, beta VAE, Smooth Encoder, AVAE etc evaluated on MNIST, FashionMNIST, SVHN, CelebA. 
There are two attack vectors considered:
1. Latent space attack: adding perturbation to an image such that its latent code moves close to the latent code of a target image.
2. Maximum damage attack: adding perturbation to an image such that its reconstruction its now far away from its vanilla reconstruction
Results show that there is consistent improvement in robustness across the board. This is evidenced by close similarity scores between image reconstructions. 
strengths_and_weaknesses: Strengths:
1. Comprehensive Evaluation - The paper uses a variety of datasets to prove their point. A minor criticism could be that larger datasets are not considered (which is valid), however, datasets used in the paper are also used by prior work. 
2. Problem is well motivated and the exposition is clear 
3. Improving robustness in generative models is an important problem so this result is significant. Experiments show large improvements in FID so the results seem solid.


Weaknesses:
1. Unclear scope - How to pick modes to match the prior? What happens if they don't match?
2. Section 4 seems a little unclear in an otherwise well written paper.
3. The paper builds upon prior work in the area so the novelty of the technique is not very high. Having said that, it is an application of prior work into a new area which makes this point a less important.

questions: 1. A it stands, the paper does not have a single image about what samples look like or even what adversarial example would look like for the model. I think having pictures for these kind of papers make them read better.
2. I found the paper fairly clear overall except the part in the evaluation subsection of Section 4, where the authors introduce a bunch of symbols at the same time. I think having clear definitions for each of reference image ($x_r$), target image ($x_t$) and adversarial image ($x_a$) in addition to clarifying the relationship between each of them would clear things up. (Maybe a diagram/image?)
3. It seems that the number of modes in dataset has to match those in the prior. I would like to see some comments on this from the authors. What happens if the number of modes don’t match? We know from Saseendran et al. that image quality drops but does robustness also fall in a similar fashion? Or maybe robustness falls faster or slower than image quality?

Typo:
Ref [26] does not have paper name. Listed several times in the paper. Key reference.

limitations: I think the authors should list that the number of modes in the prior has to match the number of modes in the dataset as a limitation.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors extend existing methods for training variational autoencoders to be robust against adversarial attacks. They implement this by training on adversarial examples during training. They show that the resulting networks are less susceptible to adversarial attacks have better MSSSIM metrics and finally that the latent representations are better suited for downstream classification tasks.
strengths_and_weaknesses: Here I have to admit that I am not very familiar with the current literature and the math behind the statistics of variational autoencoders. I have some difficulties understanding the exact novelty of the approach the authors have taken. It seems the basic idea is to prevent adversarial input examples during training and ensure some statistical properties of these examples.
But I assume the method is explained well for a reader with the sufficient background.

What comes a bit short tough is the explanation of the experiments under the headers "Ablation Study" and "Robustness to downstream applications". There the authors seem to omit the details.

The results seem to lead to a significant improvement in the results.
questions: The authors claim that their method does not increase the computational complexity, yet they use adversarial examples during training. Should the generation of the adversarial examples not increase the computational complexity?

The authors should provide more details on the sections "ablation study" and "robustness to downstream applications". Here the text is not clear what they did. I do not understand what the "ablation study" is exactly about and for the downstream applications it might help to know what kind of MLP the authors are using.

The authors present how their method improves the robustness against adversarial attacks but they show little how this affects the baseline performance. How is the MSSSIM for x_r and x_r tilde? Does the new approach provide similar decoded image quality? Also would not MSSSIM for x_a and x_a tile be interesting, how well the adversarial images can be reconstructed by the autoencoder? 

The authors say that the adversarial samples should not explore unexplored regions of the latent space. I do not quite understand why this is beneficial. Shouldn't the newly generated samples try to explore more regions then the normal samples already explore?
limitations: The authors discuss the limitations of their work mainly in terms of computational power and additional hyper parameters that need to be tuned and how datasets with different distributions might not work as well with the approach.

The authors do not discuss potential negative societal impact of their work and just state N/A. Surely if their work is relevant, it should have a potential societal impact of which a part can also be negative. The authors could elaborate a bit on this to address this point.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes new approaches for improving the robustness of Variational Autoencoders (VAEs). The paper finds that deterministic autoencoder with multimodal prior (proposed in prior work) not only improves the fidelity of generated samples, but is also more robust to adversarial attacks. On top of that, the paper proposes to incorporate adversarial samples into the training process, and add regularizations to ensure that the latents from the adversarially perturbed samples also follow the same prior distribution. Results show that the proposed method achieves better fidelity and robustness across several datasets.
strengths_and_weaknesses: 
Originality: The key techniques of this paper (incorporating adversarial samples in training, KS test) are simple adaptations from prior work. However, this specific combination for improving the robustness of VAE is interesting.

Quality: Overall, the paper is technically sound, and most of the claims are well-supported by experiments.

Clarity: The paper is easy to read. However, there are some typos and some unclear statements. See questions below.

Significance: The paper could be of interest to practitioners as it shows that the proposed technique could improve the robustness without sacrificing fidelity.  

questions: 
I appreciate that the authors point out the downside of having a new hyperparameter alpha. However, I think it is still important to at least mention how alpha was tuned for all the experiments, and provide a sensitivity analysis of alpha. This would make it easier for people to adopt the method. I will adjust the score according to the response to this question.



My other questions are:

* Table 1: I don't understand how you get x_r and x_a for maximum damage attack, and why lower MSSSIM(x_r,x_a) is better. I could not find related explanations or descriptions in the main text.

* Line 158: z_n and z_n^{adv} are undefined. I suppose they are g(x_n) and g(x_n + \delta_{x_n})?

* Line 162: Does "ignoring cross-covariance between z_{1,...,n} and z_{1,...,n}^{adv}" mean the 0s in last matrix in Eq 8? Please make it more explicit

* Eq. 6: g is undefined. I suppose it means the encoder?



Some other minor typos and questions:

* The title of the citation [26] is missing.

* Figure 1 and its caption are hard to understand by themselves. It becomes clearer after I read Section 3.2, which appears much later in this paper.

* Should the denominators of Eq 8 and Eq 10 be 4D^2 instead of 2D^2?

* Line 171: left -> right

* Table 2: Consider putting the best metrics of each column in bold?



limitations: 
The paper adequately addressed the limitations. The paper did not discuss the potential negative societal impact of the work.
review_title: null

Meta-review: metareview: This paper received generally positive reviews that, after discussion, all backed acceptance.  The paper was praised for its empirical evaluations, potential significance, clarity, and applicability.  While some questions and lower-level issues were raised, I do not feel that the reviewers raised any significant issues that would be a barrier to acceptance, with the small number of issues that were raised well addressed by the authors' responses.

My own personal view of the work is also very positive (perhaps more so than the reviewers themselves): I think this is strong work that will be of significant interest to the community.  The empirical results are a particular highlight, both in terms of the performance shown, the comprehensive set of experiments considered, and the numerous and appropriate baselines compared to.  While I do have some suggestions and minor gripes (see below) that I would like addressed in the final version of the paper, I have no hesitation in enthusiastically recommending its acceptance.

Suggestions and minor issues:
- My most important complaint with the paper is that the title is too strong and overclaiming.  It suggests a trade-off will never occur and that the result applies to all deterministic auto-encoders, rather than the specific type considered.  I do not think either of these are true: just because robustness has been improved with minimal change in FID score, does not mean there will not be a trade-off with future developments (i.e. there may well be ways of improving the image quality of [26] that would no longer give good robustness when combined with the suggested approach). Please, therefore, change the title to something more measured and precise for the camera-ready paper.
- It would be good to provide more explicit timing information about the training times of all the different models, rather than just SE.  Claims are made in the intro and conclusions about speed, but, unless I have missed something, I did not really feel these were properly supported.
- While I think the paper is mostly quite clear and well written, I do think the writing could be improved in some of the key technical sections; I generally found Section 3 to be the worst written part of the paper.  In particular, I think more high-level explanation was required.  While the maths itself is not too difficult to follow, it took me quite a few reads through to get a feel for the intuitions.  To give an example of a specific issue, the right-hand side of Figure 1 comes too early and lacks context: the reader will naturally assume that they should be able to understand what is happening when the figure is first referenced, but actually they need to get to Section 3.3 first to get an idea of what is going on.
- It might be useful for the authors to have a look at https://openreview.net/forum?id=nzvbBD_3J-g because it actively argues against using GMM priors in the more conventional VAE setting, on the basis that such inductive bias can be more effectively be incorporated through a customised decoder architecture (and not treating the latents as the representation itself), than through regularisation.  Of course, their insights may well not carry over to the deterministic auto-encoder setting, but it does hint at an interesting alternative approach and may be worth discussing or at least acknowledging.
- Please increase the text size in the figures, these are very difficult to see at present.
confidence: Certain
Decision: Accept