Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This is a very interesting paper, I believe, a solid contribution to Variational Autoencoders. The basic argument is that encoders in VAEs are highly susceptible to noise in input data, whereas decoders are not. This argument is supported with a full fledged section 2.2, reformulating ELBO objective of VAEs, and introducing a VAE with discrete latent variables and discrete observations, so as to easily understand why and where VAEs fail.

To make encoders robust to noise in inputs, it is proposed to generate new fictive data points in the neighborhood of original data points so as to ensure that the latent representations of a data point and its fictive version are similar in "some sense" as part of the proposed regularization term. The implementation of this idea is solid in the paper, relating it to theoretical concepts such as  "entropy regularized entropy transport problem", "Wasserstein distance", etc. The most important point is that, it is easy to extend an encoder of an existing VAE with the proposed algorithm, while letting a decoder be untouched as the latter is shown to be robust/smooth anyways (in sec 2.2). It is also discussed on how to generate fictive samples, including but not restricted to approaches like projected gradient descent based adversarial attacks.

Section 2.2 can be improved further, in terms of presentation. This is the most important section which can be of interest to the community to understand VAEs' limitations, a good contribution on its own. Though challenging, I encourage the authors to improve the exposition in this section as much as possible.

Introduction is written beautifully. Good job, done!

For instance, some explanation about variables, m_j, u_i, their distribution.

How do you relate the Eq. 1 with the standard ELBO. (some reference to derivation?)

Is it not possible to explain limitations of present VAEs without introducing the particular von Mise like parameterization (last equation of page 3). I am not suggesting that you should remove it. The connections between the two could be more explicit, though I understand that it is already mentioned in the paper, "parameterization emulates a high capacity network that can model any functional relationship between latent states and observations...". 

In this context, I found the explanation after Eq. 2 to be intuitive in regards to inefficiency of encoders. If I understand correctly, to put it in even simpler terms, the encoding neural network is overfitting mapping from input data points to the latent representations, not performing any learning for the unseen data points at all; on the other hand, decoder explores the space of latent variables well because it is modeled as a Gaussian?

Some of the new equations should be numbered for easy reference. 

On page 4, the flow is a bit abrupt. Right after Fig. 3, there are points 1 and 2 added without any note on what these two points (items in latex) are about.

I found point 1 very confusing in page 4. On the other hand, point 2 is beautifully written. Though, it could be made explicit in the latter on why encoders found in VAE are not smooth, referring to Fig 2, 3.

There are minor grammar mistakes making some of sentences incoherent or confusing, in the paper. Something to do with style of language. I think, overall, language can be improved. Though, technical flow of the paper is great, and introduction is written very well, pointing out very important bold insights about the literature on unsupervised representation learning. I would say, it is a very well written paper, which is an enjoyable read, despite some of the grammar mistakes which can be easily fixed by proof reading.

Experimental evaluation is sufficient. 

Last but not the least, one could argue that we are going to the literature of kernel function based methods, or markov random fields, to improve the neural network models. This is a general trend we are observing. It is interesting to see new models such as the proposed one, getting the best from both worlds. It may be worthwhile to point  out something along these lines in the paper so that other works like this can be accomplished which are bold, and advance representation learning, digging mathematical concepts from diverse domains. If I am mistaken, please feel free to point out. It is not going to be change the review. I am inspired from this work.
 
One practical challenge is to generate fictive data points which are not very near to existing data points. I am not sure if GANs can achieve that, either. Having such points is critical to deal with more structured noise. Any comments on this? 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper analyzes the shortcoming of VAE objective, and propose a regularization method based on a selection mechanism that creates a fictive data point by explicitly perturbing an observed true data point. It is lead to Wasserstein distance between representations. Experiments are made on three datasets; ColorMNIST, MNIST, and CelebA, which shows superior performance on adversarial accuracy while similar accuracy to VAE on nominal accuracy.
The paper is well-organized and well-written. The point is clear and the proposed algorithm is valid. The only problem of the paper is the improvement on the experiment is marginal. Although adversarial accuracy is far better (like 0% vs 50%), it is apparent that the vanilla VAE is fragile to the adversarial examples because the added noise is intended so. Thus I can not say this is a fair comparison and because the superiority of the proposed algorithm is shown in only this point, I am not sure the proposed algorithm is surely useful. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studies the vulnerability of representations learned by variational auto-encoders (VAE). It first show that the learned representation of VAE is susceptible to small changes, similar to the adversarial examples in supervised learning setting. Then propose a regularization method, called smooth encoder, to improve the robustness of the representation. Experiments are conducted on several benchmark datasets to show the effectiveness of the method. 

Overall I find the idea interesting and the experimental results promising. The following are my detailed comments.

a About the theory
The illustration of the problem in VAE is interesting. However, one missing point is to theoretically quantify the effect of the proposed regularization (in some simple cases). In particular, it is claimed that the regularization could make the encoder smoother and the experimental results clearly justifies it. What would be better is to show in which sense/measure the encoder is smoother and provide some theoretical guarantee about it. (for instance smaller Lipschitz constant?)

b About the Experiment
The experimental section is clear and promising. I just have one question about the evaluation on the robustness of the VAE representation. In particular, a linear classifier is concatenated right after the VAE representation and it is not clear to me where it is concatenated. Is it right after the layer of \mu and \Sigma or in later layers? If it is in the later layers, the VAE is outputting a distribution, then how does the accuracy measured?

Minor comment:
I think it is unnecessary to introduce the new term selection strategy because it is just an adversarial training with respect to a different loss. In particular, the loss is the Wasserstein distance between the latent space vectors instead of a supervised loss. For simplicity, it could be just named as latent space adversarial training. (this is just a suggestion, which will not change my decision)
review_title: Official Blind Review #3

Meta-review: This paper proposes an novel way of expanding our VAE toolkit by tying it to adversarial robustness. It should be thus of interest to the respective communities.
Decision: Accept (Poster)