Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper argues that adding noise to the inputs of an adversarial autoencoder for text improves the geometry of the learned latent space (in terms of mapping similar input sentences to nearby points in the latent space). The authors present a mathematical argument for why adding noise to the inputs would enforce latent space structure while a vanilla autoencoder would have no preference over x-z mappings.

Overall, the paper addresses an important problem of improving autoencoder based generative models of text, presents a simple solution to do so and mathematically and empirically demonstrates its effectiveness. While I think that the benchmarks are somewhat artificial with small sentences and vocabulary sizes, I think the improvements demonstrated are substantial enough.

I have a few questions & comments

1) I’m curious about whether the authors have an intuition for why input space noise is better than latent space noise? Poole et al 2014 [1] showed that additive latent space gaussian noise in autoencoders is equivalent to a contractive autoencoder penalty and contractive autoencoders have an *explicit* penalty to encourage minimal change in z when changing x (i.e.) penalizing the norm of ||dz/dx||. Additive latent space noise appears to be a key ingredient to getting the ARAE and similar work like in Subramanian et al 2018 [2] to work. Was the LAAE implemented in the same framework as your DAAE?
2) It would be great to see Forward / Reverse PPL results on bigger datasets like the BookCorpus or WMT similar to [2].
3) You may be able to get similar reconstruction vs sample quality trade-offs with ARAEs by varying the variance of the gaussian noise, similar to LAAEs.
4) In Figure 3, I would really like to see how an autoencoder that isn’t a generative model performs. How well would a vanilla autoencoder or vanilla DAE perform? This is a cool setup to evaluate latent space representation quality - you could even consider running some of the SentEval probing tasks on these representations.
5) Could you use something like gradient-based latent space walks like in [2] to characterize the latent space geometry? https://arxiv.org/abs/1711.08014 also use similar gradient-based walks to characterize latent space smoothness in deep generative models. For example, if it takes 10 latent space gradient steps with a fixed learning rate for model “a” to turn sentence “x” into a *similar* sentence “y” but 20 steps for model “b”, then maybe “a” has smoother latent space geometry.
6) Theorem 1 - What if the set of zs isn’t unique and there is some sort of encoder collapse? Does this theorem still hold? (i.e.) there exists some set of points x_1, x_2 .. x_i \in x, that all map to z_k (and even potentially in the limit that all points in x map to the same point in z space).
7) It would be good to point out that the model presented in this work is far from SOTA on sentiment style transfer benchmarks like Yelp.

[1] Analyzing noise in autoencoders and deep networks - https://arxiv.org/pdf/1406.1831.pdf
[2] Towards Text Generation with Adversarially Learned Neural Outlines - https://papers.nips.cc/paper/7983-towards-text-generation-with-adversarially-learned-neural-outlines.pdf
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper "Denoising Improves Latent Space Geometry in Text Autoencoders" tackles the problem of text autoencoding in a space which respects text similarities. It is an interesting problem for which various attempts have been proposed, while still facing difficulties for encoding in smooth spaces. The paper proposes a simple (rather straightforward) approach based on adversarial learning, with some theoretical guarantees, which obtains good performances for reconstruction and neighborhood preservation. 

My main concern is about the missing of comparison with word dropout with variational encoding [Bowman et al., 2016], which also considers perturbations of the input texts to enforce the decoder to use the latent space. While the authors cite this work, I cannot understand why they did not include it in their experiments. 

Also, theorem 3 gives an upperbound of the achievable log-likelihood, which is "substantially better when examples in the same cluster are mapped to to points in the latent space in a manner that is well-separated from encodings of other
clusters". Ok but what does it show for the approach. If it was a lower-bound of the DAAE likelihood it would be interesting. But an upperbound ? In which sense does it indicate that it will be better than AAE ? Wouldn't it be possible to theoretically analyze other baselines? Also, all the theoretical analysis is made based on strong assumptions. Are these verified on considered datasets? 


Minor questions : 
          - In introduction of the experiments section, authors mention that they tried word removal and word masking. What is the difference ? 
           - what is the language model used for forward and reverse ppl ? 

   
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presented a denoising adversarial autoencoder for sentence embeddings. The idea is that by introducing perturbations (word omissions, etc) the embeddings are more meaningful and less "memorized". Evaluations include measuring sentence perplexity in generation/reconstruction, tense changing via vector arithmetic, sentiment changes via negative/positive vector additions, and sentence interpolations. 

Strengths: I thought the idea is nice, and the results do seem to show improvements in a number of interesting tasks. 

Weaknesses: I don't really think the explanation, especially in Theorem 1, makes a lot of sense. Qualitatively speaking, it's true that "memorization" in autoencoders (where the latent space has a 1-1 mapping with the input space) is problematic when the autoencoders are too powerful, but it is not always the case, and it is too far to say that the probability in theorem 1 is ALWAYS agnostic to encoding. The fact is word2vec works just fine with no perturbations, and there is no mathematical reason why sentence embeddings are fundamentally different. What is more accurate to say is that there is a tradeoff between model complexity and latent space representation usefulness, which is also related to the regularization/overfitting tradeoff in supervised learning. Here, injecting noise in the exact same fashion proposed in this paper is a well-accepted practice. While I think it's interesting that it works well here, I wouldn't really frame it as such a novelty, in that case, and I believe the other works on denoising autoencoders should be compared against in the experiments. In general, I find the mathematical claims a bit dubious, as a main assumption seems to be that the autoencoder itself is so overparametrized that it isn't really functioning as a representation-learning tool anyway. 

I also feel that the last experiment (referenced in the appendix) needs to go in the main text if we're to see it as a contribution. There is some wording that can be tightened in the main text, to make more room. 

Overall, I would improve my rating if the paper refocused more on the experiments, included more baselines (like other denoising autoencoders) and tasks that measure something besides perplexity (such as actual sentiment prediction, or machine translation, or other somewhat unrelated downstream tasks), and decreased the emphasis on the theoretical analysis--unless there is something I am significantly misunderstanding, it does not seem to be a particularly powerful theoretical contribution. 
review_title: Official Blind Review #2

Meta-review: This work presents a simple technique for improving the latent space geometry of text autoencoders. The strengths of the paper lie in the simplicity of the method, and results show that the technique improves over the considered baselines. However, some reviewers expressed concerns over the presented theory for why input noise helps, and did not address concerns that the theory was useful. The paper should be improved if Section 4 were instead rewritten to focus on providing intuition, either with empirical analysis, results on a toy task, or clear but high level discussion of why the method helps. The current theorem statements seem either unnecessary or make strong assumptions that don't hold in practice. As a result, Section 4 in its current form is not in service to the reader's understanding why the simple method works. 
Finally, further improvements to the paper could be made with comparisons to additional baselines from prior work as suggested by reviewers.
Decision: Reject