Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The topic is interesting however the description in the paper is lacking clarity. The paper is written in a procedural fashion - I first did that, then I did that and after that I did third. Having proper mathematical description and good diagrams of what you doing would have immensely helped. Another big issue is the lack of proper validation in Section 3.4. Even if you do not know what metric to use to objectively compare your approach versus baseline there are plenty of fields suffering from a similar problem yet  doing subjective evaluations, such as listening tests in speech synthesis. Given that I see only one example I can not objectively know if your model produces examples like that 'each' time so having just one example is as good as having none. 
review_title: Discrete Autoencoders for Sequence Models

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors describe a method for encoding text into a discrete representation / latent space. On a measure that they propose, they should it outperforms an alternative Gumbel-Softmax method for both language modeling and NMT.

The proposed method seems effective, and the proposed DSAE metric is nice, though it’s surprising if previous papers have not used metrics similar to normalized reduction in log-ppl. The datasets considered in the experiments are also large, another plus. However, overall, the paper is difficult to read and parse, especially since low-level details are weaved together with higher-level points throughout, and are often not motivated.

The major critique would be the qualitative nature of results in the sections on “Decipering the latent code” and (to a lesser extent) “Mixed sample-beam decoding.” These two sections are simply too anecdotal, although it is nice being stepped through the reasoning for the single example considered in Section 3.3. Some quantitative or aggregate results are needed, and it should at least be straightforward to do so using human evaluation for a subset of examples for diverse decoding.

review_title: Autoencoders for text with a new method for using discrete latent space

final_score: rating: 4: Ok but not good enough - rejection
review_content: This is an interesting paper focusing on building discrete reprentations of sequence by autoencoder. 
However, the experiments are too weak to demonstrate the effectiveness of using discrete representations.
The design of the experiments on language model is problematic.
There are a few interesting points about discretizing the represenations by saturating sigmoid and gumbel-softmax, but the lack of comparisons to benchmarks is a critical defect of this paper. 


Generally, continuous vector representations are more powerful than discrete ones, but discreteness corresponds to some inductive biases that might help the learning of deep neural networks, which is the appealing part of discrete representations, especially the stochastic discrete representations. 
However, I didn't see the intuitions behind the model that would result in its superiority to the continuous counterpart. 
The proposal of DSAE might help evaluate the usage of the 'autoencoding function' c(s), but it is certainly not enough to convince people. 
How is the performance if c(s) is replaced with the representations achieved from autoencoder, variational autoencoder or simply the sentence vectors produced by language model?
The qualitative evaluation on 'Deciperhing the Latent Code' is not enough either. 
In addition, the language model part doesn't sound correct, because the model cheated on seeing the further before predicting the words autoregressively.
One suggestion is to change the framework to variational auto-encoder, otherwise anything related to perplexity is not correct in this case.

Overall, this paper is more suitable for the workshop track. It also needs a lot of more studies on related work.
review_title: lack of valid experiments 

Meta-review: This paper presents a different method for learning autoencoders with discrete hidden states (compared to recent discrete-like VAE type models). The reviewers in general like the method being proposed and are convinced that there is worth to the underlying proposal. However there are several shared complaints about the setup and writing of the paper.

- Several reviewers complained about the use of qualitative evaluation, particularly in the "Deciphering the latent code" section of the paper.
- One reviewer in particular had significant issues with the experimental setup of the paper and felt that there was insignificant quantitative evaluation, particularly using standard metrics for the task (compared to the metric introduced in the paper).
- There were further critiques about the "procedural" nature of the writing and the lack of formal justifications for the ideas introduced. 
Decision: Reject