Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Often it's desirable that a representation have some invariance to irrelevant changes in the input -- skip-thought vectors in particular aim to factor out certain types of phrasing variations that are irrelevant to prediction of surrounding context. In this sense, the ability to exactly reconstruct the input sentence could be seen as a detriment for a representation, since it is keeping around information that may not be helpful for future tasks. In this work, the ability to perfectly encode content seems to be presented as a mostly unqualified win. Have the authors considered auxiliary tasks to measure this sort of beneficial invariance to e.g. paraphrasing?

I'm very surprised that the auxiliary classifiers can not be overfit on the training set since they are so nonlinear, especially when training on e.g. skip-thoughts which do not explicitly throw away any information -- can the authors give some intuition as to why this is the case?
review_title: Questions

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a set of experiments investigating what kinds of information are captured in common unsupervised approaches to sentence representation learning. The results are non-trivial and somewhat surprising. For example, they show that it is possible to reconstruct word order from bag of words representations, and they show that LSTM sentence autoencoders encode interpretable features even for randomly permuted nonsense sentences.

Effective unsupervised sentence representation learning is an important and largely unsolved problem in NLP, and this kind of work seems like it should be straightforwardly helpful towards that end. In addition, the experimental paradigm presented here is likely more broadly applicable to a range of representation learning systems. Some of the results seem somewhat strange, but I see no major technical concerns, and think that that they are informative. I recommend acceptance.

One minor red flag: 
- The massive drop in CBOW performance in Figures 1b and 4b are not explained, and seem implausible enough to warrant serious further investigation. Can you be absolutely certain that those results would appear with a different codebase and different random seed implementing the same model? Fortunately, this point is largely orthogonal to the major results of the paper.

Two writing comments:
- I agree that the results with word order and CBOW are surprising, but I think it's slightly misleading to say that CBOW is predictive of word order. It doesn't represent word order at all, but it's possible to probabilistically reconstruct word order from the information that it does encode.
- Saying that "LSTM auto-encoders are more effective at encoding word order than word content" doesn't really make sense. These two quantities aren't comparable. 
review_title: Interesting analytic results on unsupervised sentence encoders

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
The authors present a methodology for analyzing sentence embedding techniques by checking how much the embeddings preserve information about sentence length, word content, and word order. They examine several popular embedding methods including autoencoding LSTMs, averaged word vectors, and skip-thought vectors. The experiments are thorough and provide interesting insights into the representational power of common sentence embedding strategies, such as the fact that word ordering is surprisingly low-entropy conditioned on word content.

Exploring what sort of information is encoded in representation learning methods for NLP is an important and under-researched area. For example, the tide of word-embeddings research was mostly stemmed after a thread of careful experimental results showing most embeddings to be essentially equivalent, culminating in "Improving Distributional Similarity with Lessons Learned from Word Embeddings" by Levy, Goldberg, and Dagan. As representation learning becomes even more important in NLP this sort of research will be even more important.

While this paper makes a valuable contribution in setting out and exploring a methodology for evaluating sentence embeddings, the evaluations themselves are quite simple and do not necessarily correlate with real-world desiderata for sentence embeddings (as the authors note in other comments, performance on these tasks is not a normative measure of embedding quality). For example, as the authors note, the ability of the averaged vector to encode sentence length is trivially to be expected given the central limit theorem (or more accurately, concentration inequalities like Hoeffding's inequality).

The word-order experiments were interesting. A relevant citation for this sort of conditional ordering procedure is "Generating Text with Recurrent Neural Networks" by Sutskever, Martens, and Hinton, who refer to the conversion of a bag of words into a sentence as "debagging."

Although this is just a first step in better understanding of sentence embeddings, it is an important one and I recommend this paper for publication.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper analyzes various unsupervised sentence embedding approaches by means of a set of auxiliary prediction tasks. By examining how well classifiers can predict word order, word content, and sentence length, the authors aim to assess how much and what type of information is captured by the different embedding models. The main focus is on a comparison between and encoder-decoder model (ED) and a permutation-invariant model, CBOW. (There is also an analysis of skip-thought vectors, but since it was trained on a different corpus it is hard to compare).

There are several interesting and perhaps counter-intuitive results that emerge from this analysis and the authors do a nice job of examining those results and, for the most part, explaining them. However, I found the discussion of the word-order experiment rather unsatisfying. It seems to me that the appropriate question should have been something like, 'How well does model X do compared to the theoretical upper bound which can be deduced from natural language statistics?' This is investigated from one angle in Section 7, but I would have preferred to the effect of natural language statistics discussed up front rather than presented as the explanation to a 'surprising' observation. I had a similar reaction to the word-order experiments.

Most of the interesting results, in my opinion, are about the ED model. It is fascinating that the LSTM encoder does not seem to rely on natural-language ordering statistics -- it seems like doing so should be a big win in terms of per-parameter expressivity. I also think that it's strange that word content accuracy begins to drop for high-dimensional embeddings. I suppose this could be investigated by handicapping the decoder.

Overall, this is a very nice paper investigating some aspects of the information content stored in various types of sentence embeddings. I recommend acceptance.
review_title: Experimental analysis of unsupervised sentence embeddings

final_score: rating: null
review_content: Figure 2(b) shows the norm of CBOW embeddings decreasing very smoothly as the sentence length increases. Have you tried quantifying your explanation of this behavior based on the central limit theorem? If so, how well does it fit the curve?

What does the analogue of Figure 2(b) look like for the ED embeddings?
review_title: Encoding of sentence length in embedding norm

final_score: rating: null
review_content: – Could you explain your motivation for testing your models only on their training examples? You explain it briefly at the start of 3.1, but I can't reconstruct the argument.

– Have you done significance testing of any kind on any of your results? The slight dip in ED performance in Figure 1a, for example, look like it could be a random fluctuation, but you mention it in your summary of the results.

– What do you think might cause the massive drop in performance for CBOW in Figure 1b?
review_title: Detail questions

Meta-review: The area chair agrees with the reviewers and think this paper would be of interest to the ICLR audience. There is clearly more to be done in this area, but the authors do a good job shedding some light on what sentence embeddings can encode. We need more work like this that helps us understand what neural networks can model.
Decision: Accept (Poster)