Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ==Update==

I appreciate the response, and continue to recommend acceptance. The evaluation metric used in this paper (SentEval) represents an important open problem in NLP—learning reusable sentence representations—and one of the problems in NLP best suited to presentation at IC*LR*. Because of this, I'm willing to excuse the fact that the paper is only moderately novel, in light of the impressive reported results.

While I would appreciate a direct (same codebase, same data) comparison with some outside baselines, this paper meets or exceeds the standards for rigor that were established by previous published work in the area, and the existing results are sufficient to support some substantial conclusions.

==========

This paper proposes an alternative formulation of Kiros's SkipThought objective for training general-purpose sentence encoder RNNs on unlabeled data. This formulation replaces the decoder in that model with a second encoder, and yields substantial improvements to both speed and model performance (as measured on downstream transfer tasks). The resulting model is, for the first time, reasonably competitive even with models that are trained end-to-end on labeled data for the downstream tasks (despite the requirement, imposed by the evaluation procedure, that only the top layer classifier be trained for the downstream tasks here), and is also competitive with models trained on large labeled datasets like SNLI. The idea is reasonable, the topic is important, and the results are quite strong. I recommend acceptance, with some caveats that I hope can be addressed.

Concerns:

A nearly identical idea to the core idea of this paper was proposed in an arXiv paper this spring, as a commenter below pointed out. That work has been out for long enough that I'd urge you to cite it, but it was not published and it reports results that are far less impressive than yours, so that omission isn't a major problem.

I'd like to see more discussion of how you performed your evaluation on the downstream tasks. Did you use the SentEval tool from Conneau et al., as several related recent papers have? If not, does your evaluation procedure differ from theirs or Kiros's in any meaningful way?

I'm also a bit uncomfortable that the paper doesn't directly compare with any baselines that use the exact same codebase, word representations, hyperparameter tuning procedure, etc.. I would be more comfortable with the results if, for example, the authors compared a low-dimensional version of their model with a low-dimensional version of SkipThought, trained in the *exact* same way, or if they implemented the core of their model within the SkipThought codebase and showed strong results there.

Minor points:

The headers in Table 1 don't make it all that clear which additions (vectors, UMBC) are cumulative with what other additions. This should be an easy fix. 

The use of the check-mark as an output in Figure 1 doesn't make much sense, since the task is not binary classification.

"Instead of training a model to reconstruct the surface form of the input sentence or its neighbors, our formulation attempts to focus on the semantic aspects of sentences. The meaning of a sentence is the property that creates bonds between a sequence of sentences and makes it logically flow." – It's hard to pin down exactly what this means, but it sounds like you're making an empirical claim here: semantic information is more important than non-semantic sources of variation (syntactic/lexical/morphological factors) in predicting the flow of a text. Provide some evidence for this, or cut it.

You make a similar claim later in the same section: "In figure 1(a) however, the reconstruction loss forces the model to predict local structural information about target sentences that may be irrelevant to its meaning (e.g., is governed by grammar rules)." This is a testable prediction: Are purely grammatical (non-semantic) variations in sentence form helpful for your task? I'd suspect that they are, at least in some cases, as they might give you clues as to style, dialect, or framing choices that the author made when writing that specific passage.

"Our best BookCorpus model (MC-QT) trains in just under 11hrs, compared to skip-thought model’s training time of 2 weeks." –  If you say this, you need to offer evidence that your model is faster. If you don't use the same hardware and low-level software (i.e., CuDNN), this comparison tells us nearly nothing. The small-scale replication of SkipThought described above should address this issue, if performed.

review_title: Great results, with minor concerns

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a framework for unsupervised learning of sentence representations by maximizing a model of the probability of true context sentences relative to random candidate sentences. Unique aspects of this skip-gram style model include separate target- and context-sentence encoders, as well as a dot-product similarity measure between representations. A battery of experiments indicate that the learned representations have comparable or better performance compared to other, more computationally-intensive models.

While the main constituent ideas of this paper are not entirely novel, I think the specific combination of tools has not been explored previously. As such, the novelty of this paper rests in the specific modeling choices and the significance hinges on the good empirical results. For this reason, I believe it is important that additional details regarding the specific architecture and training details be included in the paper. For example, how many layers is the GRU? What type of parameter initialization is used? Releasing source code would help answer these and other questions, but including more details in the paper itself would also be welcome.

Regarding the empirical results, the method does appear to achieve good performance, especially given the compute time. However, the balance between performance and computational complexity is not investigated, and I think such an analysis would add significant value to the paper. For example, I see at least three ways in which performance could be improved at the expense of additional computation: 1) increasing the candidate pool size 2) increasing the corpus size and 3) increasing the embedding size / increasing the encoder capacity. Does the good performance/efficiency reported in the paper depend on achieving a sweet spot among those three hyperparameters?

Overall, the novelty of this paper is fairly low and there is still substantial room for improvement in some of the analysis. On the other hand, I think this paper proposes an intuitive model and demonstrates good performance. I am on the fence, but ultimately I vote to accept this paper for publication.
review_title: Intuitive model for sentence representations with good performance

final_score: rating: 6: Marginally above acceptance threshold
review_content: [REVISION]

Thank you for your clarification. I appreciate the effort and think it has improved the paper. I have updated my score accordingly

====== 

This paper proposes a new objective for learning SkipThought-style sentence representations from corpora of ordered sentences. The algorithm is much faster than SkipThoughts as it swaps the word-level decoder for a contrastive classification loss. 

Comments:

Since one of the key advantages of this method is the speed, I was surprised there was not a more formal comparison of the speed of training different models. For instance, it would be more convincing if two otherwise identical encoders were trained on the same machine on the books corpus with the proposed objective and the skipthoughts decoding objective, and the representations compared after X hours of training. The reported 2 weeks required to train Skipthoughts comes from the paper, but things might be faster now with more up-to-date deep learning libraries etc. If this was what was in fact done, then it's probably just a case of presenting the comparison in a more formal way. I would also lose the sentence "we are able to train many models in the time it takes to train most unsupervised" (see next point for reasons why this is questionable).

It would have been interesting to apply this method with BOW encoders, which should be even faster than RNN-based encoders reported in this paper. The faster BOW models tend to give better performance on cosine-similarity evaluations ( quantifying the nearest-neighbour analysis that the authors use in this paper). Indeed, it would be interesting (although of course not definitive) to see comparison of the proposed algorithm (with BOW and RNN encoders) on cosine sentence similarity evaluations. 

The proposed novelty is simple and intuitive, which I think is a strength of the method. However, a simple idea makes overlap with other proposed approaches more likely, and I'd like the author to check through the public comments to ensure that all previous related ideas are noted in this paper. 

I think the authors could do more to emphasise what the point is of trying to learn sentence embeddings. An idea of the eventual applications of these embeddings would make it easier to determine, for instance, whether the supervised ensembling method applied here would be applicable in practice. Moreover, many papers have emphasised the limitations of the evaluations used in this paper (although they are still commonly used) so it would be good to acknowledge that it's hard to draw too many conclusions from such numbers. That said, the numbers are comparable Skipthoughts, so it's clear that this method learns representations of comparable quality. 

The justification for the proposed algorithm is clear in terms of efficiency, but I don't think it's immediately clear from a semantic / linguistic point of view. The statement "The meaning of a sentence is the property that creates bonds...." seems to have been cooked up to justify the algorithm, not vice versa. I would cut all of that speculation out and focus on empirically verifiable advantages. 

The section of image embeddings comes completely out of the blue and is very hard to interpret. I'm still not sure I understand this evaluation (short of looking up the Kiros et al. paper), or how the proposed model is applied to a multi-modal task.

There is much scope to add more structured analysis of the type hinted by the nearest neighbours section. Cherry picked lists don't tell the reader much, but statistics or more general linguistic trends can be found in these neighbours and aggregated, that could be very interesting. 


review_title: An elegant and simple alternative to existing methods, but empirical advantages are unclear 

Meta-review: Though the approach is not terribly novel, it is quite effective (as confirmed on a wide range of evaluation tasks). The approach is simple and likely to be useful in applications. The paper is well written.

+ simple and efficient
+ high quality evaluation
+ strong results
- novelty is somewhat limited

Decision: Accept (Poster)