Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The authors build on the work of Tang et al. (2017), who made a minor change to the skip-thought model by decoding only the next sentence, rather than the previous one also. The additional minor change in this paper is to use a CNN, rather than RNN, decoder.

I am sympathetic to the goals of the work, and believe this sort of work should be carried out, but I see the contribution as too minor to constitute a paper at the conference track of a leading international conference such as ICLR. Given the incremental nature of the work, I think this would be a good fit for something like a short paper at *ACL.

I found the more theoretical motivation of the CNN decoder not terribly convincing, and somewhat post-hoc. I feel as though analogous arguments could just as easily be made for an RNN decoder. Ultimately I see these questions - such as CNN vs. RNN for the decoder - as empirical ones.

Finally, the authors have admirably attempted a thorough comparison with existing work, in the related work section, but this section takes up a large chunk of the paper at the end, and again I would have preferred this section to be much shorter and more concise.

Summary: worthwhile empirical goal, but the paper could have been easily written using half as much space.

review_title: The contribution of the paper is to extend Skip-thought by 1) decoding only one target sentence; 2) using a CNN decoder.

final_score: rating: 7: Good paper, accept
review_content: 
-- updates to review: --

Thanks for trying to respond to my comments. I find the new results very interesting and fill in some empirical gaps that I think were worth investigating. I'm now more confident that this paper is worth publishing and I increased my rating from 6 to 7. 

I admit that this is a pretty NLP-specific paper, but to the extent that ICLR has core NLP papers (I think it does have some), I think the paper is a reasonable fit for ICLR. It might feel more at home at a *ACL conference though. 

-- original review is below: --

This paper is about modifications to the skip-thought framework for learning sentence embeddings. The results show performance comparable to or better than skip-thought while decreasing training time. 

I think the overall approach makes sense: use an RNN encoder because we know it works well, but improve training efficiency by changing the decoder to a combination of feed-forward and convolutional layers. 

I think it may be the case that this works well because the decoder is not auto-regressive but merely predicts each word independently. This is possible because the decoder will not be used after training. So all the words can be predicted all at once with a fixed maximum sentence length. In typical encoder-decoder applications, the decoder is used at test time to get predictions, so it is natural to make it auto-regressive. But in this case, the decoder is thrown away after training, so it makes more sense to make the decoder non-auto-regressive.  I think this point should be made in the paper. 

Also, I think it's worth noting that an RNN decoder could be used in a non-auto-regressive architecture as well. That is, the sentence encoding could be mapped to a sequence of length 30 as is done with the CNN decoder currently; then a (multi-layer) BiLSTM could be run over that sequence, and then a softmax classifier can be attached to each hidden vector to predict the word at that position. It would be interesting to compare that BiLSTM decoder with the proposed CNN decoder and also to compare it to a skip-thought-style auto-regressive RNN decoder. This would let us understand whether the benefit is coming more from the non-auto-regressive nature of the decoder or from the CNN vs RNN differences. 

That is, it would make sense to factor the decision of decoder design along multiple axes. One axis could be auto-regressive vs predict-all-words. Another axis could be using a CNN over the sequence of word positions or an RNN over the sequence of word positions.  For auto-regressive models, another axis could be train using previous ground-truth word vs train using previous predicted word.  Skip-thought corresponds to an auto-regressive RNN (using the previous ground-truth word IIRC).  The proposed decoder is a predict-all-words CNN.  It would be natural to also experiment with an auto-regressive CNN and a predict-all-words RNN (like what I described in the paragraph above). The paper is choosing a single point in the space and referring to it as a "CNN decoder" whereas there are many possible architectures that can be described this way and I think it would strengthen the paper to increase the precision in discussing the architecture and possible alternatives. 

Overall, I think the architectural choices and results are strong enough to merit publication. Adding any of the above empirical comparisons would further strengthen the paper. 

However, I did have quibbles with some of the exposition and some of the claims made throughout the paper. They are detailed below:

Sec. 2:

In the "Decoder" paragraph: please add more details about how the words are predicted. Are there final softmax layers that provide distributions over output words? I couldn't find this detail in the paper. What loss is minimized during training? Is it the sum of log losses over all words being predicted?

Sec. 3:

Section 3 does not add much to the paper. The motivations there are mostly suggestive rather than evidence-based. Section 3 could be condensed by about 80% or so without losing much information. Overall, the paper has more than 10 pages of content, and the use of 2 extra pages beyond the desired submission length of 8 should be better justified. I would recommend adding a few more details to Section 2 and removing most of Section 3. I'll mention below some problematic passages in Section 3 that should be removed.

Sec. 3.2:
"...this same constraint (if using RNN as the decoder) could be an inappropriate constraint in the decoding process."  What is the justification or evidence for this claim?  I think the claim should be supported by an argument or some evidence or else should be removed. If the authors intend the subsequent paragraphs to justify the claim, then see my next comments. 

Sec. 3.2:
"The existence of the ground-truth current word embedding potentially decreases the tendency for the decoder to exploit other information from the sentence representation."
But this is not necessarily an inherent limitation of RNN decoders since it could be addressed by using the embedding of the previously-predicted word rather than the ground-truth word. This is a standard technique in sequence-to-sequence learning; cf. scheduled sampling (Bengio et al., 2015). 

Sec. 3.2: 
"Although the word order information is implicitly encoded in the CNN decoder, it is not emphasized as it is explicitly in the RNN decoder. The CNN decoder cares about the quality of generated sequences globally instead of the quality of the next generated word. Relaxing the emphasis on the next word, may help the CNN decoder model to explore the contribution of context in a larger space."
Again, I don't see any evidence or justification for these arguments. Also see my discussion above about decoder variations; these are not properties of RNNs vs CNNs but rather properties of auto-regressive vs predict-all-words decoders. 

Sec. 5.2-5.3:
There are a few high-level decisions being tuned on the test sets for some of the tasks, e.g., the length of target sequences in Section 5.2 and the number of layers and channel size in Section 5.3. 

Sec. 5.4:
When trying to explain why an RNN encoder works better than a CNN encoder, the paper includes the following: "We stated above that, in our belief, explicit usage of the word order information will augment the transferability of the encoder, and constrain the search space of the parameters in the encoder. The results match our belief."
I don't think these beliefs are concrete enough to be upheld or contradicted. Both encoders explicitly use word order information. Can you provide some formal or theoretical statement about how the two encoders treat word order differently? I fear that it's only impressions and suppositions that lead to this difference, rather than necessarily something formal. 

Sec. 5.4:
In Table 1, it is unclear why the "future predictor" model is the one selected to be reported from Gan et al (2017). Gan et al has many settings and the "future predictor" setting is the worst. An explanation is needed for this choice. 

Sec. 6.1: 

In the "BYTE m-LSTM" paragraph:

"Our large RNN-CNN model trained on Amazon Book Review (the largest subset of Amazon Review) performs on par with BYTE m-LSTM model, and ours works better than theirs on semantic relatedness and entailment tasks."  I'm not sure this "on par" assessment is warranted by the results in Table 2.  BYTE m-LSTM is better on MR by 1.6 points and better on CR by 4.7 points. The authors' method is better on SUBJ by 0.7 and better on MPQA by 0.5.  So on sentiment tasks, BYTE m-LSTM is clearly better, and on the other tasks the RNN-CNN is typically better, especially on SICK-r.  


More minor things are below:

Sec. 1:
The paper contains this: "The idea of learning from the context information was first successfully applied to vector representation learning for words in Mikolov et al. (2013b)"

I don't think this is accurate. When restricting attention to neural network methods, it would be more correct to give credit to Collobert et al. (2011). But moving beyond neural methods, there were decades of previous work in using context information (counts of context words) to produce vector representations of words. 

typo: "which d reduces" --> "which reduces"

Sec. 2:
The notation in the text doesn't match that in Figure 1: w_i^1 vs. w_1 and h_i^1 vs h_1. 

Instead of writing "non-parametric composition function", describe it as "parameter-free". "Non-parametric" means that the number of parameters grows with the data, not that there are no parameters. 

In the "Representation" paragraph: how do you compute a max over vectors? Is it a separate max for each dimension? This is not clear from the notation used.

Sec. 3.1:
inappropriate word choice: the use of "great" in "a great and efficient encoding model"

Sec. 3.2:
inappropriate word choice: the use of "unveiled" in "is still to be unveiled"

Sec. 3.4:
Tying input and output embeddings can be justified with a single sentence and the relevant citations (which are present here). There is no need for speculation about what may be going on, e.g.: "the model learns to explore the non-linear compositionality of the input words and the uncertain contribution of the target words in the same space".

Sec. 4:
I think STS14 should be defined and cited where the other tasks are described. 

Sec. 5.3:
typo in Figure 2 caption: "and and"


Sec. 6.1: 

In the "Skip-thought" paragraph:

inappropriate word choice: "kindly"

The description that says "we cut off a branch for decoding" is not clear to me. What is a "branch for decoding" in this context? Please modify it to make it more clear. 


References:

Bengio S, Vinyals O, Jaitly N, Shazeer N. Scheduled sampling for sequence prediction with recurrent neural networks. NIPS 2015.

Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P. Natural language processing (almost) from scratch. Journal of Machine Learning Research 2011.

review_title: Compelling results with a (relatively) efficient encoder-decoder architecture for sentence embedding learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update:

I'm going to change my review to a 6 to acknowledge the substantial improvements you've made—I no longer fear that there are major errors in the paper, but this paper is still solidly borderline, and I'm not completely convinced that any new claim is true. The evidence presented for the main claim—that you can get by without an autoregressive decoder when pretraining encoders—is somewhat persuasive, but isn't as unequivocal as I'd hope, and even if the claim is true, it is arguably too limited a claim for an ICLR main conference paper. As R1 says, a *ACL short paper would be more appropriate.  The writing is also still unclear in places.

----

This paper presents a new RNN encoder–CNN decoder hybrid design for use in pretraining reusable sentence encoders on Kiros's SkipThought objective. The task is interesting and important, and the results are generally good: The new model outperforms SkipThought, and all other prior models for training sentence encoders on unlabeled data. However, some of the design choices seem a bit odd, and I have a large number of minor concerns about the paper. I'd like to see the authors' replies and the other reviews before I can confidently endorse this paper as correct.


Non-autoregressive decoding with a CNN strikes me as a somewhat ill-posed problem, even for in this case where you don't actually use the decoder in the final application of your model. At each position, you're training your model to predict a distribution over all words that could appear at the beginning/tenth position/twentieth position in sentences on some topic. I'd appreciate some more discussion of why this should or shouldn't hurt performance. I'd be less concerned about this if the results supporting the use of the CNN decoder were a bit more conclusive: while they are better on average across your smaller experiments, your largest experiment (2400D) shows them roughly tied.

Your paper opens with the line "Context information plays an important role in human language understanding." This sounds like it's making an empirical claim that your paper doesn't support, but it's so vague that it's hard to tell exactly what that claim is. Please clarify this or remove it.

This sentence is quite inaccurate: "The idea of learning from the context information was first successfully applied to vector representation learning for words in Mikolov et al. (2013b) and learning from the occurrence of words also succeeded in Pennington et al. (2014)." Turney and Pantel 2010 ( https://www.jair.org/media/2934/live-2934-4846-jair.pdf ) offer a survey of the substantial prior work that existed at that time.

The "Neighborhood Hypothesis" is given quite a lot of importance, given that it's a fairly small empirical effect without any corresponding theory. The fact that it's emphasized so heavily makes me suspect that I can guess the author of the paper. I'd tone down that part of your framing.

I would appreciate some more analysis of which of the non-central tricks that you describe in section 3 help. For example, max pooling seems reasonable, but you report yourself that mean pooling generally works much better in prior work. Without an explicit experiment, it's not clear why you'd add a mean pooling component.

It seems misleading to claim that your CNN is modeled after AdaSent, as that model uses a number of layers that varies with the length of the sentence (and differs from yours in a few other less-important ways). Please correct or clarify.

The use of “†” in table to denote models that predict the next sentence in a sequence doesn't make sense. It should apply to all of your models if I understand correctly. Please clarify.

You could do a better job at table placement and formatting. Table 3 is in the wrong section, for example.

You write that: "Our proposed RNN-CNN model gets similar result on SNLI as Skip-thought, but with much less training time." This seems to be based on a comparison between your model run on your hardware and their model run on their (presumably older) hardware, and possibly also with their older version of CuDNN. If that's right, you should tone down this claim or offer some more evidence.
review_title: Good results, but tons of minor issues

Meta-review: here, yet another sentence representation method is proposed. i agree with R1 and R3 that this does not contribute significantly to be a full-length conference paper.
Decision: Reject