Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ---- updates: ----

I had a ton of comments and concerns, and I think the authors did an admirable job in addressing them.  I think the paper represents a solid empirical contribution to this area and is worth publishing in ICLR. 

---- original review follows: ----

This paper is about learning sentence embeddings by combining a bunch of training signals: predicting the next & previous sentences (skip-thought), predicting the sentence's translation, classifying entailment relationships between two sentences, and predicting the constituent parse of a sentence. This is a simple idea that combines a bunch of things from prior work into one framework and yields strong results, outperforming most prior work on most tasks. 

I think this paper is impressive in how it scales up training to use so many tasks and such large training sets for each task.  That and its strong experimental results make it worthy of publication. It's not very surprising that adding more tasks and data improves performance on average across downstream tasks, but it is nice to see the experimental results in detail. While many people would think of this idea, few would have the resources and expertise necessary to do it justice.  I also like how the authors move beyond the standard sentence tasks to evaluate also on the Quora question duplicate task with different amounts of training data and also consider the sentence characteristic / syntactic property tasks.  It would be great if the authors could release their pretrained sentence representation model so that other researchers could use it. 

I do have some nitpicks here and there with the presentation and exposition, and I am concerned that at times the paper appears to be minimizing its weaknesses, but I think these are things that can be addressed in the next revision. I understand that sometimes it's tempting to minimize one's weaknesses in order to get a paper accepted because the reviewers may not understand the area very well and may get hung up on the wrong things. I understand the area well and so all the feedback I offer below comes from a place of desiring this paper's publication while also desiring it to be as accurate and helpful for the community as possible. 

Below I'll discuss my concerns with the experiments and description of the results.

Regarding the results in Table 2:

The results in Table 2 seem a little bit unstable, as it is unclear which setting to use for the classification tasks; maybe it depends on the kind of classification being performed. One model seems best for the sentiment tasks ("+2L +STP") while other models seem best for SUBJ and MPQA.  Adding parsing as a training task hurts performance on the sentence classification tasks while helping performance on the semantic tasks, as the authors note.  It is unclear which is the best general model.  In particular, when others write papers comparing to the results in this paper, which setting should they compare to?  It would be nice if the authors could discuss this. 

The results reported for the CNN-LSTM of Gan et al. do not exactly match those of any single row from Gan et al, either v1 or v2 on arxiv or the published EMNLP version. How were those specific numbers selected? 

The caption of Table 2 states "All results except ours are taken from Conneau et al. (2017)." However, Conneau et al (neither the latest arxiv version nor the published EMNLP version) does not include many of the results in the table, such as CNN-LSTM and DiscSent mentioned in the following sentence in the caption. Did the authors replicate the results of those methods themselves, or report them from other papers?

What does bold and underlining indicate in Table 2?  I couldn't find this explained anywhere. 

At the bottom of Table 2, in the section with approaches trained from scratch on these tasks, I'd suggest including the 89.7 SST result of Munkhdalai and Yu (2017) and the 96.1 TREC result of Zhou et al. (2016) (as well as potentially other results from Zhou et al, since they report results on others of these datasets). The reason this is important is because readers may observe that the paper's new method achieves higher accuracies on SST and TREC than all other reported results and mistakenly think that the new method is SOTA on those tasks.  I'd also suggest adding the results from Radford et al. (2017) who report 86.9 on MR and 91.4 on CR.  For other results on these datasets, including stronger results in non-fixed-dimensional-sentence-embedding transfer settings, see results and references in McCann et al. (2017).  While the methods presented in this paper are better than prior work in learning general purpose, fixed-dimensional sentence embeddings, they still do not produce state-of-the-art results on that many of these tasks, if any.  I think this is important to note. 

For all tasks for which there is additional training, there's a confound due to the dimensionality of the sentence embeddings across papers. Using higher-dimensional sentence embeddings leads to more parameters in the linear model being trained on the task data. So it is unclear if the increase in hidden units in rows with "+L" is improving the results because of providing more weights for the linear model or whether it is learning a better sentence representation. 

The main sentence embedding results are in Table 2, and use the SentEval framework. However, not all tasks are included. The STS Benchmark results are included, which use an additional layer trained on the STS Benchmark training data just like the SICK tasks. But the other STS results, which use cosine similarity on the embedding space directly without any retraining, are only included in the appendix (in Table 7). The new approach does not do very well on those unsupervised tasks. On two years of data it is better than InferSent and on two years it is worse. Both are always worse than the charagram-phrase results of Wieting et al (2016a), which has 66.1 on 2012, 57.2 on 2013, 74.7 on 2014, and 76.1 on 2015.  Charagram-phrase trains on automatically-generated paraphrase phrase pairs, but these are generated automatically from parallel text, the same type of resource used in the "+Fr" and "+De" models proposed in this submission, so I think it should be considered as a comparable model. 

The results in the bottom section of Table 7, reported from Arora et al (2016), were in turn copied from Wieting et al (2016b), so I think it would make sense to also cite Wieting et al (2016b) if those results are to be included. Also, it doesn't seem appropriate to designate those as "Supervised Approaches" as they only require parallel text, which is a subset of the resources required by the new model. 

There are some other details in the appendix that I find concerning:

Section 8 describes how there is some task-specific tuning of which function to compute on the encoder to produce the sentence representation for the task.  This means that part of the improvement over prior work (especially skip-thought and InferSent) is likely due to this additional tuning. So I suppose to use these sentence representations in other tasks, this same kind of tuning would have to be done on a validation set for each task?  Doesn't that slightly weaken the point about having "general purpose" sentence representations?

Section 9 provides details about how the representations are created for different training settings. I am confused by the language here. For example, the first setting ("+STN +Fr +De") is described as "A concatenation of the representations trained on these tasks with a unidirectional and bidirectional GRU with 1500 hidden units each." I'm not able to parse this. I think the authors mean "The sentence representation h_x is the concatenation of the final hidden vectors from a forward GRU (with 1500-dimensional hidden vectors) and a bidirectional GRU (also with 1500-dimensional hidden vectors)". Is this correct? 

Also in Sec 9: I found it surprising how each setting that adds a training task uses the concatenation of a representation with that task and one without that task. What is the motivation for doing this? This seems to me to be an important point that should be discussed in Section 3 or 4. And when doing this, are the concatenated representations always trained jointly from scratch with the special task only updating a subset of the parameters, or do you use the fixed pretrained sentence representation from the previous row and just concatenate it with the new one?  To be more concrete, if I want to get the encoder for the second setting ("+STN +Fr +De +NLI"), do I have to train two times or can I just train once?  That is, the train-once setting would correspond to only updating the NLI-specific representation parameters when training on NLI data; on other data, all parameters would be updated. The train-twice setting would first train a representation on "+STN +Fr +De", then set it aside, then train a separate representation on "+STN +Fr +De +NLI", then finally concatenate the two representations as my sentence representation.  Do you use train-once or train-twice? 

Regarding the results in Table 3:

What do bold and underline indicate?

What are the embeddings corresponding to the row labeled "Multilingual"?

In the caption, I can't find footnote 4. 

The caption includes the sentence "our embeddings have 1040 pairs out of 2034 for which atleast one of the words is OOV, so a comparison with other embeddings isn't fair on RW."  How were those pairs handled?  If they were excluded, then I think the authors should not report results on RW.  I suspect that most of the embeddings included in the table also have many OOVs in the RW dataset but still compute results on it using either an unknown word embedding or some baseline similarity of zero for pairs with an OOV. I think the authors should find some way (like one of those mentioned, or some other way) of computing similarity of those pairs with OOVs. It doesn't make much sense to me to omit pairs with OOVs. 

There are much better embeddings on SimLex than the embeddings whose results are reported in the table. Wieting et al. (2016a) report SimLex correlation of 0.706 and Mrkšić et al. (2017) report 0.751.  I'd suggest adding the results of some stronger embeddings to better contextualize the embeddings obtained by the new method.  Some readers may mistakenly think that the embeddings are SOTA on SimLex since no stronger results are provided in the table. 


The points below are more minor/specific:

Sec. 2:

In Sec. 2, the paper discusses its focus on fixed-length sentence representations to distinguish itself from other work that produces sentence representations that are not fixed-length. I feel the motivation for this is lacking. Why should we prefer a fixed-length representation of a sentence? For certain downstream applications, it might actually be easier for practitioners to use a representation that provides a representation for each position in a sentence (Melamud et al., 2016; Peters et al., 2017; McCann et al., 2017) rather than an opaque sentence representation. Some might argue that since sentences have different lengths, it would be appropriate for a sentence representation to have a length proportional to the length of the sentence.  I would suggest adding some motivation for the focus on fixed-length representations. 

Sec. 4.1:

"We take a simpler approach and pick a new task to train on after every parameter update sampled uniformly. An NLI minibatch is interspersed after every ten parameter updates on sequence-to-sequence tasks"
These two sentences seem contradictory. Maybe in the first sentence "pick a new task" should be changed to "pick a new sequence-to-sequence task"?

Sec. 5.1:

typo: "updating the parameters our sentence" --> "updating the parameters of our sentence"

Sec. 5.2:

typo in Table 4 caption: "and The" --> ". The"

typo: "parsing improvements performance" --> "parsing improves performance"


In general, there are many missing citations for the tasks, datasets, and prior work on them. I understand that the authors are pasting in numbers from many places and just providing pointers to papers that provide more citation info, but I think this can lead to mis-attribution of methods. I would suggest including citations for all datasets/tasks and methods whose results are being reported. 


References:

McCann, Bryan, James Bradbury, Caiming Xiong, and Richard Socher. "Learned in translation: Contextualized word vectors." CoRR 2017.

Melamud, Oren, Jacob Goldberger, and Ido Dagan. "context2vec: Learning Generic Context Embedding with Bidirectional LSTM." CoNLL 2016.

Mrkšić, Nikola, Ivan Vulić, Diarmuid Ó. Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. "Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints." TACL 2017.

Munkhdalai, Tsendsuren, and Hong Yu. "Neural semantic encoders." EACL 2017. 

Pagliardini, Matteo, Prakhar Gupta, and Martin Jaggi. "Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features." arXiv preprint arXiv:1703.02507 (2017).

Peters, Matthew E., Waleed Ammar, Chandra Bhagavatula, and Russell Power. "Semi-supervised sequence tagging with bidirectional language models." ACL 2017.

Radford, Alec, Rafal Jozefowicz, and Ilya Sutskever. "Learning to generate reviews and discovering sentiment." arXiv preprint arXiv:1704.01444 2017.

Wieting, John, Mohit Bansal, Kevin Gimpel, and Karen Livescu. "Charagram: Embedding words and sentences via character n-grams." EMNLP 2016a.

Wieting, John, Mohit Bansal, Kevin Gimpel, and Karen Livescu. "Towards universal paraphrastic sentence embeddings." ICLR 2016b.

Zhou, Peng, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. "Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling." COLING 2016.

review_title: solid empirical contribution to sentence embedding learning

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Follow-Up Comments
----

I continue to argue that this paper makes a contribution to a major open question, and clearly warrants acceptance. 

I agree with R1 that the results do not tell a completely clear story, and that the benefits of pretraining are occasionally minimal or absent. However, R1 uses this as the basis to argue for rejection, which does not seem reasonable to me at all. This limitation is an empirical fact that the paper has done a reasonable job of revealing, and it does not take away the paper's reason for existence, since many of the results are still quite strong, and the trends do support the merit of the proposed approach.

The authors mostly addressed my main concern, which was the relatively weak ablation. More combinations would be nice, but assuming reasonable resource constraints, I think the authors have done their due diligence, and the paper makes a clear contribution. I disagree with the response, though, that the authors can lean on other papers to help fill in the ablation—every paper in this area uses subtly different configurations.

I have one small lingering concern, which is not big enough to warrant acceptance: R2's point 10 is valid—the use of multiple RNNs trained on different objectives in the ablation experiments unexpected and unusual, and deserves mention in the body of the paper, rather than only in an appendix.

----
Original Review
---

This paper explores a variety of tasks for the pretraining of a bidirectional GRU sentence encoder for use in data-poor downstream tasks. The authors find that the combination of supervised training with NLI, MT, and parsing, plus unsupervised training on the SkipThought objective yields a model that robustly outperforms the best prior method on every task included in the standard SentEval suite, and several others.

This paper isn't especially novel. The main results of the paper stem from a combination of a few ideas that were ripe for combination (SkipThought from Kiros, BiLSTM-max and S/MNLI from Conneau, MT from McCann, parsing following Luong, etc.). However, the problem that the paper addresses is a major open issue within NLP, and the paper is very well done, so it would be in the best interest of all involved to make sure that the results are published promptly. I strongly support acceptance.

My one major request would be a more complete ablation analysis. It would be valuable for researchers working on other languages (among others) to know which labeled or unlabeled datasets contributed the most. Your ablation does not offer enough evidence to one to infer this---among other things, NLI and MT are never presented in isolation, and parsing is never presented without those two. Minimally, this should involve presenting results for models trained separately on each of the pretraining tasks.

I'll also echo another question from Samuel's comment: Could you say more about how you conducted the evaluation on the SentEval tasks? Did your task-specific model (or the training/tuning procedure for that model) differ much from prior work?

Details:

The paragraph starting "we take a simpler approach" is a bit confusing. If task batches are sampled *uniformly*, how is NLI be sampled less often than the other tasks?

Given how many model runs are presented, and that the results don't uniformly favor your largest/last model, it'd be helpful to include some kind of average of performance across tasks that can be used as a single-number metric for comparison. This also applies to the word representation evaluation table.

When comparing word embeddings, it would be helpful to include the 840B-word release of GloVe embeddings. Impressionistically, that is much more widely used than the older 6B-word release for which Faruqui reports numbers. This isn't essential to the paper, but it would make your argument in that section more compelling.

"glove" => GloVe; "fasttext" => fastText
review_title: Generally positive review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper shows that learning sentence representations from a diverse set of tasks (skip-thought objective, MT, constituency parsing, and natural language inference) produces .
The main contribution of the paper is to show learning from multiple tasks improves the quality of the learned representations.
Experiments on various text classification and sentiment analysis datasets show that the proposed method is competitive with existing approaches.
There is an impressive number of experiments presented in the paper, but the results are a bit mixed, and it is not always clear that adding more tasks help.

I think this paper addresses an important problem of learning general purpose sentence representations. 
However, I am unable to draw a definitive conclusion from the paper. 
From Table 2, the best performing model is not always the one with more tasks. 
For example, adding a parsing objective can either improve or lower the performance quite significantly.
Could it be that datasets such as MRPC, SICK, and STSB require more understanding of syntax?
Even if this is the case, why adding this objective hurt performance for other datasets?
Importantly, it is also not clear whether the performance improvement comes from having more unlabeled data (even if it is trained with the same training objective) or having multiple training objectives.
Another question I have is that if there is any specific reason that language modeling is not included as one of the training objectives to learn sentence representations, given that it seems to be the easiest one to collect training data for.

The results for transfer learning and low resource settings are more positive.
However, it is not surprising that pretraining parts of the model on a large amount of unlabeled data helps when there is not a lot of labeled examples.

Overall, while the main contribution of the paper is that having multiple training objectives help learning better sentence, I am not yet convinced by the experiments that this is indeed the case.
review_title: review

Meta-review: This paper presents a very cool setup for multi task learning for learning fixed length representations for sentences.  Although the authors accept the fact that fixed length representations may not be suitable for complex, long pieces of text (often, sentences), such representations may be useful for several tasks.  They use a significantly large scale setup with six interesting tasks and show that learning generic representations for sentences across tasks is useful than learning in isolation.  Two out of three reviewers presented extensive critique of the paper and there's thorough back and forth between the reviewers and the authors.  The committee believes that this paper will add positive value to the conference.
Decision: Accept (Poster)