Title: A Simple but Tough-to-Beat Baseline for Sentence Embeddings
Authors: Sanjeev Arora
Yingyu Liang
Tengyu Ma
Venue: ICLR 2017
Keywords: Natural language processing
Unsupervised Learning

Abstract: 
The success of neural network methods for computing word embeddings has motivated methods for generating semantic embeddings of longer pieces of text, such as sentences and paragraphs. Surprisingly, Wieting et al (ICLR'16) showed that such complicated methods are outperformed, especially in out-of-domain (transfer learning) settings, by simpler methods involving mild retraining of word embeddings and basic linear regression. The  method of Wieting et al. requires retraining with a substantial labeled dataset such as Paraphrase Database (Ganitkevitch et al., 2013). 

The current paper goes further, showing that the following completely unsupervised sentence embedding is a formidable baseline: Use word embeddings computed using one of the popular methods on unlabeled corpus like Wikipedia, represent the sentence by a weighted average of the word vectors, and then modify them a bit using PCA/SVD. This weighting improves performance by about 10% to 30% in textual similarity tasks, and beats sophisticated supervised methods including RNN's and LSTM's. It even improves Wieting et al.'s embeddings. 
 This simple method should be used as the baseline to beat in future, especially when labeled training data is scarce or nonexistent. 

The paper also gives a theoretical explanation of the success of the above unsupervised method using a latent variable generative model for sentences, which is a simple extension of the model in Arora et al. (TACL'16) with new "smoothing" terms that allow for 
words occurring out of context, as well as high probabilities for words like and, not in all contexts. 

TL;DR: A simple unsupervised method for sentence embedding that can get results comparable to sophisticated models like RNN's and LSTM's

Review discussion:
It'd be great to compare on some of the dataset used in Wang and Manning who use simple Naive Bayes SVM hybrids.
For example: IMDB or Stanford Sentiment Treebank.
How does your model do on those?
Thanks for the suggestion! We have tested on the Stanford Sentiment Treebank (Table 2), though we didn't find Wang and Manning'12 tested on it. We will try IMDB and potentially other datasets reported in Wang and Manning'12.
Thanks for the question! We checked on IMDB dataset, studied by Wang and Manning. Since the intended application is semisupervised or transfer learning, we also compared performance with fewer labeled examples.
# labeled examples    NBSVM        Our method (PP-WR)
50k                                 0.91         0.85
1K                                  0.84         0.82
Another comment is that sentiment analysis appears to be the best case for BOW methods, whereas it may be the worst case for word embedding methods (See Table 2) due to the well-known antonymy problem ---distributional hypothesis fails for distinguishing "good" from "bad."
In the experiments, did the sentence set S mentioned in algorithm 1 include both training and test examples? If the test examples were not included, then how to do the projection step in line 6 on the test examples?
For the textual similarity tasks, since our method is unsupervised, we do not use the training dataset and we compute the principle component on all the test examples.
For the supervised task, we do not remove the common component, since it can be absorbed into the classifier. Please see Section A.2 in the appendix for the details.
Have you considered evaluating on the SNLI dataset as well? This dataset is broader than SICK in certain ways, and might be more challenging. I am curious to see if the summation-based method presented here can do as well as the attention-based systems.
Having said that, the evaluation is quite comprehensive as is!
Thanks! We will test on it soon.
Thanks for suggesting the SNLI dataset! We compared to the standard RNN/LSTM-RNN approaches used in the paper publishing the dataset: "A large annotated corpus for learning natural language inference" by Bowman, Angeli, Potts, and Manning. Our method indeed shows slightly better performance.
Sentence model              Train    Test
100d Sum of words           79.3     75.3
100d RNN                    73.1     72.2
100d LSTM RNN               84.8     77.6
Our method                  83.9     78.2
Our test accuracy is worse than those using more sophisticated models (e.g., using attention mechanism), which are typically 83% - 88%; see the website of the SNLI project for a summary. An interesting direction is to study whether our idea can be combined with these sophisticated models to get improved performance.
Technical details: We used the code provided by the authors of the SNLI paper (we thank the authors!). We trained their classifier on our sentence embedding for 120 passes over the data, using their default hyperparameters.
This paper presents a new theoretically-principled method of representing sentences as vectors. The experiments show that vectors produced by this method perform well on similarity and entailment benchmarks, surpassing some RNN-based methods too.
Overall, this is an interesting empirical result, especially since the model is not order-sensitive (as far as I can tell). I would like to see some more discussion on why such a simple model does better than LSTMs at capturing similarity and entailment. Could this be an artifact of these benchmarks?
Thanks for the positive review!
We agree that our results suggest the benchmarks need to be rethought.
At the same time, we did a simple experiment suggesting that word order does play some role in the benchmarks. We trained and tested RNN/LSTM on the supervised tasks where the words in each sentence are randomly shuffled. Performance drops noticeably.
Thus our method ---which does ignore word order---must be much better at exploiting the semantics than RNN/LSTM. We will further explore if some ensemble idea can combine the advantages of both approaches.
1) Pearson’s r x 100 on similarity task:
rnn    lstm(no)   lstm(o.g.)
random:    54.50   77.24     79.39
normal:    73.13   85.45     83.41
2) accuracy on entailment task:
rnn    lstm(no)   lstm(o.g.)
random:    61.7   78.2       81.0
normal:    76.4   83.2       82.0
3) accuracy on sentiment task:
rnn    lstm(no)   lstm(o.g.)
random:    84.2   82.9       84.1
normal:    86.5   86.6       89.2
Technical detail: on the random order datasets, the hyperparameters are enumerated as in the paper, and the best results are reported.
This paper proposes a simple way to reweight the word embedding in the simple composition function for sentence representation. This paper also shows the connection between this new weighting scheme and some previous work.
Here are some comments on technical details:
- The word "discourse" is confusing. I am not sure whether the words "discourse" in "discourse vector c_s" and the one in "most frequent discourse" have the same meaning.
- Is there any justification about $c_0$ related to syntac?
- Not sure what thie line means: "In fact the new model was discovered by our detecting the common component c0 in existing embeddings." in section "Computing the sentence embedding"
- Is there any explanation about the results on sentiment in Table 2?
Thanks for the positive review!
- The "discourse" in "discourse vector c_s" and the one in "most frequent discourse" have the same meaning.
- The top singular vector $c_0$ of the datasets seems to roughly correspond to the syntactic information or common words.
Closest words (by cosine similarity) to $c_0$ in the SICK dataset are: "just" "when" "even" "one" "up" "little" "way" "there" "while" "but".
- Sorry for the clumsy phrasing. We meant to say that we empirically discovered the significant common component $c_0$ in word vectors built by existing methods, which inspired us to propose our theoretical model of this paper.
- We speculate that our method doesn't outperform RNNs and LSTMs for sentiment tasks because (a) the word vectors ---and more generally the distributional hypothesis of meaning ---has known limitations for capturing sentiment due to the "antonym problem", (b) also in our weighted average scheme, words like "not" that may be important for sentiment analysis are downweighted a lot.
This is a good paper with an interesting probabilistic motivation for weighted bag of words models.
The (hopefully soon) added comparison to Wang and Manning will make it stronger.
Though it is sad that for sufficiently large datasets, NB-SVM still works better.
In the second to last paragraph of the introduction you describe a problem of large cooccurrence counts which was already fixed by the Glove embeddings with their weighting function f.
Minor comments:
"The capturing the similarities" -- typo in line 2 of intro.
"Recently, (Wieting et al.,2016) learned" -- use citet instead of parenthesized citation
Thanks for the positive review!
We will fix the typos pointed out and add the comparisons with NB-SVM.

Meta-review:
role: PC
data: comment: A new method for sentence embedding that is simple and performs well. Important contribution that will attract attention and help move the field forward.

Decision: Accept (Poster)