Title: A SIMPLE BUT TOUGH-TO-BEAT BASELINE FOR SEN-TENCE EMBEDDINGS

Abstract: The success of neural network methods for computing word embeddings has motivated methods for generating semantic embeddings of longer pieces of text, such as sentences and paragraphs. Surprisingly, Wieting et al (ICLR'16) showed that such complicated methods are outperformed, especially in out-of-domain (transfer learning) settings, by simpler methods involving mild retraining of word embeddings and basic linear regression. The method of Wieting et al. requires retraining with a substantial labeled dataset such as Paraphrase Database (Ganitkevitch et al., 2013). The current paper goes further, showing that the following completely unsupervised sentence embedding is a formidable baseline: Use word embeddings computed using one of the popular methods on unlabeled corpus like Wikipedia, represent the sentence by a weighted average of the word vectors, and then modify them a bit using PCA/SVD. This weighting improves performance by about 10% to 30% in textual similarity tasks, and beats sophisticated supervised methods including RNN's and LSTM's. It even improves Wieting et al.'s embeddings. This simple method should be used as the baseline to beat in future, especially when labeled training data is scarce or nonexistent. The paper also gives a theoretical explanation of the success of the above unsupervised method using a latent variable generative model for sentences, which is a simple extension of the model in Arora et al. (TACL'16) with new "smoothing" terms that allow for words occurring out of context, as well as high probabilities for words like and, not in all contexts.

1. INTRODUCTION:

Word embeddings computed using diverse methods are basic building blocks for Natural Language Processing (NLP) and Information Retrieval (IR). They capture the similarities between words (e.g., (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a; Pennington et al., 2014) ). Recent work has tried to compute embeddings that capture the semantics of word sequences (phrases, sentences, and paragraphs), with methods ranging from simple additional composition of the word vectors to sophisticated architectures such as convolutional neural networks and recurrent neural networks (e.g., (Iyyer et al., 2015; Le & Mikolov, 2014; Kiros et al., 2015; Socher et al., 2011; Blunsom et al., 2014; Tai et al., 2015; Wang et al., 2016) ). Recently, (Wieting et al., 2016) learned general-purpose, paraphrastic sentence embeddings by starting with standard word embeddings and modifying them based on supervision from the Paraphrase pairs dataset (PPDB), and constructing sentence embeddings by training a simple word averaging model. This simple method leads to better performance on textual similarity tasks than a wide variety of methods and serves as a good initialization for textual classification tasks. However, supervision from the paraphrase dataset seems crucial, since they report that simple average of the initial word embeddings does not work very well.

Here we give a new sentence embedding method that is embarrassingly simple: just compute the weighted average of the word vectors in the sentence and then remove the projections of the average vectors on their first singular vector ("common component removal") . Here the weight of a word w is a/(a + p(w)) with a being a parameter and p(w) the (estimated) word frequency; we call this smooth inverse frequency (SIF).1 This method achieves significantly better performance than the unweighted average on a variety of textual similarity tasks, and on most of these tasks even beats some sophisticated supervised methods tested in (Wieting et al., 2016) , including some RNN and LSTM models. The method is well-suited for domain adaptation settings, i.e., word vectors trained on various kinds of corpora are used for computing the sentence embeddings in different testbeds. It is also fairly robust to the weighting scheme: using the word frequencies estimated from different corpora does not harm the performance; a wide range of the parameters a can achieve close-to-best results, and an even wider range can achieve significant improvement over unweighted average. Of course, this SIF reweighting is highly reminiscent of TF-IDF reweighting from information retrieval (Sparck Jones, 1972; Robertson, 2004) if one treats a "sentence" as a "document" and make the reasonable assumption that the sentence doesn't typically contain repeated words. Such reweightings (or related ideas like removing frequent words from the vocabulary) are a good rule of thumb but has not had theoretical justification in a word embedding setting.

The current paper provides a theoretical justification for the reweighting using a generative model for sentences, which is a simple modification for the Random Walk on Discourses model for generating text in (Arora et al., 2016) . In that paper, it was noted that the model theoretically implies a sentence embedding, namely, simple average of embeddings of all the words in it.

We modify this theoretical model, motivated by the empirical observation that most word embedding methods, since they seek to capture word cooccurence probabilities using vector inner product, end up giving large vectors to frequent words, as well as giving unnecessarily large inner products to word pairs, simply to fit the empirical observation that words sometimes occur out of context in documents. These anomalies cause the average of word vectors to have huge components along semantically meaningless directions. Our modification to the generative model of (Arora et al., 2016) allows "smoothing" terms, and then a max likelihood calculation leads to our SIF reweighting.

Interestingly, this theoretically derived SIF does better (by a few percent points) than traditional TF-IDF in our setting. The method also improves the sentence embeddings of Wieting et al., as seen in Table 1 . Finally, we discovered that -contrary to widespread belief-Word2Vec(CBOW) also does not use simple average of word vectors in the model, as misleadingly suggested by the usual expression Pr[w|w 1 , w 2 , . . . , w 5 ] ∝ exp(v w • ( 1 5 i v wi )). A dig into the implementation shows it implicitly uses a weighted average of word vectors -again, different from TF-IDF-and this weighting turns out to be quite similar in effect to ours. (See Section 3.1.)

2. RELATED WORK:

Word embeddings. Word embedding methods represent words as continuous vectors in a low dimensional space which capture lexical and semantic properties of words. They can be obtained from the internal representations from neural network models of text (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a) or by low rank approximation of co-occurrence statistics (Deerwester et al., 1990; Pennington et al., 2014) . The two approaches are known to be closely related (Levy & Goldberg, 2014; Hashimoto et al., 2016; Arora et al., 2016) .

Our work is most directly related work to (Arora et al., 2016) , which proposed a random walk model for generating words in the documents. Our sentence vector can be seen as approximate inference of the latent variables in their generative model.

Phrase/Sentence/Paragraph embeddings. Previous works have computed phrase or sentence embeddings by composing word embeddings using operations on vectors and matrices e.g., (Mitchell & Lapata, 2008; 2010; Blacoe & Lapata, 2012) . They found that coordinate-wise multiplication of the vectors performed very well among the binary operations studied. Unweighted averaging is also found to do well in representing short phrases (Mikolov et al., 2013a) . Another approach is recursive neural networks (RNNs) defined on the parse tree, trained with supervision (Socher et al., 2011) or without (Socher et al., 2014) . Simple RNNs can be viewed as a special case where the parse tree is replaced by a simple linear chain. For example, the skip-gram model (Mikolov et al., 2013b) is extended to incorporate a latent vector for the sequence, or to treat the sequences rather than the word as basic units. In (Le & Mikolov, 2014) each paragraph was assumed to have a latent paragraph vector, which influences the distribution of the words in the paragraph. Skip-thought of (Kiros et al., 2015) tries to reconstruct the surrounding sentences from surrounded one and treats the hidden parameters as their vector representations. RNNs using long short-term memory (LSTM) capture long-distance dependency and have also been used for modeling sentences (Tai et al., 2015) . Other neural network structures include convolution neural networks, such as (Blunsom et al., 2014) that uses a dynamic pooling to handle input sentences of varying length and do well in sentiment prediction and classification tasks.

The directed inspiration for our work is (Wieting et al., 2016) which learned paraphrastic sentence embeddings by using simple word averaging and also updating standard word embeddings based on supervision from paraphrase pairs; the supervision being used for both initialization and training.

3. A SIMPLE METHOD FOR SENTENCE EMBEDDING:

We briefly recall the latent variable generative model for text in (Arora et al., 2016) . The model treats corpus generation as a dynamic process, where the t-th word is produced at step t. The process is driven by the random walk of a discourse vector c t ∈ d . Each word w in the vocabulary has a vector in d as well; these are latent variables of the model. The discourse vector represents "what is being talked about." The inner product between the discourse vector c t and the (time-invariant) word vector v w for word w captures the correlations between the discourse and the word. The probability of observing a word w at time t is given by a log-linear word production model from Mnih and Hinton:

Pr[w emitted at time t | c t ] ∝ exp ( c t , v w ) .

(1)

The discourse vector c t does a slow random walk (meaning that c t+1 is obtained from c t by adding a small random displacement vector), so that nearby words are generated under similar discourses. It was shown in (Arora et al., 2016) that under some reasonable assumptions this model generates behavior -in terms of word-word cooccurrence probabilities-that fits empirical works like word2vec and Glove. The random walk model can be relaxed to allow occasional big jumps in c t , since a simple calculation shows that they have negligible effect on cooccurrence probabilities of words.

The word vectors computed using this model are reported to be similar to those from Glove and word2vec(CBOW).

Our improved Random Walk model. Clearly, it is tempting to define the sentence embedding as follows: given a sentence s, do a MAP estimate of the discourse vectors that govern this sentence.

We note that we assume the discourse vector c t doesn't change much while the words in the sentence were emitted, and thus we can replace for simplicity all the c t 's in the sentence s by a single discourse vector c s . In the paper (Arora et al., 2016) , it was shown that the MAP estimate of c s is -up to multiplication by scalar-the average of the embeddings of the words in the sentence.

In this paper, towards more realistic modeling, we change the model (1) as follows. This model has two types of "smoothing term", which are meant to account for the fact that some words occur out of context, and that some frequent words (presumably "the", "and " etc.) appear often regardless of the discourse. We first introduce an additive term αp(w) in the log-linear model, where p(w) is the unigram probability (in the entire corpus) of word and α is a scalar. This allows words to occur even if their vectors have very low inner products with c s . Secondly, we introduce a common discourse vector c 0 ∈ d which serves as a correction term for the most frequent discourse that is often related to syntax. (Other possible correction is left to future work.) It boosts the co-occurrence probability of words that have a high component along c 0 .

Concretely, given the discourse vector c s , the probability of a word w is emitted in the sentence s is modeled by,

EQUATION

where

cs = βc 0 + (1 -β)c s , c 0 ⊥ c s

where α and β are scalar hyperparameters, and Z cs = w∈V exp ( cs , v w ) is the normalizing constant (the partition function). We see that the model allows a word w unrelated to the discourse Algorithm 1 Sentence Embedding Input: Word embeddings {v w : w ∈ V}, a set of sentences S, parameter a and estimated probabilities {p(w) : w ∈ V} of the words. Output: Sentence embeddings {v s : s ∈ S} 1: for all sentence s in S do 2:

v s ← 1

|s| w∈s a a+p(w) v w 3: end for 4: Form a matrix X whose columns are {v s : s ∈ S}, and let u be its first singular vector 5: for all sentence s in S do 6:

v s ← v s -uu v s 7: end for c s to be emitted for two reasons: a) by chance from the term αp(w); b) if w is correlated with the common discourse vector c 0 .

Computing the sentence embedding. The word embeddings yielded by our model are actually the same. 2 The sentence embedding will be defined as the max likelihood estimate for the vector c s that generated it. ( In this case MLE is the same as MAP since the prior is uniform.) We borrow the key modeling assumption of (Arora et al., 2016) , namely that the word v w 's are roughly uniformly dispersed, which implies that the partition function Z c is roughly the same in all directions. So assume that Z cs is roughly the same, say Z for all cs . By the model (2) the likelihood for the sentence is

p[s | c s ] = w∈s p(w | c s ) = w∈s αp(w) + (1 -α) exp ( v w , cs ) Z . Let f w (c s ) = log αp(w) + (1 -α) exp ( v w , cs )

Z denote the log likelihood of sentence s. Then, by simple calculus we have,

∇f w (c s ) = 1 αp(w) + (1 -α) exp ( v w , cs ) /Z 1 -α Z exp ( v w , cs ) v w .

Then by Taylor expansion, we have,

f w (c s ) ≈ f w (0) + ∇f w (0) cs = constant + (1 -α)/(αZ) p(w) + (1 -α)/(αZ) v w , cs .

Therefore, the maximum likelihood estimator for cs on the unit sphere (ignoring normalization) is approximately,3 arg max

EQUATION

That is, the MLE is approximately a weighted average of the vectors of the words in the sentence. Note that for more frequent words w, the weight a/(p(w) + a) is smaller, so this naturally leads to a down weighting of the frequent words.

To estimate c s , we estimate the direction c 0 by computing the first principal component of cs 's for a set of sentences. 4 In other words, the final sentence embedding is obtained by subtracting the projection of cs 's to their first principal component. This is summarized in Algorithm 1. Figure 1 : The subsampling probabilities in word2vec are similar to our weighting scheme.

3.1. CONNECTION TO SUBSAMPLING PROBABILITIES IN WORD2VEC:

Word2vec (Mikolov et al., 2013b ) uses a sub-sampling technique which downsamples word w with probability proportional to 1/ p(w) where p(w) is the marginal probability of the word w. This heuristic not only speeds up the training but also learns more regular word representations. Here we explain that this corresponds to an implicit reweighting of the word vectors in the model and therefore the statistical benefit should be of no surprise.

Recall the vanilla CBOW model of word2vec:

EQUATION

It can be shown that the loss (MLE) for the single word vector v w (from this occurrence) can be abstractly written in the form,

g(v w ) = γ( vt , v w ) + negative sampling terms , where γ(x) = log(1/(1 + e -x

)) is the logistic function. Therefore, the gradient of g(v w ) is

EQUATION

where α is a scalar. That is, without the sub-sampling trick, the update direction is the average of the word vectors in the window.

The sub-sampling trick in (Mikolov et al., 2013b) randomly selects the summands in equation ( 5) to "estimate" the gradient. Specifically, the sampled update direction is

EQUATION

where J k 's are Bernoulli random variables with Pr

[J k = 1] = q(w t-k ) min 1, 10 -5

p(w t-k ) . However, we note that ∇g(v w ) is (very) biased estimator! We have that the expectation of ∇g(v w ) is a weighted sum of the word vectors, E ∇g(v w ) = α(q(w t-5 )v wt-5 + q(w t-4 )v wt-4 + q(w t-3 )v wt-3 + q(w t-2 )v wt-2 + q(w t-1 )v wt-1 ) .

In fact, the expectation E[ ∇g(v w )] corresponds to the gradient of a modified word2vec model with the average vt (in equation ( 4)) being replaced by the weighted average 5 k=1 q(w t-k )v w t-k . Such a weighted model can also share the same form of what we derive from our random walk model as in equation (3). Moreover, the weighting q(w i ) closely tracks our weighting scheme a/(a + p(w)) when using parameter a = 10 -4 ; see Figure 1 for an illustration. Therefore, the expected gradient here is approximately the estimated discourse vector in our model! Thus, word2vec with sub-sampling gradient heuristic corresponds to a stochastic gradient update method for using our weighting scheme.

Results collected from (Wieting et al., 2016) 

4. EXPERIMENTS:



4.1. TEXTUAL SIMILARITY TASKS:

Datasets. We test our methods on the 22 textual similarity datasets including all the datasets from SemEval semantic textual similarity (STS) tasks (2012-2015) (Agirre et al., 2012; 2013; 2014; Agirrea et al., 2015) , and the SemEval 2015 Twitter task (Xu et al., 2015) and the SemEval 2014 Semantic Relatedness task (Marelli et al., 2014) . The objective of these tasks is to predict the similarity between two given sentences. The evaluation criterion is the Pearson's coefficient between the predicted scores and the ground-truth scores.

Experimental settings. We will compare our method with the following:

1. Unsupervised: ST, avg-GloVe, tfidf-GloVe. ST denotes the skip-thought vectors (Kiros et al., 2015) , avg-GloVe denotes the unweighted average of the GloVe vectors (Pennington et al., 2014) ,5 and tfidf-GloVe denotes the weighted average of GloVe vectors using TF-IDF weights.

2. Semi-supervised: avg-PSL. This method uses the unweighted average of the PARAGRAM-SL999 (PSL) word vectors from (Wieting et al., 2015) . The word vectors are trained using labeled data, but the sentence embedding are computed by unweighted average without training.

3. Supervised: PP, PP-proj., DAN, RNN, iRNN, LSTM (o.g.), LSTM (no). All these methods are initialized with PSL word vectors and then trained on the PPDB dataset. PP and PPproj. are proposed in (Wieting et al., 2016) . The first is an average of the word vectors, and the second additionally adds a linear projection. The word vectors are updated during the training. DAN denotes the deep averaging network of (Iyyer et al., 2015) . RNN denotes the classical recurrent neural network, and iRNN denotes a variant with the activation being the identity, and the weight matrices initialized to identity. The LSTM is the version from (Gers et al., 2002) , either with output gates (denoted as LSTM(o.g.)) or without (denoted as LSTM (no)).

Our method can be applied to any types of word embeddings. So we denote the sentence embeddings obtained by applying our method to word embeddings method "XXX" as "XXX+WR". 6 To get a completely unsupervised method, we apply it to the GloVe vectors, denoted as GloVe+WR. The weighting parameter a is fixed to 10 -3 , and the word frequencies p(w) are estimated from the Figure 2 : Effect of weighting scheme in our method on the average performance on STS 2012 tasks.

Best viewed in color. (a) Performance v.s. weighting parameter a. Three types of word vectors (PSL, GloVe, SN) are tested using p(w) estimated on the enwiki dataset. The best performance is usually achieved at a = 10 -3 to a = 10 -4 . (b) Performance v.s. datasets used for estimating p(w). Four datasets (enwiki, poliblogs, commoncrawl, text8) are used to estimate p(w) which is then used in our method. The parameter a is fixed to be 10 -3 . The performance is almost the same for different settings.

commoncrawl dataset.7 This is denoted by GloVe+WR in Table 1 . We also apply our method on the PSL vectors, denoted as PSL+WR, which is a semi-supervised method.

Results. The results are reported in Table 1 . Each year there are 4 to 6 STS tasks. For clarity, we only report the average result for the STS tasks each year; the detailed results are in the appendix.

The unsupervised method GloVe+WR improves upon avg-GloVe significantly by 10% to 30%, and beats the baselines by large margins. It achieves better performance than LSTM and RNN and is comparable to DAN, even though the later three use supervision. This demonstrates the power of this simple method: it can be even stronger than highly-tuned supervisedly trained sophisticated models. Using TF-IDF weighting scheme also improves over the unweighted average, but not as much as our method.

The semi-supervised method PSL+WR achieves the best results for four out of the six tasks and is comparable to the best in the rest of two tasks. Overall, it outperforms the avg-PSL baseline and all the supervised models initialized with the same PSL vectors. This demonstrates the advantage of our method over the training for those models.

We also note that the top singular vectors c 0 of the datasets seem to roughly correspond to the syntactic information or common words. For example, closest words (by cosine similarity) to c 0 in the SICK dataset are "just", "when", "even", "one", "up", "little", "way", "there", "while", and "but."

Finally, in the appendix, we showed that our two ideas all contribute to the improvement: for GloVe vectors, using smooth inverse frequency weighting alone improves over unweighted average by about 5%, using common component removal alone improves by 10%, and using both improves by 13%.

4.1.1. EFFECT OF WEIGHTING PARAMETER ON PERFORMANCE:

We study the sensitivity of our method to the weighting parameter a, the method for computing word vectors, and the estimated word probabilities p(w). First, we test the performance of three (Wieting et al., 2016) , and those in Column 7 for skip-thought are from (Lei Ba et al., 2016) .

types of word vectors (PSL, GloVe, and SN) on the STS 2012 tasks. SN vectors are trained on the enwiki dataset (Wikimedia, 2012) using the method in (Arora et al., 2016) , while PSL and GloVe vectors are those used in Table 1 . We enumerate a ∈ {10 -i , 3 × 10 -i : 1 ≤ i ≤ 5} and use the p(w) estimated on the enwiki dataset. Figure 2a shows that for all three kinds of word vectors, a wide range of a leads to significantly improved performance over the unweighted average. Best performance occurs from a = 10 -3 to a = 10 -4 .

Next, we fix a = 10 -3 and use four very different datasets to estimate p(w): enwiki (wikipedia, 3 billion tokens), poliblogs (Yano et al., 2009) (political blogs, 5 million), commoncrawl (Buck et al., 2014) (Internet crawl, 800 billion), text8 (Mahoney, 2008) (wiki subset, 1 million). Figure 2b shows performance is almost the same for all four settings.

The fact that our method can be applied on different types of word vectors trained on different corpora also suggests it should be useful across different domains. This is especially important for unsupervised methods, since the unlabeled data available may be collected in a different domain from the target application.

4.2. SUPERVISED TASKS:

The sentence embeddings obtained by our method can be used as features for downstream supervised tasks. We consider three tasks: the SICK similarity task, the SICK entailment task, and the Stanford Sentiment Treebank (SST) binary classification task (Socher et al., 2013) . To highlight the representation power of the sentence embeddings learned unsupervisedly, we fix the embeddings and only learn the classifier. Setup of supervised tasks mostly follow (Wieting et al., 2016) to allow fair comparison, i.e., the classifier a linear projection followed by the classifier in (Kiros et al., 2015) . The linear projection maps the sentence embeddings into 2400 dimension (the same as the skip-thought vectors), and is learned during the training. We compare our method to PP, DAN, RNN, and LSTM, which are the methods used in Section 4.1. We also compare to the skip-thought vectors (with improved training in (Lei Ba et al., 2016) ).

Results. Our method gets better or comparable performance compared to the competitors. It gets the best results for two of the tasks. This demonstrates the power of our simple method. We emphasize that our embeddings are unsupervisedly learned, while DAN, RNN, LSTM are trained with supervision. Furthermore, skip-thought vectors are much higher dimensional than ours (though projected into higher dimension, the original 300 dimensional embeddings contain all the information).

The advantage is not as significant as in the textual similarity tasks. This is possibly because similarity tasks rely directly upon cosine similarity, which favors our method's approach of removing the common components (which can be viewed as a form of denoising), while in supervised tasks, with the cost of some label information, the classifier can pick out the useful components and ignore the common ones.

Finally, we speculate that our method doesn't outperform RNN's and LSTM's for sentiment tasks because (a) the word vectors -and more generally the distributional hypothesis of meaning -has known limitations for capturing sentiment due to the "antonym problem", (b) also in our weighted average scheme, words like "not" that may be important for sentiment analysis are downweighted a lot. To address (a), there is existing work on learning better word embeddings for sentiment analysis (e.g., (Maas et al., 2011) ). To address (b), it is possible to design weighting scheme (or learn weights) for this specific task. Table 3 : Comparison of results on the original datasets and the ones with words randomly shuffled in sentences. The rows with label "original" are the results on the original datasets, and those with label "random" are the results on the randomly shuffled datasets. The row for similarity (SICK) shows Pearson's r × 100 and the other rows show accuracy.

4.3. THE EFFECT OF THE ORDER OF WORDS IN SENTENCES:

A interesting feature of our method is that it ignores the word order. This is in contrast to that RNN's and LSTM's can potentially take advantage of the word order. The fact that our method achieves better or comparable performance on these benchmarks raise the following question: is word order not important in these benchmarks? We conducted an experiment suggesting that word order does play some role.

We trained and tested RNN/LSTM on the supervised tasks where the words in each sentence are randomly shuffled, and the results are reported in Table 3 .8 It can be observed that the performance drops noticeably. Thus our method -which ignores word order-must be much better at exploiting the semantics than RNN's and LSTM's. An interesting future direction is to explore if some ensemble idea can combine the advantages of both approaches.

5. CONCLUSIONS:

This work provided a simple approach to sentence embedding, based on the discourse vectors in the random walk model for generating text (Arora et al., 2016) . It is simple and unsupervised, but achieves significantly better performance than baselines on various textual similarity tasks, and can even beat sophisticated supervised methods such as some RNN and LSTM models. The sentence embeddings obtained can be used as features in downstream supervised tasks, which also leads to better or comparable results compared to the sophisticated methods.

A DETAILS OF EXPERIMENTAL SETTING:

A.1 UNSUPERVISED TASK: TEXTUAL SIMILARITY

The competitors. We give a brief overview of the competitors. RNN is the classical recurrent neural network:

h t = f (W x W xt w + W h h t-1 + b)

where f is the activation, W x , W h and b are parameters, and x t is the t-th token in the sentence. The sentence embedding of RNN is just the hidden vector of the last token. iRNN is a special RNN with the activation being the identity, the weight matrices initialized to identity, and b initialized to zero. LSTM (Hochreiter & Schmidhuber, 1997 ) is a recurrent neural network architecture designed to capture long-distance dependencies. Here, the version from (Gers et al., 2002) is used.

The supervised methods are initialized with PARAGRAM-SL999 (PSL) vectors, and trained using the approach of (Wieting et al., 2016) on the XL section of the PPDB data (Pavlick et al., 2015) which contains about 3 million unique phrase pairs. After training, the final models can be used to generate sentence embeddings on the test data. For hyperparameter tuning they used 100k examples sampled from PPDB XXL and trained for 5 epochs. Then after finding the hyperparameters that maximize Spearman's coefficients on the Pavlick et al. PPDB task, they are trained on the entire XL section of PPDB for 10 epochs. See (Wieting et al., 2016) and related papers for more details about these methods.

The tfidf-GloVe method is a weighted average of the GloVe vectors, where the weights are defined by the TF-IDF scheme. More precisely, the embedding of a sentence s is

EQUATION

where IDF w is the inverse document frequency of w, and |s| denotes the number of words in the sentence. Here, the TF part of the TF-IDF scheme is taken into account by the sum over w ∈ s. Furthermore, when computing IDF w , each sentence is viewed as a "document":

IDF w := log 1 + N 1 + N w

where N is the total number of sentences and N w is the number of sentences containing w, and 1 is added to avoid division by 0. In the experiments, we use all the textual similarity datasets to compute IDF w .

Detailed experimental results. In the main body we present the average results for STS tasks by year. Each year there are actually 4 to 6 STS tasks, as shown in Table 4 . Note that tasks with the same name in different years are actually different tasks. Here we provide the results for each tasks in Table 5 . PSL+WR achieves the best results on 12 out of 22 tasks, PP and PP-proj. achieves on 3, tfidf-GloVe achieves on 2, and DAN, iRNN, and GloVe+WR achieves on 1. In general, our method improves the performance significantly compared to the unweighted average, though on rare cases such as MSRpar it can decrease the performance. Table 5 : Experimental results (Pearson's r × 100) on textual similarity tasks. The highest score in each row is in boldface. The methods can be supervised (denoted as Su.), semi-supervised (Se.), or unsupervised (Un.). "GloVe+WR" stands for the sentence embeddings obtained by applying our method to the GloVe word vectors; "PSL+WR" is for PSL word vectors. See the main text for the description of the methods.

Effects of smooth inverse frequency and common component removal. There are two key ideas in our methods: smooth inverse frequency weighting (W) and common component removal (R). It is instructive to see their effects separately. Let GloVe+W denote the embeddings using only smooth inverse frequency, and GloVe+R denote that using only common component removal. Similarly define PSL+W and PSL+R. The results for these methods are shown in Table 6 . When using GloVe vectors, W alone improves the performance of the unweighted average baseline by about 5%, R alone improves by 10%, W and R together improves by 13%. When using PSL vectors, W gets 10%, R gets 10%, W and R together gets 13%. In summary, both techniques are important for obtaining significant advantage over the unweighted average.

A.2 SUPERVISED TASKS:

Setup of supervised tasks mostly follow (Wieting et al., 2016) to allow fair comparison: the sentence embeddings are fixed and fed into some classifier which are trained. For the SICK similarity task, given a pair of sentences with embeddings v L and v R , first do a linear projection:

h L = W p v L , h R = W p v R

where W p is of size 300 × d p and is learned during training. d p = 2400 matches the dimension of the skip-thought vectors. Then h L and h R are used in the objective function from (Tai et al., 2015) . Almost the same approach is used for the entailment task.For the sentiment task, the classifier has a fully-connected layer with a sigmoid activation followed by a softmax layer.

Recall that our method has two steps: smooth inverse frequency weighting and common component removal. For the experiments here, we do not perform the common component removal, since it can be absorbed into the projection step. For the weighted average, the hyperparameter a is enumerated in {10 -i , 3 × 10 -i : 2 ≤ i ≤ 3}. The other hyperparameters are enumerated as in (Wieting et al., 2016) , and the same validation approach is used to select the final values. 

A.3 ADDITIONAL SUPERVISED TASKS:

Here we report the experimental results on two more datasets, comparing to known results on them.

SNLI. The first experiment is for the 3-class classification task on the SNLI dataset (Bowman et al., 2015) . To compare to the results in (Bowman et al., 2015) , we used their experimental setup.

In particular, we applied our method to 300 dimensional GloVe vectors and used an additional tanh neural network layer to map these 300d embeddings into 300 dimensional space, then used the code provided by the authors of (Bowman et al., 2015) , trained the classifier on our 100 dimensional sentence embedding for 120 passes over the data, using their default hyperparameters. The results are shown in Table 7 . Our method indeed gets slightly better performance.

Our test accuracy is worse than those using more sophisticated models (e.g., using attention mechanism), which are typically 83% -88%; see the website of the SNLI project9 for a summary. An interesting direction is to study whether our idea can be combined with these sophisticated models to get improved performance. et al., 2015) . All methods used 100 dimensional sentence embeddings.

Sentence model:

IMDB. The second experiment is the sentiment analysis task on the IMDB dataset, studied in (Wang & Manning, 2012) . Since the intended application is semi-supervised or transfer learning, we also compared performance with fewer labeled examples.

# labeled examples NB-SVM Our method 50k 0.91 0.85 1k 0.84 0.82 200 0.73 0.77 Table 8 : Accuracy in sentiment analysis on the IMDB dataset for NB-SVM (Wang & Manning, 2012) and our method.

Our method gets worse performance on the full dataset, but its decrease in performance is better with less labeled examples, showing the benefit of using word embeddings. Note that our sentence embeddings are unsupervised, while that in the NB-SVM method takes advantage of the labels.

Another comment is that sentiment analysis appears to be the best case for Bag-Of-Word methods, whereas it may be the worst case for word embedding methods (See Table 2 ) due to the well-known antonymy problem -distributional hypothesis fails for distinguishing "good" from "bad."

:

The code is available on https://github.com/PrincetonML/SIF

We empirically discovered the significant common component c0 in word vectors built by existing methods, which inspired us to propose our theoretical model of this paper.

3 Note that max c: c =1 C + c, g = g/ g for any constant

C. 4 Here the first principal component is computed without centralizing cs'.

We used the vectors that are publicly available at http://nlp.stanford.edu/projects/glove/. They are 300dimensional vectors that were trained on the 840 billion token Common Crawl corpus.

6 "W" stands for the smooth inverse frequency weighting scheme, and "R" stands for removing the common components.

It is possible to tune the parameter a to get better results. The effect of a and the corpus for estimating word frequencies are studied in Section 4.1.1.

On the random order datasets, the hyperparameters are enumerated as in the other experiments, and the best results are reported.

http://nlp.stanford.edu/projects/snli/

Figures and Tables:

Experimental results (Pearson's r × 100) on textual similarity tasks. The highest score in each row is in boldface. The methods can be supervised (denoted as Su.), semi-supervised (Se.), or unsupervised (Un.). "GloVe+WR" stands for the sentence embeddings obtained by applying our method to the GloVe word vectors; "PSL+WR" is for PSL word vectors. See the main text for the description of the methods.

Table content: <table><tr><td>except tfidf-GloVe</td><td>Our approach</td></tr></table>

Results on similarity, entailment, and sentiment tasks. The sentence embeddings are computed unsupervisedly, and then used as features in downstream supervised tasks. The row for similarity (SICK) shows Pearson's r × 100 and the last two rows show accuracy. The highest score in each row is in boldface. Results in Column 2 to 6 are collected from

Table content: <table><tr><td/><td>PP</td><td>DAN</td><td>RNN</td><td>LSTM (no)</td><td>LSTM (o.g.)</td><td>skip-thought</td><td>Ours</td></tr><tr><td>similarity (SICK)</td><td>84.9</td><td>85.96</td><td>73.13</td><td>85.45</td><td>83.41</td><td>85.8</td><td>86.03</td></tr><tr><td>entailment (SICK)</td><td>83.1</td><td>84.5</td><td>76.4</td><td>83.2</td><td>82.0</td><td>-</td><td>84.6</td></tr><tr><td>sentiment (SST)</td><td>79.4</td><td>83.4</td><td>86.5</td><td>86.6</td><td>89.2</td><td>-</td><td>82.2</td></tr></table>

The STS tasks by year. Note that tasks with the same name in different years are actually different tasks.

Table content: <table><tr><td>STS'12</td><td>STS'13</td><td>STS'14</td><td>STS'15</td></tr><tr><td>MSRpar</td><td colspan="2">headline deft forum</td><td>anwsers-forums</td></tr><tr><td>MSRvid</td><td>OnWN</td><td>deft news</td><td>answers-students</td></tr><tr><td>SMT-eur</td><td>FNWN</td><td>headline</td><td>belief</td></tr><tr><td>OnWN</td><td>SMT</td><td>images</td><td>headline</td></tr><tr><td>SMT-news</td><td/><td>OnWN</td><td>images</td></tr><tr><td/><td/><td>tweet news</td><td/></tr></table>

Experimental results (Pearson's r × 100) on textual similarity tasks using only smooth inverse frequency, using only common component removal, or using both.

Table content: <table><tr><td/><td/><td colspan="2">Unsupervised</td><td/><td/><td colspan="2">Semi-supervised</td><td/></tr><tr><td>Tasks</td><td>avg-GloVe</td><td>GloVe+W</td><td>GloVe+R</td><td>GloVe+WR</td><td>avg-PSL</td><td>PSL+W</td><td>PSL+R</td><td>PSL+WR</td></tr><tr><td>MSRpar</td><td>47.7</td><td>43.6</td><td>36.4</td><td>35.6</td><td>41.6</td><td>40.9</td><td>42.5</td><td>43.3</td></tr><tr><td>MSRvid</td><td>63.9</td><td>78.7</td><td>79.4</td><td>83.8</td><td>60.0</td><td>80.4</td><td>76.4</td><td>84.1</td></tr><tr><td>SMT-eur</td><td>46.0</td><td>51.1</td><td>48.5</td><td>49.9</td><td>42.4</td><td>45.0</td><td>45.1</td><td>44.8</td></tr><tr><td>OnWN</td><td>55.1</td><td>54.3</td><td>68.3</td><td>66.2</td><td>63.0</td><td>67.8</td><td>71.0</td><td>71.8</td></tr><tr><td>SMT-news</td><td>49.6</td><td>42.2</td><td>45.6</td><td>45.6</td><td>57.0</td><td>56.2</td><td>50.7</td><td>53.6</td></tr><tr><td>STS'12</td><td>52.5</td><td>54.0</td><td>55.6</td><td>56.2</td><td>52.8</td><td>58.1</td><td>57.2</td><td>59.5</td></tr><tr><td>headline</td><td>63.8</td><td>63.8</td><td>68.9</td><td>69.2</td><td>68.8</td><td>72.6</td><td>72.7</td><td>74.1</td></tr><tr><td>OnWN</td><td>49.0</td><td>68.0</td><td>75.4</td><td>82.8</td><td>48.0</td><td>69.8</td><td>73.5</td><td>82.0</td></tr><tr><td>FNWN</td><td>34.2</td><td>23.0</td><td>34.9</td><td>39.4</td><td>37.9</td><td>49.3</td><td>40.7</td><td>52.4</td></tr><tr><td>SMT</td><td>22.3</td><td>29.5</td><td>36.4</td><td>37.9</td><td>31.0</td><td>39.2</td><td>37.3</td><td>38.5</td></tr><tr><td>STS'13</td><td>42.3</td><td>44.0</td><td>53.9</td><td>56.6</td><td>46.4</td><td>57.7</td><td>56.0</td><td>61.8</td></tr><tr><td>deft forum</td><td>27.1</td><td>29.1</td><td>39.8</td><td>41.2</td><td>37.2</td><td>45.8</td><td>45.3</td><td>51.4</td></tr><tr><td>deft news</td><td>68.0</td><td>68.5</td><td>66.6</td><td>69.4</td><td>67.0</td><td>75.1</td><td>67.4</td><td>72.6</td></tr><tr><td>headline</td><td>59.5</td><td>59.3</td><td>64.6</td><td>64.7</td><td>65.3</td><td>68.9</td><td>68.5</td><td>70.1</td></tr><tr><td>images</td><td>61.0</td><td>74.1</td><td>78.4</td><td>82.6</td><td>62.0</td><td>82.9</td><td>80.2</td><td>84.8</td></tr><tr><td>OnWN</td><td>58.4</td><td>68.0</td><td>77.6</td><td>82.8</td><td>61.1</td><td>77.6</td><td>77.7</td><td>84.5</td></tr><tr><td>tweet news</td><td>51.2</td><td>57.3</td><td>73.2</td><td>70.1</td><td>64.7</td><td>73.6</td><td>77.9</td><td>77.5</td></tr><tr><td>STS'14</td><td>54.2</td><td>59.4</td><td>66.7</td><td>68.5</td><td>59.5</td><td>70.7</td><td>69.5</td><td>73.5</td></tr><tr><td>answers-forum</td><td>30.5</td><td>41.4</td><td>58.4</td><td>63.9</td><td>38.8</td><td>56.0</td><td>61.0</td><td>70.1</td></tr><tr><td>answers-student</td><td>63.0</td><td>61.5</td><td>73.2</td><td>70.4</td><td>69.2</td><td>73.3</td><td>76.8</td><td>75.9</td></tr><tr><td>belief</td><td>40.5</td><td>47.7</td><td>69.5</td><td>71.8</td><td>53.2</td><td>64.3</td><td>71.3</td><td>75.3</td></tr><tr><td>headline</td><td>61.8</td><td>64.0</td><td>70.1</td><td>70.7</td><td>69.0</td><td>74.5</td><td>74.6</td><td>75.9</td></tr><tr><td>images</td><td>67.5</td><td>75.4</td><td>77.9</td><td>81.5</td><td>69.9</td><td>83.4</td><td>79.9</td><td>84.1</td></tr><tr><td>STS'15</td><td>52.7</td><td>58.0</td><td>69.8</td><td>71.7</td><td>60.0</td><td>70.3</td><td>72.7</td><td>76.3</td></tr><tr><td>SICK'14</td><td>65.9</td><td>70.5</td><td>70.6</td><td>72.2</td><td>66.4</td><td>73.1</td><td>70.3</td><td>72.9</td></tr><tr><td>Twitter'15</td><td>30.3</td><td>33.8</td><td>50.6</td><td>48.0</td><td>36.3</td><td>45.7</td><td>51.9</td><td>49.0</td></tr></table>

Accuracy in 3-class classification on the SNLI dataset for each model. The results in the first three rows are collected from (Bowman

Table content: <table><tr><td/><td>Train Test</td></tr><tr><td colspan="2">100d Sum of words 79.3 75.3</td></tr><tr><td>100d RNN</td><td>73.1 72.2</td></tr><tr><td>100d LSTM RNN</td><td>84.8 77.6</td></tr><tr><td>Our method</td><td>83.9 78.2</td></tr></table>

Peer review (iclr2017):
This is a good paper with an interesting probabilistic motivation for weighted bag of words models. The (hopefully soon) added comparison to Wang and Manning will make it stronger. Though it is sad that for sufficiently large datasets, NB-SVM still works better. In the second to last paragraph of the introduction you describe a problem of large cooccurrence counts which was already fixed by the Glove embeddings with their weighting function f. Minor comments: *The capturing the similarities* -- typo in line 2 of intro. *Recently, (Wieting et al.,2016) learned* -- use citet instead of parenthesized citation