Title: UNSUPERVISED DOCUMENT REPRESENTATION USING PARTITION WORD-VECTORS AVERAGING Anonymous

Abstract: Learning effective document-level representation is essential in many important NLP tasks such as document classification, summarization, etc. Recent research has shown that simple weighted averaging of word vectors is an effective way to represent sentences, often outperforming complicated seq2seq neural models in many tasks. While it is desirable to use the same method to represent documents as well, unfortunately, the effectiveness is lost when representing long documents involving multiple sentences. One reason for this degradation is due to the fact that a longer document is likely to contain words from many different themes (or topics), and hence creating a single vector while ignoring all the thematic structure is unlikely to yield an effective representation of the document. This problem is less acute in single sentences and other short text fragments where presence of a single theme/topic is most likely. To overcome this problem, in this paper we present P-SIF, a partitioned word averaging model to represent long documents. P-SIF retains the simplicity of simple weighted word averaging, while taking a document's thematic structure into account. In particular, P-SIF learns topicspecific vectors from a document and finally concatenates them all to represent the overall document. Through our experiments over multiple real-world datasets and tasks, we demonstrate P-SIF's effectiveness compared to simple weighted averaging and many other state-of-the-art baselines. We also show that P-SIF is particularly effective in representing long multi-sentence documents. We will release P-SIF's embedding source code and data-sets for reproducing results.

1. INTRODUCTION:

Distributed word embeddings such as word2vec (Mikolov et al., 2013b) have shown their success in representing words as latent continuous features in low-dimensional fixed spaces, which can capture their semantic meaning. These embeddings have shown impressive success for improving the performance of machine learning algorithms such as text classification tasks. The success of these word representations is recently formally explained using the random walk-based latent variable model by (Arora et al., 2016a) .

Many approaches such as (Socher et al., 2013; Le & Mikolov, 2014; Ling et al., 2015; Liu et al., 2015a) are proposed which go beyond words to capture the semantic meaning of sentences. These techniques either used simple composition of the word-vectors or sophisticated neural network architectures for sentence representation. Recently, (Arora et al., 2017) proposed smooth inverse frequency of word vector averaging model to embed a sentence. They further improved their embedding by removing the first principal component of the weighted average vectors. However, all these approaches are limited to capturing the meaning of a single sentence and represent the sentence in the same space as words, thus reducing their expressive power. Generally, a longer text contain words from multiple different themes (or topics), and creating a single vector from simple averaging of word-vectors will disregard all the thematic structure. Hence, these techniques are largely unable to capture the semantic meaning of larger pieces of text (phrases, sentences and paragraphs).

To address this problem, in this paper we present a novel document embedding method called partition SIF weighted averaging (P-SIF), which can efficiently embed documents with multiple sentences. P-SIF learns topic-specific vectors from a document and finally concatenates them all to represent the overall document. Thus, P-SIF retains the simplicity of simple weighted word averaging, while taking a document's thematic structure into account. (Mekala et al., 2017) recently proposed a clustering based word-vector averaging approach for embedding larger text documents. However, they did not provide any justification or experiments to explain the functionality of their embedding approach. This paper provides an analytical explanation about the success of the mentioned representations and also achieves significant improvements over previous embedding techniques on several natural language tasks. Following are the characteristics of P-SIF.

• P-SIF is capable of embedding larger documents with multiple sentences, as it pays attention to the topic/thematic structure of the document. • P-SIF is simple because it is based on weighted word vectors averaging rather than complicated tensor factorization or neural networks methods. • P-SIF is an unsupervised method since it is obtained using pre-trained unsupervised word embeddings without using any label information. • P-SIF outperforms many existing approaches on textual similarity, textual classification and other supervised tasks.

The remaining part of the paper is organized as follows. Section 2 discusses related work in document representations. Section 3 provides motivation for the partition averaging through a qualitative example. Section 4 describes our embedding algorithm. Section 5 introduces and explains the background needed. These sections are followed by experiments in Section 6 and the analysis with the discussion in Section 7.

2. RELATED WORK:

Word embeddings. There are mainly two methodologies proposed for unsupervised wordembedding representations, either by internal representations learned through neural network models of text (Bengio et al., 2003; Collobert & Weston, 2008) , and Mikolov et al. (2013b) or by low-rank approximation of co-occurrence statistics by (Levy & Goldberg, 2014; Hashimoto et al., 2016) , and Arora et al. (2016a) . (Levy & Goldberg, 2014) show that both techniques are equivalent.

Sentence embeddings. Earlier work has computed sentence embedding by coordinate wise vector and matrix based compositional operation over word vectors, e.g., (Levy & Goldberg, 2014) use unweighted averaging of word vectors for representing short phrases, (Singh & Mukerjee, 2015) proposed tfidf-weighted averaging of word vectors to form document vectors, (Socher et al., 2013) proposed a recursive neural network (RNN) defined over a parse tree and trained with supervision.

Next, (Le & Mikolov, 2014) proposed PV-DM and PV-DBOW model which treats each sentence as a shared global latent vector (or pseudo word). Other approaches use seq2seq models such as Recurrent Neural Networks (Mikolov et al., 2010) and Long Short Term Memory (Gers et al., 2002) which can handle long term dependency, hence, capturing the syntax structure. Other neural network models include use of hierarchy and convolution neural networks, such as (Kim, 2014) and (Kalchbrenner et al., 2014) . (Wieting et al., 2015) learned paraphrastic sentence embedding by modifying word embeddings based on the supervision from the Paraphrase pairs dataset (PPDB).

Recently, a lot of work is harnessing topic modeling (Blei et al., 2003) along with word vectors to learn better word and sentence representations, e.g., TWE (Liu et al., 2015a) , NTSG (Liu et al., 2015a) , WTM (Fu et al., 2016) , w2v-LDA (Nguyen et al., 2015) , TV+MeanWV (Li et al., 2016a) , LTSG (Law et al., 2017) , Gaussian-LDA (Das et al., 2015) , Topic2Vec (Niu et al., 2015) , Lda2vec (Moody, 2016) , and MvTM (Li et al., 2016b) .

Document embeddings. (Kiros et al., 2015) proposed skip-thought document embedding vectors which transformed the idea of abstracting the distributional hypothesis from word level to sentence level. (Wieting et al., 2016a) proposed a neural network model which optimizes the word embeddings based on the cosine similarity of the sentence embeddings. (Gupta et al., 2016) proposed a method called Bag of Words Vector (BoWV), which employs a clustering based technique and tf-idf values to form a composite document vector. They represented documents in higher dimensions by using hard clustering over word embeddings. (Mekala et al., 2017) later extended the model to SCDV by using a fuzzy clustering technique and direct idf weighting of word vectors. The learned representations tried to capture a global context of sentence, similar to an n-gram model. Their method outperformed previous state of art on a variety of NLP tasks. Let's consider a corpus (C) with N documents with the corresponding most frequent words vocabulary (V ). Figure 1 , represents the word-vectors space of V , where similar meaning words occur closer to each other. We can apply sparse coding to partition the word vectors space to a five (total topics K = 5) topic vector space, as shown in figure 1. These five topic vector spaces represent the five topics present in corpus (C). Few words are polysemic and belong to multiple topics with some proportion (see Table 1 ). In figure 1 we represent the topic number of the word in subscript. Let's consider a document d n : 'Data journalists deliver data science news to general public. They often take part in interpreting the data models. In addition, they create graphical designs and interview the directors and CEO's.'

One can directly average the word vectors to represent the document ( v dn ), as in (SIF), shown in equation equation 1. Here, + represents element-wise vector addition.

EQUATION

In representation equation 1 it can be seen that we are averaging word vectors which have a very different semantic meaning, e.g., words belonging to partition 1 such as graphical, design, and data are averaged with different semantic meaning words of partition 2 such as datascience, model, and data.

In addition, the document is represented in the same d dimensional space as word vectors. Overall, averaging represents the documents as a single point in the vector space and does not consider the 5 different semantic topics or themes.

However, we do a weighted (topic proportion) average of words within a partition and concatenate the average word vectors across partitions to represent the document ( v dn ), as in (P-SIF), shown in equation equation 2. Here, + represents a element-wise vector addition, × represents scalar-vector multiplication, and ⊕ represents concatenation.

EQUATION

In representation equation 2, the final representation takes 5 different semantic topics into account.

Words belonging to different semantic topics are separated by concatenation (⊕) as they represent separate meanings, whereas words coming from same topics are simple averages since they represent the same meaning. E.g., average of words belonging to partition 1 such as graphical, design, and data are concatenated to average of words of partition 2 such as data -science, model, and data. The final document vector v dn is represented in a higher 5 × d dimension vector space, thus having more representational power, where d is the dimension of word vectors. From equation 2, it is clear that the final representation takes the weight to which each word belongs to various topics into account as well, thus handling the multi-sense nature of words, e.g., words such as data belong to partition 1 with probability 0.3 and partition 2 with probability 0.7. Hence, partitioned averaging with topic weighting is essential for representing longer text documents. Thus, P-SIF takes documents' thematic structure into account while doing simple weighted word vectors averaging.

4. THE PROPOSED ALGORITHM: P-SIF:

In this section, we present the proposed document embedding learning algorithm. We introduce the formal notation needed for the discussion.

• C represents text corpus and V represents vocabulary of words in corpus.

• v w ∈ R d represents the word vector of word w, where d is dimension of word vector.

• p(w) represents the unigram probability of word w in the corpus.

• c 0 and v c0 ∈ R d represent common context and it's corresponding context vector.

The details to construct document embeddings are given in algorithm 1. The feature formation algorithm can be divided into three major steps :

Dictionary Learning for Word Vectors: Given word vectors v w ∈ R d , a sparsity parameter k, and an upper bound m , we find a set of unit norm vectors A 1 , A 2 , . . . , A m , such that vw = m j=1 α (w,j) Aj + ηw , where at most k of the coefficients α (w,1) , . . . , α (w,m) are nonzero (so-called sparsity constraint), and η w is a noise vector.

Sparse coding is usually solved for given m, k by using alternating minimization (Arora et al., 2016b) and (Aharon et al., 2006) to find the A i s that minimizes the following L 2 -reconstruction error : vw -m j=1 α (w,j) Aj . Here, A 1 , . . . , A m will represents important topic basis in the corpus, which we refer to as the atoms of the topic. Furthermore, restricting m to be much smaller than the number of the words ensures that the same topic needs to be used for multiple words. A j is an interesting or significant topic because the sparse coding ensures that each basis element is softly chosen by many words.

Word Topics Vector Formation: For each word w, we created K different word-cluster vectors of d dimensions cv wk by weighting the word embedding with its learned dictionary coefficient α w,k of the k th context.1 . We then concatenated all the K word-cluster vectors cv wk into a K×d dimensional embedding to form a word-topics vector tv w ∈ R K×d .

cv w,k ← v w × α w,k ; tv w ← K k=1 cv wk

Here, represents concatenation operation, and × represents vector-scalar multiplication.

Smooth Inverse Frequency (SIF) re-weighted: Finally, for all words appearing in document D n , we weighted the word-topics vectors tv i by smooth inverse frequency a a+p(w) . Next, we removed the common context from the weighted average document vectors by removing the first principal component from the weighted average vectors.2 . (Arora et al., 2017) empirically shows that SIF weighting outperforms the tf-idf weighting. They provide a theoretical explanation for their superior performance. 

v dn ← v dn -u. u T v

5. P-SIF DISCUSSIONS:

For single sentence documents all words of a document belong to a single topic. However, for multiple sentence-documents, words of a document generally originate from multiple topics. To capture this phenomenon, topic modeling algorithms such as LDA (Blei et al., 2003) are used to represent the documents. These representations essentially represent the global contexts of the documents as a distribution over topics. However, these representations do not take the local context initiating from the distributional semantics into account such as word vectors. To consider both local and global contexts, we represented each word as a word-topic-vector tv w . In this section, we describe our simple and efficient unsupervised method for document representation in details.

Sparse Dictionary Learning (Algorithm 1: Lines 1 -3): (Arora et al., 2016b) shows that atoms of sparse coding over word-vectors represent all prominent topics in the corpus. Furthermore, they showed that multiple word senses of a word reside as a linear superposition within the word embedding and can be recovered by simple sparse coding. Therefore, one can use the sparse coding of word vectors to detect multiple senses of words and total senses of all the words (number of basis). To find these topics, we used sparse coding algorithms such as k-svd (Aharon et al., 2006) over word vectors v w . For a given word (w), the k non-zero coefficient essentially represents the distribution of words over topics. The k non-zero α w for a given word w, basically represents the multi-sense nature of the words.

Word Topics Vector (Algorithm 1: Lines 3 -9): Since our multi-sentence documents have words coming from multiple topics, we did not directly average word-vectors with a simple averaging technique. We concatenated the word embeddings over the topic distribution of the words. This helps to represent semantically similar words in the same topic, while words which are semantically different are represented in different topics. Concatenation of word embeddings over topics also helps in the expression of the multi sense nature of the words. We weigh word-vectors by coefficients of the learned dictionary to capture the cross correlation (α i α j ) between contexts.

SIF Weight Averaging and Common Component Removal (Algorithm 1: Lines 9 -16): Instead of assuming a single topic for the whole document, we showed that the total number of topics over a given corpus is K (as shown by (Arora et al., 2016b) ) and for a given word, k out of K of them would be active. Compared to SIF which directly averages SIF weighted words-vectors, we first partitioned according to the topics through dictionary learning over word-topics vectors. We finally averaged SIF weighted word-topics vectors. Lastly, we removed the first principal components from the document vectors to remove the common component.

Sparse Dictionary Learning vs. Fuzzy Clustering. Sparse coding can also be treated as a linear algebraic analogue of overlapping clustering, where the A i 's act as cluster centers and each v w is assigned to each cluster in a soft way (using the coefficients α (w,j) , of which only k are nonzero) to a linear combination of at most k clusters. In practice, sparse coding optimization produces coefficients α (w,j) which are almost all positive, even though unconstrained. One can use fuzzy clustering where each word belongs to every cluster with some probability P (c k |w i ), which can be thought as a substitute for α (w,k) , similar to the approach in SCDV (Mekala et al., 2017) . Practically, when the number of contexts is large, dictionary learning performs better than fuzzy clustering due to 1) better optimization and non-redundant clusters and 2) automatic handling of the tail of P (c k |w i ) distribution through sparsity constraints (Olshausen & Field, 1997; Gao et al., 2010; Yang et al., 2009) . We observed that for a single sentence document with a small number of topics it is better to use fuzzy clustering because it has a better optimization and learns efficient clusters. Additionally, in fuzzy clustering, the P (c k |w i ) is a probability between 0 and 1, which sums to 1 over all contexts. However, to overcome the noise from the non-zero tail probability one can apply hard thresholding on the final vectors, which are directly learned by few non-zero coefficients (α w,j ) of the dictionary. Our approach is averaging the words which belong to similar topics and concatenating these averages across different topics. Therefore, our approach is a strict generalization of the sentence embedding approach by (Arora et al., 2017) which is a special case where total topics or themes is K = 1.

6. EXPERIMENTAL RESULTS:

We performed several experiments on several text similarity and classification tasks. We will address the following research questions through our experiments Q1. Why partition and sparsity is required during word vectors averaging for representing documents? Q2. Does P-SIF represent large text documents better compared to other techniques used for representing a single sentence? Q3. What is the effect of SIF weighting and common component removal on document representation?

6.1 TEXTUAL SIMILARITY TASK Datasets. We performed our experiments on the SemEval dataset (2012) (2013) (2014) (2015) (2016) (2017) . These experiments involved 27 semantic textual similarity (STS) tasks (2012 -2016) (Agirre et al., 2012; 2013; 2014; 2015; 2016) , the SemEval 2015 Twitter task (Xu et al., 2015) , and the SemEval 2014 Semantic relatedness task (Marelli et al., 2014) . The objectives of these tasks are to predict the similarity between two sentences. We used the Pearsons coefficient (Fleiss & Cohen, 1973) between the predicted scores and the ground-truth scores for the evaluation. Please refer to the supplementary section F.1 for the experimental setting details.

1. Unsupervised: We used ST, avg-Glove, tfidf-Glove and Glove + WR as a baseline. ST denotes the skip-thought vectors by (Kiros et al., 2015) , avg-Glove denotes the unweighted average of the Glove Vectors by (Pennington et al., 2014b) 3 , and tfidf-Glove denotes the tf- idf weighted average of Glove vectors. We also compared our method with the SIF weighting (W ) common component removal (R) Glove vectors (Glove + W R) by (Arora et al., 2017) . For STS 16 we also compared our embedding with Skip-Thoughts (Kiros et al., 2015) and Sent2Vec (Pagliardini et al., 2018) embeddings. 2. Semi-Supervised: We used avg-PSL, PSL + WR, and the avg-PSL used the unweighted average of the PARAGRAM-SL999 (PSL) word vectors by (Wieting et al., 2015) as a baseline, obtained by training on PPDB dataset (Ganitkevitch et al., 2013) . The word vectors are trained using unlabeled data. Furthermore, Sentence embeddings are obtained from unweighted word vectors averaging. We also compared our method with the SIF weighting (W) common component removal (R) PSL word vectors (PSL + WR) by (Arora et al., 2017) . 3. Supervised: We compared our method with PP, PP-proj., DAN, RNN, iRNN, LSTM (o.g), LSTM(no) and GRAN. All these methods are initialized with PSL word vectors and then trained on PPDB dataset (Ganitkevitch et al., 2013) . PP (Wieting et al., 2016a) is the average of word vectors while PP-proj is the average of word vectors followed by a linear projection. The word vectors are updated during the training. DAN denotes the deep averaging network of (Iyyer et al., 2015) . RNN is a Recurrent neural network, iRNN is identity activated Recurrent Neural Network based on identity initialize weight matrices. The LSTM is the version from (Gers et al., 2002) , either with output gates (denoted as LSTM (o.g.))

or without (denoted as LSTM (no)). GRAN denotes the state of art supervised averaging based Gated Recurrent Averaging Network from (Wieting & Gimpel, 2017) . For STS 16 we also compared our embedding with Tree-LSTM (Tai et al., 2015) embedding.

More details on the values of the hyper-parameters used for the experiments are described in details in the supplementary section F.1.

Results and Analysis:

The average results for each year are reported in Table 2 . We denoted our embeddings by P-SIF + PSL (+ PSL denotes using the PSL word vectors). We reported the average results for the STS tasks. The detailed results on each sub-dataset are in the supplementary section A. We observed that P-SIF + PSL outperforms PSL + WR on all datasets, thus supporting the usefulness of our partitioned averaging. Our method also outperformed Neural Network models such as LSTM and RNN. Our proposed techniques performed the best over 16 datasets out of 22 compared to methods from (Wieting et al., 2016a) and (Arora et al., 2017) (refer to supplementary section A for details). Our method outperformed supervised averaging based Gated Recurrent Averaging Network (GRAN) on 11 datasets. Furthermore, our results outperformed recently pro-posed unsupervised methods such as Skip-Thoughts (Kiros et al., 2015) , Sent2Vec (Pagliardini et al., 2018) , and supervised method such as Tree-LSTM (Tai et al., 2015) . We observed that partitioning through fuzzy cluster algorithm such as GMM generates a better performance compared to partitioning through sparse dictionary algorithms such as k-svd for most Semantic Textual Similarity (STS) tasks. The main reason for this peculiar observation was related to the fact that the STS datasets contains documents which are single sentences of a maximum length of 40 words. As discussed in section 5 (sparse dictionary learning vs. fuzzy clustering), for single sentence documents with a small number of topics, fuzzy clustering optimizes better than sparse dictionary learning. Therefore, we used GMM for the STS task which results in partitioning words into suitable clusters. Both ksvd and GMM outperform SIF, however, the improvement was more observable with the GMM-based partitioning. It should be noted that this observation does not hold for 20NewsGroup and Reuters datasets since the documents in these datasets have multiple sentences with a total number of words >> 40. We also report some qualitative results in the supplementary section D with a supporting example in section E. In addition, we report our results on the SICK supervised classification task in supplementary section B.

6.2. TEXTUAL CLASSIFICATION TASK:

The document embeddings obtained by our method can be used as direct features for downstream many supervised tasks.

Datasets We ran multi-class experiments on 20NewsGroup dataset4 and multi-label classification experiments on Reuters-21578 dataset5 . We used script6 for preprocessing the dataset. Please refer to supplementary section F.1 for the hyperparameter's details.

Baselines We considered the following baselines: The Bag-of-Words (BoW) model ( (Harris, 1954) ), the Bag of Word Vector (BoWV) (Gupta et al., 2016) model, Sparse Composite Document Vector (SCDV) (Mekala et al., 2017) 7 paragraph vector models (Le & Mikolov, 2014), Topical word embeddings (TWE-1) (Liu et al., 2015b) , Neural Tensor Skip-Gram Model (NTSG-1 to NTSG-3) (Liu et al., 2015a) , tf-idf weighted average word-vector model (Singh & Mukerjee, 2015) and weighted Bag of Concepts (weight-BoC) (Kim et al., 2017) where we built document-topic vectors by counting the member words in each topic, and Doc2VecC (Chen, 2017) where averaging and training of word vectors are done jointly. Moreover, we used SIF (Arora et al., 2017) smooth inverse frequency weight with common component removal from weighted average vectors as a baseline. We also compared our results with other topic modeling based document embedding methods such as WTM (Fu et al., 2016 ), w2v-LDA (Nguyen et al., 2015) , LDA (Chen & Liu, 2014) , TV+MeanWV (Li et al., 2016a )), LTSG (Law et al., 2017) , Gaussian-LDA (Das et al., 2015) , Topic2Vec (Niu et al., 2015) , Lda2Vec (Moody, 2016) and MvTM (Li et al., 2016b) . Please refer to supplementary section F.2 for the hyperparameter's details. Multi-class classification We evaluated the classifier's performance using standard metrics such as accuracy, macro-averaging precision, recall and F-measure. Table 5 shows a comparison on multiple state-of-art document representations on the 20NewsGroup dataset. (Das et al., 2015) 72.2 70.8 70.7 70.0 weight-AvgVec (Singh & Mukerjee, 2015) 81.9 81.7 81.9 81.7 BoW (Harris, 1954) 79.7 79.5 79.0 79.0 weight-BOC (Kim et al., 2017) 71 Multi-label classification: We evaluated multi-label classification performance using Precision@K, nDCG@k (Bhatia et al., 2015) , Coverage error, Label ranking average precision score (LRAPS)8 and F1-score. Table 4 shows the evaluation results for multi-label text classification on the Reuters-21578 dataset.

We observed that P-SIF outperforms all other methods by a significant margin on both 20NewsGroup and Reuters datasets. We observed that the dictionary learns more diverse and non-redundant topics compared to fuzzy clustering (SCDV) since we only require 40 partitions rather than 60 partitions in SCDV to obtain the best performance. Simple tf-idf weighted averaging based document representation does not show significant improvement in performance by increasing word vector dimensions. We achieved a < 0.4 % improvement in accuracy when the word vectors dimensions increase from 200 to 500 on 20NewsGroup. We observed that increasing word vectors dimensions beyond 500 does not improve the performance of SIF and P-SIF. We further improved the performance on both datasets using Doc2vecC initialized word-vectors with the help of the P-SIF algorithm. We represented this approach by P-SIF (Doc2VecC) in Table 5 and Table 4 . On 20NewsGroup we required only 20 partitioned instead of 40 with Doc2VecC initialized word vectors. This shows that better word vector representations help in learning more diverse and non-redundant partitions. We also reported our results on each of the 20 classes of 20NewsGroup separately in the supplementary section C.

7. ANALYSIS AND DISCUSSION:

Effect of Document-Length: We conducted a small experiment to show that our model performs better compared to SIF when we have large size documents. We have divided 26 STS (sentence textual similarity) datasets by average document length, i.e., number of words in documents in bins of (10-20, 20-30, 30-40, 40-50) words. Next, we averaged the relative performance improvement in accuracy with respect to SIF P-SIF-SIF

SIF:

% and over all datasets in each bin. In Figure 2 , we observe that for complex multi-sentence documents with more words P-SIF relatively performs better. We have also observed that short texts require fewer number of partitions to achieve their best performance which was quite intuitive since short text documents will map into fewer topics. 

Effect of Partition::

We have concatenated the word embeddings over topic distribution of words which assigned semantically similar words to identical partitions and semantically different words to different partitions. Partitioning and concatenation of word embeddings over topics also helps in the representation of multi-sense words, which would have been left-out by simple averaging of the word embeddings in document representation otherwise. We represented our words by partitioned average of word-vector embeddings distributed over partitions, which grew the representation by the multiplication of the number of partitions, compared to word vector embeddings. Such high dimensional data structure regularizers, e.g., sparse encodings, help in overcoming the curse of dimensionality. We used k-svd (Aharon et al., 2006) a generalized k-means iterative method that alternates between the sparse coding of the examples based on the current dictionary and updating the dictionary atoms to fit the data. SCDV (Mekala et al., 2017) does manual sparse-encoding to ensure sparsity. (Mekala et al., 2017) clustering methodology uses GMM for partitioning the vocabulary space and capturing the topics. Instead of GMM, we used a dictionary learning based approach which imposes a sparsity constraint implicitly during optimization. Since, k-svd is internally and alternatively optimized between sparse encodings and topic representations, we are able to solve the curse of higher dimensionality by automatic sparse representations of the partitions. Emperically, on both datasets, we observed that the dictionary learns more diverse and non-redundant topics compared to fuzzy clustering. We required only 40 partitions rather than 60 in SCDV to obtain the best performance. Additionally, we achieved the best performance with (P-SIF(Doc2VecC)) with just (20 * 300) dimensions of word embeddings (mostly sparse) as compared to (60 * 300) dimensions of word-embeddings (mostly non-sparse) in SCDV. Thus, we obtained a performance gain of 1.5% with 0.33 of the size of the SCDV embeddings.

Effect of SIF and Common Component Removal::

We have also shown improvement in shorter text tasks via sentence textual similarity task. We have used SIF based weighted word averaging methods instead of tf-idf averaging (Mekala et al., 2017) or simple averaging of word-embeddings. SIF takes advantage of the generative process of document-formation and represents documents as simple smooth inverse frequency averaging of word embeddings. We also observe that removing the first principal component helps in removing the common component occurring due to frequent common words from document vectors. This observation reduces noise and redundancy from the composed document vectors which make vectors more discriminating.

8. CONCLUSIONS AND FUTURE WORK:

We proposed an unsupervised document feature formation technique based on a partitioned word vector averaging method. Our embedding retains the simplicity of simple weighted word averaging, while taking documents' thematic structure into account. Our simple and efficient approach achieved significantly better performance on several textual similarity tasks, textual classification tasks and out of domain tasks. In the future, three extensions of our work would be 1. Learning a supervised weighting scheme of words according to the task to improve words weighting, 2. Learning and using context sensitive word embeddings instead of normal skip-gram embeddings to resolve cluster disambuity, and 3. Learning a lower dimensional space manifold to represent word-topic vectors for continuous representations useful in deep leaning applications.

Supplementary: Unsupervised Document Representation using Partition Word-Vectors Averaging:

A UNSUPERVISED TASK: TEXTUAL SIMILARITY

In this supplementary section, we present the details results for STS tasks for each year. Each year, there are 4 to 6 STS tasks, as shown in Table 6 . Note that tasks with the same name in different years are different tasks in reality. We provide detailed results for each tasks in STS 12 -15 in Table 7 .

Our method outperforms all other methods from (Arora et al., 2017) and (Wieting et al., 2016a ) on all 16 out of 22 tasks. Our method performs significantly better in comparison to all unsupervised embedding methods. In addition, P-SIF is very close to the best performance by supervised methods on the rest of the datasets. The performance difference between best method and our method compared with best method is much small. Our method was also able to outperform state of art supervised averaging based Gated Recurrent Averaging Network (GRAN) (Wieting & Gimpel, 2017) on 11 datasets shown in Table 7 . Our results also outperform state of art methods on many recent supervised embedding methods on STS 16 task, as shown in Table 8 . 

B OTHER SUPERVISED TASKS:

We also consider three out of domain supervised tasks: the SICK similarity task, the SICK entailment task, and the Stanford Sentiment Treebank (SST) binary classification task by (Socher et al., 2013) . We use the setup similar to (Wieting et al., 2016a) and (Arora et al., 2017) for a fair comparison, including the linear projection maps which take the embedding into 2400 dimension (same as skip-thought vectors), and is learned during the training. We compare our method to PP, DAN, RNN, LSTM, and skip-thoughts and other baselines, details results in Table 9 .

Results and Analysis. Our method (P-SIF) gets better performance compared to PSL + WR on all the three tasks similarity, entailment and sentiment. We obtained best results for two of the supervised tasks, despite many of these methods (DAN, RNN, LSTM) are trained with supervision. Furthermore, the skip thought vectors use has a higher dimension of 2400 dimension instead of 300 dimensions (which we projected to 2400 for a fair comparison). Our method wasn't able to outperform sentiment task compared to supervised tasks because a) due to antonym problem word vectors capture sentimental meaning of words and b) also in our weighted average scheme, we didn't give more weights to sentiment words such as not,good, bad, there may be some important sentiment words which are down-weighted by SIF weighting scheme. However, we outperform PSL + WR by a significant margin and have lesser performance gap with the best supervised approach.

Table 7 : Experimental results (Pearsons r × 100) on textual similarity tasks. The highest score in each row is in boldface. The methods can be supervised (denoted as Su.), semi-supervised (Se.), or unsupervised (Un.). See main text for description of the methods. Many results are collected from (Wieting et al., 2016a) and (Wieting & Gimpel, 2017) (Wieting et al., 2016a) , and those in Column 7 for skip-thought are from (Kiros et al., 2015) , Column 8 for PSL + WR are from (Arora et al., 2017) . 

Tasks

C CLASS PERFORMANCE: 20NEWSGROUP:

We also report the precision and recall results of separate 20 classes of the 20 NewsGroup dataset. We compared our embedding (P-SIF) with Bag of Words, SCDV embeddings and our embeddings. • GT : represent the given ground truth similarity score in range of 0-5.

• NGT : represent normalized ground truth similarity score. NGT is obtain by dividing GT score by 5 so that it's in range of 0-1. • SIF sc : represent the SIF embedding similarity score in range of 0-1.

• P-SIF sc : represent the P-SIF embedding similarity score in range of 0-5.

• SIF err : represent absolute error SIF sc -N GT between normalized ground truth similarity score and SIF embedding similarity score. • P-SIF err : represent absolute error P-SIF sc -N GT between ground truth similarity score and P-SIF embedding similarity score. • Diff err : represent absolute difference between SIF err and P-SIF err . Examples where P-SIF perform better Diff err = P-SIF err -SIF err (used in Table 11 ). Examples where SIF perform better Diff err = SIF err -P-SIF err (used in Table 12 ) • Rel err : represent relative difference between SIF err and P-SIF err . Example where P-SIF perform better Rel err = Dif ferr SIFerr (used in Table 11 ). Examples where SIF perform better Rel err = Dif ferr P-SIFerr (used in Table 12 )

E QUALITATIVE EXAMPLE:

Let's consider a corpus (C) with N documents with corresponding most frequent words vocabulary (V ). Figure 3 represents the word-vectors space of V , where similar meaning words are closer. We can apply sparse coding and partition the words-vector space in five (total topics K = 5) topic vector spaces. Some words are polysemic and belong to multiple topics with some proportion, as shown in Figure 3 . For example, words such as baby, person, dog and kangaroo, belong to multiple topics with significant proportion. Words and corresponding vectors in these topic vector spaces are represented by topic numbers in subscript. Table 13 shows an example pair from the STS Task 2012 MSRVid dataset, and the corresponding SIF (averaging) and P-SIF (partition averaging) representation vectors. We can see that in SIF representation we are averaging words vectors which semantically have different meanings. The document is represented in the same d dimensional word-vectors space.

Overall, the SIF represent the document as a single point in the vector space and doesn't take account of different semantic meanings of the topics. Whereas, in P-SIF representation we treat the 5 different semantic topics distinctly. Words belonging to different semantic topics are separated by concatenation (⊕) as they represent different meanings, whereas words coming from same topic are average as it's represent same meaning. The final document vector v dn has more representational power as it's represented in a higher 5 × d dimensional vector space. Thus partitioned averaging with topic weighting is important for representing documents. Empirically, P-SIF gave score dissimilar sentences (d 1 n ,d 2 n ) 0.16 (0-1 scale) where the ground truth of 0.15 (rescale to 0-1 scale), whereas SIF gave similarity score of 0.57 (0-1 scale), farther then ground score. Thus we obtain a relative improvement of 98% in the error difference from ground truth. Here, simple averaging-based embedding of d 1 n and d 2 n , bring the document representation closer. But, partitioned based averaging P-SIF of d 1 n and d 2 n as, project the document in a higher dimensional space.

Table 13 : STS Task 2012 MSRVid dataset similarity example pair where P-SIF gave score dissimilar sentences (d 1 n ,d 1 n ) 0.16 (0-1 scale) where the ground truth of 0.15 (rescale to 0-1 scale), whereas SIF gave similarity score of 0.57 (0-1 scale). Thus, we obtain a relative improvement of 98% in the error difference. Here, ⊕ represent concatenation. v zero is the zero padding vector. We use the PARAGRAM-SL999 (PSL) from (Wieting et al., 2015) as word embeddings, obtained by training on PPDB (Ganitkevitch et al., 2013 ) dataset9 . We use the fix weighting parameter α value of 10 -3 , and the word frequencies p(w) are estimated from the common-crawl dataset. We tune the number of contexts (K) to minimize the reconstruction loss over word vectors. We fix the non-zero coefficient m = K/2, for the SIF experiments. For GMM based partitioning of the words vocabulary we tune the number of cluster parameter K through 5-fold cross validation.

F.2 TEXTUAL CLASSIFICATION TASK::

We fix the document embeddings and only learn the classifier. We learn word vector embedding using Skip-Gram with a window size of 10, Negative Sampling (SGNS) of 10 and minimum word frequency of 20. We use 5-fold cross-validation on the F 1 score to tune hyperparameters. We use LinearSVM for multi-class classification and Logistic regression with the OneVsRest setting for multi-label classification. We fix the number of dictionary elements either 40 or 20 (with Doc2vecC initialize word vectors) and non-zero coefficient to m = K/2 during dictionary learning for all experiments. We use the best parameter settings as reported in all our baselines to generate their results. We use 200 dimensions for tf-idf weighted word-vector model, 400 for paragraph vector model, 80 topics and 400 dimensional vectors for TWE, NTSG, LTSG and 60 topics and 200 dimensional word vectors for SCDV (Mekala et al., 2017) . We will released P-SIF's embedding source code with necessary parameters details and other data-sets used in the paper for reproducing results.

I PROOF SKETCH: DERIVATION OF OUR EMBEDDING:

To derive our embedding, we propose a generative model which treats corpus generation as a dynamic process, where the t th word is produced at step t. The process is driven by random walk over a unit norm sphere with centre at the origin. Let, v ct be the vector from origin to current walk point at time t. We called this vector as context vector, as it represents the context in the discussion. Below, we introduce formal notation needed for the discussion

• C represents text corpus and V represents vocabulary of words in corpus.

• v w ∈ R d represents the word vector of word w , where d is dimension of word vector.

• c t represents context and v ct ∈ R d represents context vector at time t, where d is dimension of context vector. • Z c represents partition function for the random context vector v ct , given by Z c = w exp( v ct , v w ).

• p(w) represents the unigram probability of word w in the corpus.

• c 0 and v c0 ∈ R d represent common context and its corresponding context vector based on syntax.

Using log linear model of Mnih & Hinton (2007) , we define the probability of observing a word w from the random walk with current context c t at time t as

EQUATION

It is easy to show that such random walk under some reasonable assumptions (Arora et al. (2016a) ) can gives word-word co-occurrence probabilities similar to empirical works like word2vec (Mikolov et al. (2013a) ) and Glove (Pennington et al. (2014a) ). To account for frequent stop-words which occur more often regardless of context and common context related to document syntax, two correction terms need be added: one based on p(w) and other on common context vector v c0 in Equation equation 3. These terms allow words with low inner product with c t , a chance to appear either from term p(w) if they are frequent or by common context c 0 , if they have large dot product with c 0 . Given a context vector c t , the probability of a word w in document d being generated by context c t is given by,

EQUATION

where, c t = β c 0 + (1 -β) c t , c 0 , c t = 0, λ and β are scalar hyper-parameters.

For generating a document from above random walk based latent variable model, we consider the following two assumptions :

1. Total number of theme/topics in the entire corpus is K. The K themes/topics can be determine by sparse dictionary learning as shown by Arora et al. (2016b) over word vectors v w = m j=1 α w,j A j + η w , where, A j 's are unit norm vectors representing basis of theme, α w,j are coefficient determining whether w is generated with A j10 , and η w is a noise vector. 2. Word vectors v w are uniformly distributed, thus making the partition function Z c , roughly same in all direction for given context c, where c belongs to one of the K themes/topics, as described earlier. The context vector does not change significantly much while words are generated from random walk, as shown by Arora et al. (2017) except during jumps a.k.a when theme/topic change.

For a document d, the likelihood of document is being generated by the K contexts, is given by :

EQUATION

Here, p(w|c j ) is the probability that word w is generated by context c j , the value of which is determine by 1) overall frequency of word w in corpus a.k.a prior probability (p(w)) and 2) relative frequency of w appear with context j w.r.t other contexts (determine by α (w,j) ).

Using simple calculus, treating p(w) as constant, we can show that ∇(f w (c j )) equals,

EQUATION

Then, by using the Taylor expansion, we can show here, a = 1-λ λZj Thus, the MLE estimate is approximately a weighted average of the word vectors generated from context j in the document d from random walk. We can get the overall context representation v c d of document, by simple concatenation over all K themes/topics.

fw(cj) ≈ fw(cj = 0) + ∇(fw(cj = 0)) T vc j (9) fw(cj) ≈ constant + ∇(fw(cj = 0)) T vc j (10)

EQUATION

Here, represents concatenation operation. For a document if no words is generated from the context c j the we can substitute the context vector v cj by 0 vector, for representation v c d in K × d dimensions.

Relation to SIF model: Arora et al. (2017) , shows under the two assumptions :

• uniform distribution of word vectors v w 's which implies that the partition function Z t is roughly same in all direction for the a sentence. • the context vector v ct , remain constant while the words in the sentence are emitted, implying replacing of v ct in sentence's by a single vector v cs and partition function by Z s the sentence embedding of a sentence can be obtained by v cs = {w∈s} a p(w)+a v w , here, a = 1-λ λZs . However, the above assumptions does not hold true for a document with multiple sentences, where one can expect to have more frequent random jumps during random walk12 . Instead of assuming a single context for whole document c h , we assume that the total number of theme/topics over a given corpus is bounded by K (as shown by Arora et al. (2016b) ) and the random walk can perform jumps to switch context from one context to the rest K -1 contexts. The partition function remain same in all directions for only words coming from a same context c j , instead of words coming from all K contexts. Thus, our approach is generalization of sentence embedding approach by Arora et al. (2017) (special case K = 1).

J KERNEL CONNECTION TO WORD MOVER DISTANCE:

Below, we introduce formal notation needed for the discussion:

• C represents text corpus and V represents vocabulary of words in corpus.

• wv w ∈ R d represents the word vector of word w , where d is dimension of word vector.

• tv w ∈ R K represents the theme/topic vector of word w, where K is number of themes/topics. t w,j = α w,j = P (j|w) Consider the following document similarity kernels: We can conclude the following from the respective kernels: We empirically showed that our proposed embedding (kernel) outperform the word mover distance (Kusner et al., 2015) and word mover embedding (Wu et al., 2018) and many other baselines in Table 14 on several datasets13 . 

K 1 (d A , d B ) = 1 nm

• K 1 (d A ,

K RECENT BASELINE FOR TEXTUAL SIMILARITY TASK:

We compared out P-SIF embedding with many other recently proposed baseline like ELMO (Peters et al., 2018) , p-means (Rücklé et al., 2018) , FastText (Joulin et al., 2016) , Skip-Thoughts (Kiros et al., 2015) , (Conneau et al., 2017) , Charphrase (Wieting et al., 2016b) , WME (Wu et al., 2018) and u-SIF Ethayarajh (2018) . We used the SentEval package (Conneau & Kiela, 2018) 

:

Empirically, we observed that this weighting generally improves the performance

We did not remove the common component from final vectors, when we used Doc2VecC(Chen, 2017) initialized word vectors with P-SIF, because frequent words' word-vectors become close to 0

http://qwone.com/$\sim$jason/20Newsgroups/

https://goo.gl/NrOfu

https://gist.github.com/herrfz/7967781

https://github.com/dheeraj7596/SCDV

https://goo.gl/4GrR3M

For a fair comparison with SIF we uses PSL vectors instead of unsupervised Glove and Word2Vec vectors

In practice, only k (much lesser than total number of themes/topic K) αw,j will be non-zero, because a word can't belong to all contexts (exception for frequent stop-words)

Note that argmax c: c =1 C + c, g = g g for any constant C

It is trivial to assume that these jumps occur more frequently in multiple sentences document, because of increased chances of context change

For datasets and baseline details refer to(Wu et al., 2018)

Figures and Tables:

Figure 1: Word vector space for corpus C. Words in different partitions are represented by different subscripts and separated by hyperplanes. Bold fonts represent words' presence in the document

Figure 2: Relative performance improvement pf P-SIF over SIF in accuracy P-SIF-SIF SIF % w.r.t average numbers of words in documents for 27 sts (sentence textual similarity) datasets.

Figure 3: Word vector space for corpus C. Words in different topic is represented by different subscript and separated by hyperplanes. Bold represent words from example documents. F EXPERIMENTAL DETAILS F.1 TEXTUAL SIMILARITY TASK:

d A , d B ) = K 3 (d A , d B ) + K 4 (d A , d B )

d B ) represent document similarity between document represented by average word vectors d x = i wv x i • K 1 (d A , d B ) represent document similarity between document represented by partition average word vectors • K 3 (d A , d B ) represent document similarity between document represented by relax word mover distance when words of d A are matched to d B • K 4 (d A , d B ) represent document similarity between document represented by relax word mover distance when words of d B are matched to d A • K 5 (d A , d B ) represent document similarity between document represented by word mover distance.

and embedding evaluation paper Perone et al. (2018) for baselines. Except u-SIF we outperform all other embedding. Our P-SIF results was very close to u-SIF in most tasks. Details results are provided in Table 15

Word embeddings { vw : w ∈ V }, Documents {dn : dn ∈ D}, a set of sentences D, parameter a and estimated probabilities {p(w) : w ∈ V } of the words, a sparsity parameter k, and an upper bound m. whose columns are { v dn : dn ∈ D}, and let u be the first singular vector; for each document dn ∈ D do v dn ← v dn -u u T v dn ; end

Table content: <table><tr><td/><td/><td>dn</td></tr><tr><td colspan="3">Algorithm 1: P-SIF Embedding</td></tr><tr><td colspan="3">Result: Document vectors { v dn : dn ∈ D}</td></tr><tr><td colspan="3">/ * Dictionary learning for word-vectors</td><td>* /</td></tr><tr><td colspan="2">for each word w in V do</td><td/></tr><tr><td colspan="3">vw = m j=1 αw,j Aj + ηw;</td></tr><tr><td>end</td><td/><td/></tr><tr><td colspan="3">/ * Word topic-vector formation</td><td>* /</td></tr><tr><td colspan="2">for each word w in V do</td><td/></tr><tr><td colspan="3">for each coefficient, α w,k of word w do</td></tr><tr><td colspan="3">cv w,k ← vw × α w,k ;</td></tr><tr><td>end</td><td/><td/></tr><tr><td colspan="3">tvw ← K k=1 cv wk ; / * is concatenation, × is scalar vector multiplication</td><td>* /</td></tr><tr><td>end</td><td/><td/></tr><tr><td colspan="3">/ * SIF reweighed document vector embedding</td><td>* /</td></tr><tr><td colspan="3">for each document dn in D do</td></tr><tr><td>v dn ← 1 |dn|</td><td>w∈dn</td><td>a a+p(w) tvw;</td></tr><tr><td>end</td><td/><td/></tr><tr><td>Form a matrix X</td><td/><td/></tr></table>

Experimental results (Pearsons r × 100) on textual similarity tasks. The highest score in each row is in bold. See the experiment settings section 6 under the textual similarity task for the description of the methods. Many results are collected from(Wieting et al., 2016a)  and(Wieting & Gimpel, 2017) (GRAN) except tfidf-GloVe and our new representation

Table content: <table><tr><td>Supervised</td><td/><td/><td/><td colspan="2">Supervised</td><td/><td/><td/><td colspan="3">UnSupervised</td><td colspan="3">Semi Supervised</td><td>P-SIF</td></tr><tr><td>or not</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Tasks</td><td>PP</td><td>PP</td><td colspan="4">DAN RNN iRNN LSTM</td><td>LSTM</td><td colspan="2">GRAN ST</td><td>avg</td><td>tfidf</td><td>avg</td><td>Glove</td><td>PSL</td><td>P-SIF</td></tr><tr><td/><td/><td>-</td><td/><td/><td/><td>(no)</td><td>(o.g.)</td><td/><td/><td>Glove</td><td>Glove</td><td>PSL</td><td>+WR</td><td>+WR</td><td>+PSL</td></tr><tr><td/><td/><td>proj</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>STS12</td><td>58.7</td><td>60.0</td><td>56.0</td><td>48.1</td><td>58.4</td><td>51.0</td><td>46.4</td><td>62.5</td><td>30.8</td><td>52.5</td><td>58.7</td><td>52.8</td><td>56.2</td><td>59.5</td><td>65.70</td></tr><tr><td>STS13</td><td>55.8</td><td>56.8</td><td>54.2</td><td>44.7</td><td>56.7</td><td>45.2</td><td>41.5</td><td>63.4</td><td>24.8</td><td>42.3</td><td>52.1</td><td>46.4</td><td>56.6</td><td>61.8</td><td>63.98</td></tr><tr><td>STS14</td><td>70.9</td><td>71.3</td><td>69.5</td><td>57.7</td><td>70.9</td><td>59.8</td><td>51.5</td><td>75.9</td><td>31.4</td><td>54.2</td><td>63.8</td><td>59.5</td><td>68.5</td><td>73.5</td><td>74.80</td></tr><tr><td>STS15</td><td>75.8</td><td>74.8</td><td>72.7</td><td>57.2</td><td>75.6</td><td>63.9</td><td>56.0</td><td>77.7</td><td>31.0</td><td>52.7</td><td>60.6</td><td>60.0</td><td>71.7</td><td>76.3</td><td>77.29</td></tr><tr><td>SICK14</td><td>71.6</td><td>71.6</td><td>70.7</td><td>61.2</td><td>71.2</td><td>63.9</td><td>59.0</td><td>72.9</td><td>49.8</td><td>65.9</td><td>69.4</td><td>66.4</td><td>72.2</td><td>72.9</td><td>73.41</td></tr><tr><td>Twitter15</td><td>52.9</td><td>52.8</td><td>53.7</td><td>45.1</td><td>52.9</td><td>47.6</td><td>36.1</td><td>50.2</td><td>24.7</td><td>30.3</td><td>33.8</td><td>36.3</td><td>48.0</td><td>49.0</td><td>54.86</td></tr></table>

Experimental results (Pearsons r × 100) on textual similarity tasks on STS 16. The highest score is in bold.

Table content: <table><tr><td>Tasks</td><td>Skip</td><td>LSTM</td><td>Tree</td><td colspan="3">Sent2Vec Doc2Vec Glove</td><td>Glove</td><td>PSL</td><td>PSL</td><td>Glove</td><td>PSL</td><td>P-SIF</td></tr><tr><td/><td>thoughts</td><td/><td>LSTM</td><td/><td/><td>Avg</td><td>tf-idf</td><td>Avg</td><td>tf-idf</td><td>+WR</td><td>+WR</td><td>+PSL</td></tr><tr><td>STS16</td><td>51.4</td><td>64.9</td><td>64.0</td><td>73.7</td><td>69.4</td><td>47.2</td><td>51.1</td><td>63.3</td><td>66.9</td><td>72.4</td><td>72.5</td><td>73.7</td></tr></table>

Performance on multi-label classification on Reuters. P-SIF represents our new embeddings. P-SIF(Doc2VecC) represents embeddings obtained using Doc2VecC-initialized wordvectors. Values in bold show the best performance.

Table content: <table><tr><td>Model</td><td>Prec@1</td><td>Prec</td><td>nDCG</td><td>Coverage</td><td>LRAPS</td><td>F1-Score</td></tr><tr><td/><td>nDCG@1</td><td>@5</td><td>@5</td><td>Error</td><td/><td/></tr><tr><td>P-SIF (Doc2VecC)</td><td>94.92</td><td>37.98</td><td>50.40</td><td>6.03</td><td>93.95</td><td>82.87</td></tr><tr><td>P-SIF</td><td>94.77</td><td>37.33</td><td>49.97</td><td>6.24</td><td>93.72</td><td>82.41</td></tr><tr><td>SCDV(Mekala et al., 2017)</td><td>94.20</td><td>36.98</td><td>49.55</td><td>6.48</td><td>93.30</td><td>81.75</td></tr><tr><td>Doc2VecC(Chen, 2017)</td><td>93.45</td><td>36.86</td><td>49.28</td><td>6.83</td><td>92.66</td><td>81.29</td></tr><tr><td>BoWV(Gupta et al., 2016)</td><td>92.90</td><td>36.14</td><td>48.55</td><td>8.16</td><td>91.46</td><td>79.16</td></tr><tr><td>TWE-1(Liu et al., 2015b)</td><td>90.91</td><td>35.49</td><td>47.54</td><td>8.16</td><td>91.46</td><td>79.16</td></tr><tr><td>SIF(Arora et al., 2017)</td><td>90.40</td><td>35.09</td><td>47.32</td><td>8.98</td><td>88.10</td><td>76.78</td></tr><tr><td>PV-DM(Le &amp; Mikolov, 2014)</td><td>87.54</td><td>33.24</td><td>44.21</td><td>13.15</td><td>86.21</td><td>70.24</td></tr><tr><td>PV-DBoW(Le &amp; Mikolov, 2014)</td><td>88.78</td><td>34.51</td><td>46.42</td><td>11.28</td><td>87.43</td><td>73.68</td></tr><tr><td>AvgVec(Singh &amp; Mukerjee, 2015)</td><td>89.09</td><td>34.73</td><td>46.48</td><td>9.67</td><td>87.28</td><td>71.91</td></tr><tr><td>tfidf AvgVec(Singh &amp; Mukerjee, 2015)</td><td>89.33</td><td>35.04</td><td>46.83</td><td>9.42</td><td>87.90</td><td>71.97</td></tr></table>

Performance on multi-class classification on 20NewsGroups. P-SIF represents our new embeddings. P-SIF (Doc2VecC) represents embeddings obtained from Doc2VecC-initialized wordvectors. Values in bold show the best performance.

Table content: <table><tr><td>Model</td><td>Acc</td><td>Prec</td><td>Rec</td><td>F-mes</td></tr><tr><td>P-SIF (Doc2VecC)</td><td>86.0</td><td>86.1</td><td>86.1</td><td>86.0</td></tr><tr><td>P-SIF</td><td>85.4</td><td>85.5</td><td>85.4</td><td>85.2</td></tr><tr><td>SCDV(Mekala et al., 2017)</td><td>84.6</td><td>84.6</td><td>84.5</td><td>84.6</td></tr><tr><td>Doc2VecC(Chen, 2017)</td><td>84.0</td><td>84.1</td><td>84.1</td><td>84.0</td></tr><tr><td>BoE(Jin et al., 2016)</td><td>83.1</td><td>83.1</td><td>83.1</td><td>83.1</td></tr><tr><td>NTSG-1(Liu et al., 2015a)</td><td>82.6</td><td>82.5</td><td>81.9</td><td>81.2</td></tr><tr><td>NTSG-2(Liu et al., 2015a)</td><td>82.5</td><td>83.7</td><td>82.8</td><td>82.4</td></tr><tr><td>SIF(Arora et al., 2017)</td><td>82.3</td><td>82.6</td><td>82.9</td><td>82.2</td></tr><tr><td>BoWV(Gupta et al., 2016)</td><td>81.6</td><td>81.1</td><td>81.1</td><td>80.9</td></tr><tr><td>NTSG-3(Liu et al., 2015a)</td><td>81.9</td><td>83.0</td><td>81.7</td><td>81.1</td></tr><tr><td>LTSG(Law et al., 2017)</td><td>82.8</td><td>82.4</td><td>81.8</td><td>81.8</td></tr><tr><td>WTM(Fu et al., 2016)</td><td>80.9</td><td>80.3</td><td>80.3</td><td>80.0</td></tr><tr><td>w2v-LDA(Nguyen et al., 2015)</td><td>77.7</td><td>77.4</td><td>77.2</td><td>76.9</td></tr><tr><td>TV+MeanWV(Li et al., 2016a)</td><td>72.2</td><td>71.8</td><td>71.5</td><td>71.6</td></tr><tr><td>MvTM(Li et al., 2016b)</td><td>72.2</td><td>71.8</td><td>71.5</td><td>71.6</td></tr><tr><td>TWE-1(Liu et al., 2015b)</td><td>81.5</td><td>81.2</td><td>80.6</td><td>80.6</td></tr><tr><td>Lda2Vec(Moody, 2016)</td><td>81.3</td><td>81.4</td><td>80.4</td><td>80.5</td></tr><tr><td>LDA</td><td/><td/><td/><td/></tr></table>

The STS tasks by year. Tasks with the same name in different years are different tasks

Table content: <table><tr><td>STS12</td><td>STS13</td><td>STS14</td><td>STS15</td><td>STS16</td></tr><tr><td>MSRpar</td><td colspan="2">headline deft forum</td><td>anwsers-forums</td><td>headlines</td></tr><tr><td>MSRvid</td><td>OnWN</td><td>deft news</td><td>answers-students</td><td>plagiarism</td></tr><tr><td>SMT-eur</td><td>FNWN</td><td>headline</td><td>belief</td><td>posteditng</td></tr><tr><td>OnWN</td><td>SMT</td><td>images</td><td>headline</td><td>answer-answer</td></tr><tr><td>SMT-news</td><td/><td>OnWN</td><td>images</td><td>question-question</td></tr><tr><td/><td/><td>tweet news</td><td/><td/></tr></table>

(GRAN) except tfidf-GloVe and our representation. Experimental results (Pearsons r × 100) on textual similarity tasks on STS 16. The highest score in each row is in boldface.

Table content: <table><tr><td>Supervised</td><td/><td/><td/><td colspan="2">Supervised</td><td/><td/><td/><td colspan="4">UnSupervised</td><td colspan="3">Semi Supervised</td><td>P-SIF</td></tr><tr><td>or not</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Tasks</td><td>PP</td><td>PP</td><td>DAN</td><td>RNN</td><td colspan="2">iRNN LSTM</td><td>LSTM</td><td colspan="2">GRAN ST</td><td>avg</td><td/><td>tfidf</td><td>avg</td><td>Glove</td><td>PSL</td><td>P-SIF</td></tr><tr><td/><td/><td>-</td><td/><td/><td/><td>(no)</td><td>(o.g.)</td><td/><td/><td colspan="2">Glove</td><td>Glove</td><td>PSL</td><td>+WR</td><td>+WR</td><td>+PSL</td></tr><tr><td/><td/><td>proj</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>MSRpar</td><td>42.6</td><td>43.7</td><td>40.3</td><td>18.6</td><td>43.4</td><td>16.1</td><td>9.3</td><td>47.7</td><td>16.8</td><td>47.7</td><td/><td>50.3</td><td>41.6</td><td>35.6</td><td>43.3</td><td>52.4</td></tr><tr><td>MSRvid</td><td>74.5</td><td>74.0</td><td>70.0</td><td>66.5</td><td>73.4</td><td>71.3</td><td>71.3</td><td>85.2</td><td>41.7</td><td>63.9</td><td/><td>77.9</td><td>60.0</td><td>83.8</td><td>84.1</td><td>85.6</td></tr><tr><td>SMT-eur</td><td>47.3</td><td>49.4</td><td>43.8</td><td>40.9</td><td>47.1</td><td>41.8</td><td>44.3</td><td>49.3</td><td>35.2</td><td>46.0</td><td/><td>54.7</td><td>42.4</td><td>49.9</td><td>44.8</td><td>58.7</td></tr><tr><td>OnWN</td><td>70.6</td><td>70.1</td><td>65.9</td><td>63.1</td><td>70.1</td><td>65.2</td><td>56.4</td><td>71.5</td><td>29.7</td><td>55.1</td><td/><td>64.7</td><td>63.0</td><td>66.2</td><td>71.8</td><td>72.2</td></tr><tr><td>SMT-news</td><td>58.4</td><td>62.8</td><td>60.0</td><td>51.3</td><td>58.1</td><td>60.8</td><td>51.0</td><td>58.7</td><td>30.8</td><td>49.6</td><td/><td>45.7</td><td>57.0</td><td>45.6</td><td>53.6</td><td>59.5</td></tr><tr><td>STS12</td><td>58.7</td><td>60.0</td><td>56.0</td><td>48.1</td><td>58.4</td><td>51.0</td><td>46.4</td><td>62.5</td><td>30.8</td><td>52.5</td><td/><td>58.7</td><td>52.8</td><td>56.2</td><td>59.5</td><td>65.7</td></tr><tr><td>headline</td><td>72.4</td><td>72.6</td><td>71.2</td><td>59.5</td><td>72.8</td><td>57.4</td><td>48.5</td><td>76.1</td><td>34.6</td><td>63.8</td><td/><td>69.2</td><td>68.8</td><td>69.2</td><td>74.1</td><td>75.7</td></tr><tr><td>OnWN</td><td>67.7</td><td>68.0</td><td>64.1</td><td>54.6</td><td>69.4</td><td>68.5</td><td>50.4</td><td>81.4</td><td>10.0</td><td>49.0</td><td/><td>72.9</td><td>48.0</td><td>82.8</td><td>82.0</td><td>84.4</td></tr><tr><td>FNWN</td><td>43.9</td><td>46.8</td><td>43.1</td><td>30.9</td><td>45.3</td><td>24.7</td><td>38.4</td><td>55.6</td><td>30.4</td><td>34.2</td><td/><td>36.6</td><td>37.9</td><td>39.4</td><td>52.4</td><td>54.8</td></tr><tr><td>SMT</td><td>39.2</td><td>39.8</td><td>38.3</td><td>33.8</td><td>39.4</td><td>30.1</td><td>28.8</td><td>40.3</td><td>24.3</td><td>22.3</td><td/><td>29.6</td><td>31.0</td><td>37.9</td><td>38.5</td><td>41.0</td></tr><tr><td>STS13</td><td>55.8</td><td>56.8</td><td>54.2</td><td>44.7</td><td>56.7</td><td>45.2</td><td>41.5</td><td>63.4</td><td>24.8</td><td>42.3</td><td/><td>52.1</td><td>46.4</td><td>56.6</td><td>61.8</td><td>64.0</td></tr><tr><td>deft forum</td><td>48.7</td><td>51.1</td><td>49.0</td><td>41.5</td><td>49.0</td><td>44.2</td><td>46.1</td><td>55.7</td><td>12.9</td><td>27.1</td><td/><td>37.5</td><td>37.2</td><td>41.2</td><td>51.4</td><td>53.2</td></tr><tr><td>deft news</td><td>73.1</td><td>72.2</td><td>71.7</td><td>53.7</td><td>72.4</td><td>52.8</td><td>39.1</td><td>77.1</td><td>23.5</td><td>68.0</td><td/><td>68.7</td><td>67.0</td><td>69.4</td><td>72.6</td><td>75.2</td></tr><tr><td>headline</td><td>69.7</td><td>70.8</td><td>69.2</td><td>57.5</td><td>70.2</td><td>57.5</td><td>50.9</td><td>72.8</td><td>37.8</td><td>59.5</td><td/><td>63.7</td><td>65.3</td><td>64.7</td><td>70.1</td><td>70.2</td></tr><tr><td>images</td><td>78.5</td><td>78.1</td><td>76.9</td><td>67.6</td><td>78.2</td><td>68.5</td><td>62.9</td><td>85.8</td><td>51.2</td><td>61.0</td><td/><td>72.5</td><td>62.0</td><td>82.6</td><td>84.8</td><td>84.8</td></tr><tr><td>OnWN</td><td>78.8</td><td>79.5</td><td>75.7</td><td>67.7</td><td>78.8</td><td>76.9</td><td>61.7</td><td>85.1</td><td>23.3</td><td>58.4</td><td/><td>75.2</td><td>61.1</td><td>82.8</td><td>84.5</td><td>88.1</td></tr><tr><td>tweet news</td><td>76.4</td><td>75.8</td><td>74.2</td><td>58.0</td><td>76.9</td><td>58.7</td><td>48.2</td><td>78.7</td><td>39.9</td><td>51.2</td><td/><td>65.1</td><td>64.7</td><td>70.1</td><td>77.5</td><td>77.5</td></tr><tr><td>STS14</td><td>70.9</td><td>71.3</td><td>69.5</td><td>57.7</td><td>70.9</td><td>59.8</td><td>51.5</td><td>75.8</td><td>31.4</td><td>54.2</td><td/><td>63.8</td><td>59.5</td><td>68.5</td><td>73.5</td><td>74.8</td></tr><tr><td>answers-forum</td><td>68.3</td><td>65.1</td><td>62.6</td><td>32.8</td><td>67.4</td><td>51.9</td><td>50.7</td><td>73.1</td><td>36.1</td><td>30.5</td><td/><td>45.6</td><td>38.8</td><td>63.9</td><td>70.1</td><td>70.7</td></tr><tr><td>answers-student</td><td>78.2</td><td>77.8</td><td>78.1</td><td>64.7</td><td>78.2</td><td>71.5</td><td>55.7</td><td>72.9</td><td>33.0</td><td>63.0</td><td/><td>63.9</td><td>69.2</td><td>70.4</td><td>75.9</td><td>79.6</td></tr><tr><td>belief</td><td>76.2</td><td>75.4</td><td>72.0</td><td>51.9</td><td>75.9</td><td>61.7</td><td>52.6</td><td>78</td><td>24.6</td><td>40.5</td><td/><td>49.5</td><td>53.2</td><td>71.8</td><td>75.3</td><td>75.3</td></tr><tr><td>headline</td><td>74.8</td><td>75.2</td><td>73.5</td><td>65.3</td><td>75.1</td><td>64.0</td><td>56.6</td><td>78.6</td><td>43.6</td><td>61.8</td><td/><td>70.9</td><td>69.0</td><td>70.7</td><td>75.9</td><td>76.8</td></tr><tr><td>images</td><td>81.4</td><td>80.3</td><td>77.5</td><td>71.4</td><td>81.1</td><td>70.4</td><td>64.2</td><td>85.8</td><td>17.7</td><td>67.5</td><td/><td>72.9</td><td>69.9</td><td>81.5</td><td>84.1</td><td>84.1</td></tr><tr><td>STS15</td><td>75.8</td><td>74.8</td><td>72.7</td><td>57.2</td><td>75.6</td><td>63.9</td><td>56.0</td><td>77.7</td><td>31.0</td><td>52.7</td><td/><td>60.6</td><td>60.0</td><td>71.7</td><td>76.3</td><td>77.3</td></tr><tr><td>SICK14</td><td>71.6</td><td>71.6</td><td>70.7</td><td>61.2</td><td>71.2</td><td>63.9</td><td>59.0</td><td>72.9</td><td>49.8</td><td>65.9</td><td/><td>69.4</td><td>66.4</td><td>72.2</td><td>72.9</td><td>73.4</td></tr><tr><td>Twitter15</td><td>52.9</td><td>52.8</td><td>53.7</td><td>45.1</td><td>52.9</td><td>47.6</td><td>36.1</td><td>50.2</td><td>24.7</td><td>30.3</td><td/><td>33.8</td><td>36.3</td><td>48.0</td><td>49.0</td><td>54.9</td></tr><tr><td>Tasks</td><td/><td>Skip</td><td colspan="2">LSTM Tree</td><td/><td colspan="3">Sent2Vec Doc2Vec Glove</td><td colspan="2">Glove</td><td/><td>PSL</td><td>PSL</td><td>Glove</td><td>PSL</td><td>P-SIF</td></tr><tr><td/><td/><td>thoughts</td><td/><td colspan="2">LSTM</td><td/><td/><td>Avg</td><td colspan="2">tf-idf</td><td/><td>Avg</td><td>tf-idf</td><td>+WR</td><td colspan="2">+WR</td><td>+PSL</td></tr><tr><td>headlines</td><td/><td>51.019</td><td>75.7</td><td>74.08</td><td/><td>75.06</td><td>69.16</td><td>49.66</td><td colspan="2">52.76</td><td colspan="2">70.86</td><td>72.24</td><td>72.86</td><td colspan="2">74.48</td><td>75.6</td></tr><tr><td>plagiarism</td><td/><td>66.708</td><td>71.73</td><td>67.62</td><td/><td>80.06</td><td>80.6</td><td>59.84</td><td colspan="2">61.48</td><td colspan="2">77.96</td><td>80.06</td><td>79.46</td><td colspan="2">79.74</td><td>81.6</td></tr><tr><td>post editing</td><td/><td>69.947</td><td>72.31</td><td>70.65</td><td/><td>82.85</td><td>82.85</td><td>59.89</td><td colspan="2">62.34</td><td colspan="2">80.41</td><td>81.45</td><td>82.03</td><td colspan="2">82.05</td><td>83.7</td></tr><tr><td>answer answer</td><td/><td>28.626</td><td>44.17</td><td>52.27</td><td/><td>57.73</td><td>41.12</td><td>19.8</td><td colspan="2">22.47</td><td/><td>38.5</td><td>41.56</td><td>58.15</td><td colspan="2">59.98</td><td>60.2</td></tr><tr><td>question question</td><td/><td>40.459</td><td>60.69</td><td>55.26</td><td/><td>73.03</td><td>73.03</td><td>46.84</td><td colspan="2">56.58</td><td colspan="2">48.69</td><td>59.1</td><td>69.36</td><td colspan="2">66.41</td><td>67.2</td></tr><tr><td>STS16</td><td/><td>51.4</td><td>64.9</td><td>64.0</td><td/><td>73.7</td><td>69.4</td><td>47.2</td><td>51.1</td><td/><td/><td>63.3</td><td>66.9</td><td>72.4</td><td>72.5</td><td>73.7</td></tr></table>

Results on similarity, entailment, and sentiment tasks. The row for similarity (SICK) shows Pearsons r × 100 and the last two rows show accuracy. The highest score in each row is in boldface. Results in Column 2 to 6 are collected from

Table content: <table/>

P-SIF embedding outperform all other embeddings on multiple classes.

Table content: <table><tr><td colspan="13">Table 10: Class performance on the 20newsgroup dataset. P-SIF represent our embedding. P-SIF</td></tr><tr><td colspan="13">(Doc2VecC) represent embedding obtained using Doc2VecC trained word-vectors with 20 clusters.</td></tr><tr><td/><td/><td>BoW</td><td/><td/><td>SCDV</td><td/><td/><td>P-SIF</td><td/><td colspan="3">P-SIF (Doc2VecC)</td></tr><tr><td>Class Name</td><td>Pre.</td><td>Rec.</td><td>F-mes</td><td>Pre.</td><td>Rec.</td><td>F-mes</td><td>Pre.</td><td>Rec.</td><td>F-mes</td><td>Pre.</td><td>Rec.</td><td>F-mes</td></tr><tr><td>alt.atheism</td><td>67.8</td><td>72.1</td><td>69.88</td><td>80.2</td><td>79.5</td><td>79.85</td><td>83.3</td><td>80.2</td><td>81.72</td><td>83</td><td>79.9</td><td>81.42</td></tr><tr><td>comp.graphics</td><td>67.1</td><td>73.5</td><td>70.15</td><td>75.3</td><td>77.4</td><td>76.34</td><td>76.6</td><td>78.1</td><td>77.34</td><td>76.8</td><td>79.2</td><td>77.98</td></tr><tr><td>comp.os.ms-windows.misc</td><td>77.1</td><td>66.5</td><td>71.41</td><td>78.6</td><td>77.2</td><td>77.89</td><td>76.3</td><td>77.7</td><td>76.99</td><td>77.2</td><td>78.2</td><td>77.7</td></tr><tr><td>comp.sys.ibm.pc.hardware</td><td>62.8</td><td>72.4</td><td>67.26</td><td>75.6</td><td>73.5</td><td>74.54</td><td>73.4</td><td>74.5</td><td>73.95</td><td>71.1</td><td>74.2</td><td>72.62</td></tr><tr><td>comp.sys.mac.hardware</td><td>77.4</td><td>78.2</td><td>77.8</td><td>83.4</td><td>85.5</td><td>84.44</td><td>87.1</td><td>84.4</td><td>85.73</td><td>87.5</td><td>87.5</td><td>87.5</td></tr><tr><td>comp.windows.x</td><td>83.2</td><td>73.2</td><td>77.88</td><td>87.6</td><td>78.6</td><td>82.86</td><td>89.3</td><td>78</td><td>83.27</td><td>88.8</td><td>78.5</td><td>83.33</td></tr><tr><td>misc.forsale</td><td>81.3</td><td>88.2</td><td>84.61</td><td>81.4</td><td>85.9</td><td>83.59</td><td>82.7</td><td>88</td><td>85.27</td><td>82.4</td><td>86.4</td><td>84.35</td></tr><tr><td>rec.autos</td><td>80.7</td><td>82.8</td><td>81.74</td><td>91.2</td><td>90.6</td><td>90.9</td><td>93</td><td>90.1</td><td>91.53</td><td>92.8</td><td>90.7</td><td>91.74</td></tr><tr><td>rec.motorcycles</td><td>92.3</td><td>87.9</td><td>90.05</td><td>95.4</td><td>95.7</td><td>95.55</td><td>93.6</td><td>95.5</td><td>94.54</td><td>97</td><td>96.5</td><td>96.75</td></tr><tr><td>rec.sport.baseball</td><td>89.8</td><td>89.2</td><td>89.5</td><td>93.2</td><td>94.7</td><td>93.94</td><td>93.3</td><td>95.2</td><td>94.24</td><td>95.2</td><td>95.7</td><td>95.45</td></tr><tr><td>rec.sport.hockey</td><td>93.3</td><td>93.7</td><td>93.5</td><td>96.3</td><td>99.2</td><td>97.73</td><td>95.6</td><td>98.5</td><td>97.03</td><td>96.8</td><td>98.8</td><td>97.79</td></tr><tr><td>sci.crypt</td><td>92.2</td><td>86.1</td><td>89.05</td><td>92.5</td><td>94.7</td><td>93.59</td><td>89.8</td><td>93.2</td><td>91.47</td><td>93.4</td><td>96.7</td><td>95.02</td></tr><tr><td>sci.electronics</td><td>70.9</td><td>73.3</td><td>72.08</td><td>74.6</td><td>74.9</td><td>74.75</td><td>79.6</td><td>78.6</td><td>79.1</td><td>78</td><td>79.3</td><td>78.64</td></tr><tr><td>sci.med</td><td>79.3</td><td>81.3</td><td>80.29</td><td>91.3</td><td>88.4</td><td>89.83</td><td>91.9</td><td>88.6</td><td>90.22</td><td>92.7</td><td>89.9</td><td>91.28</td></tr><tr><td>sci.space</td><td>90.2</td><td>88.3</td><td>89.24</td><td>88.5</td><td>93.8</td><td>91.07</td><td>89.4</td><td>94</td><td>91.64</td><td>90.7</td><td>94.4</td><td>92.51</td></tr><tr><td>soc.religion.christian</td><td>77.3</td><td>87.9</td><td>82.26</td><td>83.3</td><td>92.3</td><td>87.57</td><td>84</td><td>94.3</td><td>88.85</td><td>86</td><td>92.5</td><td>89.13</td></tr><tr><td>talk.politics.guns</td><td>71.7</td><td>85.7</td><td>78.08</td><td>72.7</td><td>90.6</td><td>80.67</td><td>73.1</td><td>91.2</td><td>81.15</td><td>77.3</td><td>89.8</td><td>83.08</td></tr><tr><td>talk.politics.mideast</td><td>91.7</td><td>76.9</td><td>83.65</td><td>96.2</td><td>95.4</td><td>95.8</td><td>97</td><td>94.5</td><td>95.73</td><td>97.5</td><td>94.2</td><td>95.82</td></tr><tr><td>talk.politics.misc</td><td>71.7</td><td>56.5</td><td>63.2</td><td>80.9</td><td>59.7</td><td>68.7</td><td>81</td><td>59</td><td>68.27</td><td>82</td><td>62</td><td>70.61</td></tr><tr><td>talk.religion.misc</td><td>63.2</td><td>55.4</td><td>59.04</td><td>73.5</td><td>57.2</td><td>64.33</td><td>72.2</td><td>59</td><td>64.94</td><td>67.4</td><td>62.4</td><td>64.8</td></tr><tr><td colspan="3">D QUALITATIVE RESULTS</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

represent successful example pair from STS 2012 MSRvid dataset where P-SIF give similarity score closer to ground truth than SIF. Table12represent failed example pair from STS 2012 MSRvid dataset where SIF give similarity score closer to ground truth than P-SIF. We now introduce the headline notation use in the Table11 and 12.

Table content: <table/>

STS 2012 MSRVid example where P-SIF score were closer to the ground truth, whereas SIF score were more further from the ground truth

Table content: <table><tr><td>sentence1</td><td>sentence2</td><td>GT</td><td>NGT</td><td>SIFsc</td><td>P-SIFsc</td><td>SIFerr</td><td>P-SIFerr</td><td>Differr</td><td>Relerr</td></tr><tr><td>People are playing baseball .</td><td>The cricket player hit the ball .</td><td>0.5</td><td>0.1</td><td>0.2928</td><td>0.0973</td><td>0.1928</td><td>0.0027</td><td>0.1901</td><td>0.986</td></tr><tr><td>A woman is carrying a boy .</td><td>A woman is carrying her baby .</td><td>2.333</td><td>0.4666</td><td>0.5743</td><td>0.4683</td><td>0.1077</td><td>0.0017</td><td>0.106</td><td>0.9843</td></tr><tr><td>A man is riding a motorcycle .</td><td>A woman is riding a horse .</td><td>0.75</td><td>0.15</td><td>0.5655</td><td>0.157</td><td>0.4155</td><td>0.007</td><td>0.4085</td><td>0.9833</td></tr><tr><td>A woman slices a lemon .</td><td>A man is talking into a microphone .</td><td>0</td><td>0</td><td>-0.1101</td><td>-0.0027</td><td>0.1101</td><td>0.0027</td><td>0.1074</td><td>0.9754</td></tr><tr><td>A man is hugging someone .</td><td>A man is taking a picture .</td><td>0.4</td><td>0.08</td><td>0.2021</td><td>0.0767</td><td>0.1221</td><td>0.0033</td><td>0.1188</td><td>0.9731</td></tr><tr><td>A woman is dancing .</td><td>A woman plays the clarinet .</td><td>0.8</td><td>0.16</td><td>0.3539</td><td>0.1653</td><td>0.1939</td><td>0.0053</td><td>0.1886</td><td>0.9727</td></tr><tr><td>A train is moving .</td><td>A man is doing yoga .</td><td>0</td><td>0</td><td>0.1674</td><td>-0.0051</td><td>0.1674</td><td>0.0051</td><td>0.1623</td><td>0.9695</td></tr><tr><td>Runners race around a track .</td><td>Runners compete in a race .</td><td>3.2</td><td>0.64</td><td>0.7653</td><td>0.6438</td><td>0.1253</td><td>0.0038</td><td>0.1214</td><td>0.9694</td></tr><tr><td>A man is driving a car .</td><td>A man is riding a horse .</td><td>1.2</td><td>0.24</td><td>0.3584</td><td>0.2443</td><td>0.1184</td><td>0.0043</td><td>0.114</td><td>0.9636</td></tr><tr><td>A man is playing a guitar .</td><td>A woman is riding a horse .</td><td>0.5</td><td>0.1</td><td>-0.0208</td><td>0.0955</td><td>0.1208</td><td>0.0045</td><td>0.1163</td><td>0.9629</td></tr><tr><td>A man is riding on a horse .</td><td>A girl is riding a horse .</td><td>2.6</td><td>0.52</td><td>0.6933</td><td>0.5082</td><td>0.1733</td><td>0.0118</td><td>0.1615</td><td>0.9319</td></tr><tr><td>A woman is deboning a fish .</td><td>A man catches a fish .</td><td>1.25</td><td>0.25</td><td>0.4538</td><td>0.2336</td><td>0.2038</td><td>0.0164</td><td>0.1875</td><td>0.9196</td></tr><tr><td>A man is playing a guitar .</td><td>A man is eating pasta .</td><td>0.533</td><td>0.1066</td><td>-0.0158</td><td>0.0962</td><td>0.1224</td><td>0.0104</td><td>0.112</td><td>0.915</td></tr><tr><td>A woman is dancing .</td><td>A man is eating .</td><td>0.143</td><td>0.0286</td><td>-0.1001</td><td>0.0412</td><td>0.1287</td><td>0.0126</td><td>0.1161</td><td>0.9023</td></tr><tr><td>The ballerina is dancing .</td><td>A man is dancing .</td><td>1.75</td><td>0.35</td><td>0.512</td><td>0.3317</td><td>0.162</td><td>0.0183</td><td>0.1437</td><td>0.8871</td></tr><tr><td>A woman plays the guitar .</td><td>A man sings and plays the guitar .</td><td>1.75</td><td>0.35</td><td>0.5036</td><td>0.3683</td><td>0.1536</td><td>0.0183</td><td>0.1353</td><td>0.8807</td></tr><tr><td>A girl is styling her hair .</td><td>A girl is brushing her hair .</td><td>2.5</td><td>0.5</td><td>0.7192</td><td>0.5303</td><td>0.2192</td><td>0.0303</td><td>0.1889</td><td>0.8618</td></tr><tr><td>A guy is playing hackysack</td><td>A man is playing a key-board .</td><td>1</td><td>0.2</td><td>0.3718</td><td>0.2268</td><td>0.1718</td><td>0.0268</td><td>0.145</td><td>0.8441</td></tr><tr><td>A man is riding a bicycle .</td><td>A monkey is riding a bike .</td><td>2</td><td>0.4</td><td>0.6891</td><td>0.4614</td><td>0.2891</td><td>0.0614</td><td>0.2277</td><td>0.7876</td></tr><tr><td>A woman is swimming underwater .</td><td>A man is slicing some carrots .</td><td>0</td><td>0</td><td>-0.2158</td><td>-0.0562</td><td>0.2158</td><td>0.0562</td><td>0.1596</td><td>0.7397</td></tr><tr><td>A plane is landing .</td><td>A animated airplane is landing .</td><td>2.8</td><td>0.56</td><td>0.801</td><td>0.6338</td><td>0.241</td><td>0.0738</td><td>0.1672</td><td>0.6937</td></tr><tr><td>The missile exploded .</td><td>A rocket exploded .</td><td>3.2</td><td>0.64</td><td>0.8157</td><td>0.6961</td><td>0.1757</td><td>0.0561</td><td>0.1196</td><td>0.6806</td></tr><tr><td>A woman is peeling a potato .</td><td>A woman is peeling an apple .</td><td>2</td><td>0.4</td><td>0.6938</td><td>0.5482</td><td>0.2938</td><td>0.1482</td><td>0.1456</td><td>0.4956</td></tr><tr><td>A woman is writing .</td><td>A woman is swimming .</td><td>0.5</td><td>0.1</td><td>0.3595</td><td>0.2334</td><td>0.2595</td><td>0.1334</td><td>0.1261</td><td>0.4859</td></tr><tr><td>A man is riding a bike .</td><td>A man is riding on a horse .</td><td>2</td><td>0.4</td><td>0.6781</td><td>0.564</td><td>0.2781</td><td>0.164</td><td>0.1142</td><td>0.4105</td></tr><tr><td>A panda is climbing .</td><td>A man is climbing a rope .</td><td>1.6</td><td>0.32</td><td>0.4274</td><td>0.3131</td><td>0.1074</td><td>0.0069</td><td>0.1005</td><td>0.9361</td></tr><tr><td>A man is shooting a gun .</td><td>A man is spitting .</td><td>0</td><td>0</td><td>0.2348</td><td>0.1305</td><td>0.2348</td><td>0.1305</td><td>0.1043</td><td>0.444</td></tr></table>

STS 2012 MSRVid example where P-SIF score were far away from ground truth, whereas SIF score were closer to actual ground truth

Table content: <table><tr><td>sentence1</td><td>sentence2</td><td>GT</td><td>NGT</td><td>SIFsc</td><td>P-SIFsc</td><td>SIFerr</td><td>P-SIFerr</td><td>Differr</td><td>Relerr</td></tr><tr><td>takes off his sunglasses .</td><td>A boy is screaming .</td><td>0.5</td><td>0.1</td><td>0.1971</td><td>0.3944</td><td>0.0971</td><td>0.2944</td><td>0.1973</td><td>0.6703</td></tr><tr><td>The rhino grazed on the grass .</td><td>A rhino is grazing in a field .</td><td>4</td><td>0.8</td><td>0.7275</td><td>0.538</td><td>0.0725</td><td>0.262</td><td>0.1895</td><td>0.7234</td></tr><tr><td>An animal is biting a persons finger .</td><td>A slow loris is biting a persons finger .</td><td>3</td><td>0.6</td><td>0.6018</td><td>0.7702</td><td>0.0018</td><td>0.1702</td><td>0.1684</td><td>0.9892</td></tr><tr><td>Animals are playing in water .</td><td>Two men are playing ping pong .</td><td>0</td><td>0</td><td>0.0706</td><td>0.2238</td><td>0.0706</td><td>0.2238</td><td>0.1532</td><td>0.6846</td></tr><tr><td>Someone is feeding a animal .</td><td>Someone is playing a piano .</td><td>0</td><td>0</td><td>-0.0037</td><td>0.1546</td><td>0.0037</td><td>0.1546</td><td>0.1509</td><td>0.976</td></tr><tr><td>The lady sliced a tomatoe .</td><td>Someone is cutting a tomato .</td><td>4</td><td>0.8</td><td>0.693</td><td>0.5591</td><td>0.107</td><td>0.2409</td><td>0.1339</td><td>0.5559</td></tr><tr><td>The lady peeled the potatoe .</td><td>A woman is peeling a potato .</td><td>4.75</td><td>0.95</td><td>0.7167</td><td>0.5925</td><td>0.2333</td><td>0.3575</td><td>0.1242</td><td>0.3474</td></tr><tr><td>A man is slicing something .</td><td>A man is slicing a bun .</td><td>3</td><td>0.6</td><td>0.5976</td><td>0.4814</td><td>0.0024</td><td>0.1186</td><td>0.1162</td><td>0.9802</td></tr><tr><td>A boy is crawling into a dog house .</td><td>A boy is playing a wooden flute .</td><td>0.75</td><td>0.15</td><td>0.1481</td><td>0.2674</td><td>0.0019</td><td>0.1174</td><td>0.1155</td><td>0.9839</td></tr><tr><td>A man and woman are talking .</td><td>A man and woman is eating .</td><td>1.6</td><td>0.32</td><td>0.3574</td><td>0.4711</td><td>0.0374</td><td>0.1511</td><td>0.1137</td><td>0.7527</td></tr><tr><td>A man is cutting a potato .</td><td>A woman plays an electric guitar .</td><td>0.083</td><td>0.0166</td><td>-0.1007</td><td>-0.2128</td><td>0.1173</td><td>0.2294</td><td>0.112</td><td>0.4884</td></tr><tr><td>A person is cutting a meat .</td><td>A person riding a mechanical bull</td><td>0</td><td>0</td><td>0.0152</td><td>0.1242</td><td>0.0152</td><td>0.1242</td><td>0.1091</td><td>0.8778</td></tr><tr><td>A woman is playing the flute .</td><td>A man is playing the guitar .</td><td>1</td><td>0.2</td><td>0.1942</td><td>0.0876</td><td>0.0058</td><td>0.1124</td><td>0.1065</td><td>0.948</td></tr></table>

• d A , d B represent two document containing n and m words respectively. w A 1 , w A 2 . . . w A n represent words of d A and w B 1 , w B 2 . . . w B n represents word of d B .

Table content: <table/>

Comparison of our embedding P-SIF (SGNS) with recently proposed word mover distance and word mover embedding. All dataset and baseline are taken from(Wu et al., 2018)

Table content: <table><tr><td>.</td></tr></table>

Comparison of our P-SIF embedding with recently proposed embedding techniques on various STS tasks. Many baselines taken from(Conneau & Kiela, 2018),Perone et al. (2018).(Wu et al., 2018) andEthayarajh (2018)

Table content: <table><tr><td>Task</td><td colspan="3">ELMO ELMO p-mean Fast Skip Infer</td><td>Char</td><td>WME PSIF u-SIF</td></tr><tr><td/><td>orig</td><td>orig</td><td colspan="2">Text Thou Sent pharse +PSL +PSL +PSL</td></tr><tr><td/><td>+all</td><td>+top</td><td>ghts</td></tr><tr><td>STS 12</td><td/><td/><td/></tr></table>

Performance on 20NewsGroup classification using ELMO embedding

Table content: <table><tr><td>Average</td><td>tfidf</td><td colspan="6">layers dimension Accuracy Precision Recall f1-score</td></tr><tr><td>word</td><td>FALSE</td><td>1,2</td><td>2048</td><td>74.07</td><td>74.02</td><td>74.07</td><td>73.94</td></tr><tr><td>word</td><td colspan="2">FALSE 1,2,3</td><td>3072</td><td>73.21</td><td>73.25</td><td>73.21</td><td>73.14</td></tr><tr><td>word</td><td>FALSE</td><td>2,3</td><td>2048</td><td>71.14</td><td>71.03</td><td>71.14</td><td>70.99</td></tr><tr><td>word</td><td>TRUE</td><td>1,2,3</td><td>3072</td><td>70.5</td><td>70.81</td><td>70.5</td><td>70.6</td></tr><tr><td>word</td><td>FALSE</td><td>2</td><td>1024</td><td>70.76</td><td>70.48</td><td>70.76</td><td>70.48</td></tr><tr><td>word</td><td>FALSE</td><td>3</td><td>1024</td><td>70.01</td><td>69.83</td><td>70.01</td><td>69.79</td></tr><tr><td>word</td><td>FALSE</td><td>1</td><td>1024</td><td>69.16</td><td>68.88</td><td>69.16</td><td>68.85</td></tr><tr><td>word</td><td>TRUE</td><td>1,2</td><td>2048</td><td>67.79</td><td>67.99</td><td>67.79</td><td>67.83</td></tr><tr><td>word</td><td>TRUE</td><td>2,3</td><td>2048</td><td>67.25</td><td>68.01</td><td>67.25</td><td>67.48</td></tr><tr><td>sent</td><td colspan="2">FALSE 1,2,3</td><td>3072</td><td>66.9</td><td>67.01</td><td>66.9</td><td>66.9</td></tr><tr><td>sent</td><td>FALSE</td><td>1</td><td>3072</td><td>66.5</td><td>66.04</td><td>66.3</td><td>66.2</td></tr><tr><td>sent</td><td>FALSE</td><td>1,2</td><td>3072</td><td>66.3</td><td>66.04</td><td>66.6</td><td>66.4</td></tr><tr><td>word</td><td>TRUE</td><td>2</td><td>1024</td><td>64.79</td><td>64.92</td><td>64.79</td><td>64.77</td></tr><tr><td>word</td><td>TRUE</td><td>3</td><td>1024</td><td>64.45</td><td>64.99</td><td>64.45</td><td>64.57</td></tr><tr><td>word</td><td>TRUE</td><td>1</td><td>1024</td><td>59.19</td><td>59.32</td><td>59.19</td><td>59.17</td></tr><tr><td>P-SIF</td><td>-</td><td>-</td><td>-</td><td>86.0</td><td>86.1</td><td>86.1</td><td>86.0</td></tr><tr><td>(Doc2VecC)</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>P-SIF</td><td>-</td><td>-</td><td>-</td><td>85.4</td><td>85.5</td><td>85.4</td><td>85.2</td></tr></table>

Performance on Reuters classification using p-means embedding

Table content: <table><tr><td>Power</td><td colspan="2">Znorms Dim</td><td>Prec@1</td><td colspan="4">Prec@5 nDCG@5 Coverage LRAPS</td><td>F1</td></tr><tr><td/><td/><td/><td>nDCG@1</td><td/><td/><td>Error</td><td/><td>-meas</td></tr><tr><td>1,2</td><td>FALSE</td><td>400</td><td>93.07</td><td>36.69</td><td>48.96</td><td>7.66</td><td>91.73</td><td>76.76</td></tr><tr><td>1,2</td><td>TRUE</td><td>400</td><td>92.49</td><td>36.23</td><td>48.45</td><td>13.93</td><td>90.58</td><td>79.13</td></tr><tr><td>1,2,-1,+1</td><td>FALSE</td><td>800</td><td>91.9</td><td>35.79</td><td>47.91</td><td>9.19</td><td>90.04</td><td>75.93</td></tr><tr><td>P-SIF</td><td>-</td><td>-</td><td>94.92</td><td>37.98</td><td>50.40</td><td>6.03</td><td>93.95</td><td>82.87</td></tr><tr><td>(Doc2VecC)</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>P-SIF</td><td>-</td><td>-</td><td>94.77</td><td>37.33</td><td>49.97</td><td>6.24</td><td>93.72</td><td>82.41</td></tr></table>

Peer review (iclr2019):
 Pros: The paper shows that we could have a better document/sentence embedding by partitioning the word embedding space based on a topic model and summing the embedding within each partition. The writing and presentation of the paper are clear. The method is simple, intuitive, and the experiments show that this type of method seems to achieve state-of-the-art results on predicting semantic similarity between sentences, especially for longer sentences. Cons: The main concern is the novelty of this work. The method is very similar to SCDV (Mekala et al., 2017). The high-level flow figure in appendix H is nearly identical as the Figure 1 and 2 in Mekala et al., 2017. The main difference seems to be that this paper advocates K-SVD (extensively studies in Arora et al. 2016) as their topic model and SCDV (Mekala et al., 2017) uses GMM. However, in the semantic similarity experiments (STS12-16 and Twitter15), the results actually use GMM. So I suppose the results tell us that we can achieve state-of-the-art performances if you directly combine tricks in SIF (Arora et al., 2017) and tricks in SCDV (Mekala et al., 2017). In the document classification experiment, the improvement looks small and the baselines are not strong enough. The proposed method should be compared with other strong unsupervised baselines such as ELMo [1] and p-mean [2]. Overall: The direction this paper explores is promising but the contributions in this paper seem to be incremental. I suggest the authors to try either of the following extensions to strengthen the future version of this work. 1. In addition to documentation classification, show that the embedding is better than the more recent proposed strong baselines like ELMo in various downstream tasks. 2. Derive some theories. One possible direction is that I guess the measuring the document similarity based on proposed embedding could be viewed as an approximation of Wasserstein similarity between the all the words in both documents. The matching step in Wasserstein is similar to the pooling step in your topic model. You might be able to say something about how good this approximation is. Some theoretical work about doing the nearest neighbor search based on vector quantization might be helpful in this direction. Minor questions: 1. I think another common approach in sparse coding is just to apply L1 penalty to encourage sparsity. Does this K-SVD optimization better than this L1 penalty approach? 2. How does the value k in K-SVD affect the performances? 3. In Aorora et al. 2016b, they constrain alpha to be non-negative. Did you do the same thing here? 4. How important this topic modeling is? If you just randomly group words and sum the embedding in the group, is that helpful? 5. In Figure 2, I would also like to see another curve of performance gain on the sentences with different lengths using K-SVD rather than GMM. Minor writing suggestions: 1. In the 4th paragraph of section 3, "shown in equation equation 2", and bit-wise should be element-wise 2. In the 4th paragraph of section 4, I think the citation after alternating minimization should be Arora et al. 2016b and Aharon et al. 2006 rather than Arora et al., 2016a 3. In the 2nd paragraph of section 6.1, (Jeffrey Pennington, 2014) should be (Pennington et al., 2014). In addition, the author order in the corresponding Glove citation in the reference section is incorrect. The correct order should be Jeffrey Pennington, Richard Socher, Christopher D. Manning. 4. In the 3rd paragraph of section 6.1, "Furthermore, Sentence" 5. In the 6th paragraph of section 6.1, I thought skip-thoughts and Sent2Vec are unsupervised methods. 6. In Table 2 and 3, it would be easier to read if the table is transposed and use the longer name for each method (e.g., use skip-thought rather than ST) 7. In Table 2,3,4,5, it would be better to show the dimensions of embedding for each method 8. Table 10 should also provide F1 9. Which version of GMM is used in STS experiment? The one using full or diagonal covariance matrix? [1] Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. NAACL [2] Rücklé, A., Eger, S., Peyrard, M., & Gurevych, I. (2018). Concatenated p-mean Word Embeddings as Universal Cross-Lingual Sentence Representations. arXiv preprint arXiv:1803.01400.