Title: ZERO-TRAINING SENTENCE EMBEDDING VIA OR-THOGONAL BASIS

Abstract: We propose a simple and robust non-parameterized approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is the word's novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace. Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representations. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.

1. INTRODUCTION:

The concept of word embeddings has been prevalent in NLP community in recent years, as they can characterize semantic similarity between any pair of words, achieving promising results in a large number of NLP tasks (Mikolov et al., 2013; Pennington et al., 2014; Salle et al., 2016) . However, due to the hierarchical nature of human language, it is not sufficient to comprehend text solely based on isolated understanding of each word. This has prompted a recent rise in search for semantically robust embeddings for longer pieces of text, such as sentences and paragraphs.

Based on learning paradigms, the existing approaches to sentence embeddings can be categorized into two categories: i) parameterized methods and ii) non-parameterized methods.

Parameterized sentence embeddings. These models are parameterized and require training to optimize their parameters. SkipThought (Kiros et al., 2015) is an encoder-decoder model that predicts adjacent sentences. Pagliardini et al. (2018) proposes an unsupervised model, Sent2Vec, to learn an n-gram feature in a sentence to predict the center word from the surrounding context. Quick thoughts (QT) (Logeswaran & Lee, 2018) replaces the encoder with a classifier to predict context sentences from candidate sequences. Khodak et al. (2018) proposes à la carte to learn a linear mapping to reconstruct the center word from its context. Conneau et al. (2017) generates the sentence encoder InferSent using Natural Language Inference (NLI) dataset. Universal Sentence Encoder (Yang et al., 2018; Cer et al., 2018) utilizes the transformer (Vaswani et al., 2017) for sentence embeddings. The model is first trained on large scale of unsupervised data from Wikipedia and forums, and then trained on the Stanford Natural Language Inference (SNLI) dataset. Wieting & Gimpel (2017) propose the gated recurrent averaging network (GRAN), which is trained on Paraphrase Database (PPDB) and English Wikipedia. Subramanian et al. (2018) leverages a multi-task learning framework to generate sentence embeddings. Wieting et al. (2015a) learns the paraphrastic sentence representations as the simple average of updated word embeddings.

Non-parameterized sentence embedding. Recent work (Arora et al., 2017) shows that, surprisingly, a weighted sum or transformation of word representations can outperform many sophisticated neural network structures in sentence embedding tasks. These methods are parameter-free and require no further training upon pre-trained word vectors. Arora et al. (2017) constructs a sentence embedding called SIF as a sum of pre-trained word embeddings, weighted by reverse document frequency. Rücklé et al. (2018) concatenates different power mean word embeddings as a sentence vector in p-mean. As these methods do not have a parameterized model, they can be easily adapted to novel text domains with both fast inference speed and high-quality sentence embeddings. In view of this trend, our work aims to further advance the frontier of this group and make its new state-of-the-art.

In this paper, we propose a novel sentence embedding algorithm, Geometric Embedding (GEM), based entirely on the geometric structure of word embedding space. Given a d-dim word embedding matrix A ∈ R d×n for a sentence with n words, any linear combination of the sentence's word embeddings lies in the subspace spanned by the n word vectors. We analyze the geometric structure of this subspace in R d . When we consider the words in a sentence one-by-one in order, each word may bring in a novel orthogonal basis to the existing subspace. This new basis can be considered as the new semantic meaning brought in by this word, while the length of projection in this direction can indicate the intensity of this new meaning. It follows that a word with a strong intensity should have a larger influence in the sentence's meaning. Thus, these intensities can be converted into weights to linearly combine all word embeddings to obtain the sentence embedding. In this paper, we theoretically frame the above approach in a QR factorization of the word embedding matrix A. Furthermore, since the meaning and importance of a word largely depends on its close neighborhood, we propose the sliding-window QR factorization method to capture the context of a word and characterize its significance within the context.

In the last step, we adapt a similar approach as Arora et al. (2017) to remove top principal vectors before generating the final sentence embedding. This step is to ensure commonly shared background components, e.g. stop words, do not bias sentence similarity comparison. As we build a new orthogonal basis for each sentence, we propose to have disparate background components for each sentence. This motivates us to put forward a sentence-specific principal vector removal method, leading to better empirical results.

We evaluate our algorithm on 11 NLP tasks. In all of these tasks, our algorithm outperforms all non-parameterized methods and many parameterized approaches. For example, compared to SIF (Arora et al., 2017) , the performance is boosted by 5.5% on STS benchmark dataset, and by 2.5% on SST dataset. Plus, the running time of our model compares favorably with existing models.

The rest of this paper is organized as following. In Section 2, we describe our sentence embedding algorithm GEM. We evaluate our model on various tasks in Section 3 and Section 4. Finally, we summarize our work in Section 5.

2. APPROACH:

2.1 QUANTIFY NEW SEMANTIC MEANING Let us consider the idea of word embeddings (Mikolov et al., 2013) , where a word w i is projected as a vector v wi ∈ R d . Any sequence of words can be viewed as a subspace in R d spanned by its word vectors. Before the appearance of the ith word, S is a subspace in R d spanned by {v w1 , v w2 , ..., v wi-1 }. Its orthonormal basis is {q 1 , q 2 , ..., q i-1 }. The embedding v wi of the ith word w i can be decomposed into

EQUATION

where i-1 j=1 r j q j is the part in v wi that resides in subspace S, and q i is orthogonal to S and is to be added to S. The above algorithm is also known as Gram-Schmidt Process. In the case of rank deficiency, i.e., v wi is already a linear combination of {q 1 , q 2 , ...q i-1 }, q i is a zero vector and r i = 0. In matrix form, this process is also known as QR factorization, defined as follows.

QR factorization. Define an embedding matrix of n words as A = [A :,1 , A :,2 , ..., A :,n ] ∈ R d×n , where A :,i is the embedding of the ith word w i in a word sequence (w 1 , . . . , w i , . . . , w n ). A ∈ R d×n can be factorized into A = QR, where the non-zero columns in Q ∈ R d×n are the orthonormal basis, and R ∈ R n×n is an upper triangular matrix.

The process above computes the novel semantic meaning of a word w.r.t all preceding words. As the meaning of a word influences and is influenced by its close neighbors, we now calculate the novel orthogonal basis vector q i of each word w i in its neighborhood, rather than only w.r.t the preceding words.

Definition 1 (Contextual Window Matrix) Given a word w i , and its m-neighborhood window inside the sentence (w i-m , . . . , w i-1 , w i , w i+1 , . . . , w i+m ) , define the contextual window matrix of word w i as:

S i = [v wi-m , . . . , v wi-1 , v wi+1 , . . . , v wi+m , v wi ] ∈ R d×(2m+1)

(2)

Here we shuffle v wi to the end of S i to compute its novel semantic information compared with its context. Now the QR factorization of S i is

S i = Q i R i (3)

Note that q i is the last column of Q i , which is also the new orthogonal basis vector to this contextual window matrix.

Next, in order to generate the embedding for a sentence, we will assign a weight to each of its words. This weight should characterize how much new and important information a word brings to the sentence. The previous process yields the orthogonal basis vector q i . We propose that q i represents the novel semantic meaning brought by word w i . We will now discuss how to quantify i) the novelty of q i to other meanings in w i , ii) the significance of q i to its context, and iii) the corpus-wise uniqueness of q i w.r.t the whole corpus.

2.2. NOVELTY:

We propose that a word w i is more important to a sentence if its novel orthogonal basis vector q i is a large component in v wi . This can be quantified as a novelty score:

EQUATION

)

where r is the last column of R i , and r -1 is the last element of r.

Connection to least square. From QR factorization theory, the novel orthogonal basis q i is also the normalized residual in the least square problem min Cxv wi 2 , i.e. q T i v wi = r -1 = min Cxv wi 2 , where C = S i :,1:2m . And q T i v wi = r -1 is the minimum distance from word vector v wi to the hyper-plane spanned by w i 's context words w i-m , . . . , w i-1 , w i+1 , . . . , w i+m .

It follows that α n is the exponential of the normalized distance between v wi and the subspace spanned by its context.

2.3. SIGNIFICANCE:

The significance of a word is related to how semantically aligned it is to the meaning of its context. To identify principal directions, i.e. meanings, in the contextual window matrix S i , we employ Singular Value Decomposition.

Singular Value Decomposition. Given a matrix:

A ∈ R d×n , there exists U ∈ R d×n with orthog- onal columns, diagonal matrix Σ = diag(σ 1 , ..., σ n ), σ 1 ≥ σ 2 ≥ ... ≥ σ n ≥ 0, and orthogonal matrix V ∈ R n×n , such that A = U ΣV T .

The columns of U , {U :,j } n j=1 , are an orthonormal basis of A's columns subspace and we propose that they represent a set of semantic meanings from the context. Their corresponding singular values {σ j } n j=1 , denoted by σ(A), represent the importance associated with {U :,j } n j=1 . The SVD of w i 's contextual window matrix is

S i = U i Σ i V iT ∈ R d×(2m+1) . It follows that q T i U i is the coordinate of q i in the basis of {U i :,j } 2m+1 j=1 .

Intuitively, a word is more important if its novel semantic meaning has a better alignment with more principal meanings in its contextual window. This can be quantified as σ(S i ) (q T i U i ) 2 , where denotes element-wise product. Therefore, we define the significance of w i in its context to be:

α s = σ(S i ) (q T i U i ) 2 2m + 1 (5)

It turns out α s can be rewritten as

α s = q T i U i Σ i 2 2m + 1 = q T i U i Σ i V i 2 2m + 1 = q T i S i 2 2m + 1 = q T i v wi 2m + 1 = r -1 2m + 1 (6)

We use the fact that V i is an orthogonal matrix and q i is orthogonal to all but the last column of S i , v wi . Therefore, α s is essentially the distance between w i and the context hyper-plane, normalized by the context size.

2.4. CORPUS-WISE UNIQUENESS:

Similar to the idea of inverse document frequency (IDF) (Sparck Jones, 1972) , a word that is commonly present in the corpus is likely to be a stop word, thus its corpus-wise uniqueness is small. In our solution, we compute the principal directions of the corpus and then measure their alignment with the novel orthogonal basis vector q i . If there is a high alignment, w i will be assigned a relatively low corpus-wise uniqueness score, and vice versa.

2.4.1. COMPUTE PRINCIPAL DIRECTIONS OF CORPUS:

As proposed in Arora et al. (2017) , given a corpus containing a set of sentences, each sentence embedding is first computed as a linear combination of its word embeddings, thus generating a sentence embedding matrix X = [c 1 , c 2 , . . . , c N ] ∈ R d×N for a corpus S with N sentences. Then principal vectors of X are computed.

In comparison, we do not form the sentence embedding matrix X after we finalize the sentence embedding. Instead, we obtain an intermediate coarse-grained sentence embedding matrix X c = [g 1 , . . . , g N ] as follows. Suppose the SVD of the sentence matrix of the ith sentence is S = [v w1 , . . . , v wn ] = U ΣV T . Then the coarse-grained embedding for the ith sentence is defined as:

EQUATION

where f (σ j ) is a monotonically increasing function. We then compute the top

K principal vectors {d 1 , ..., d K } of X c , with singular values σ 1 ≥ σ 2 ≥ ... ≥ σ K .

2.4.2. UNIQUENESS SCORE:

In contrast to Arora et al. (2017) , we select different principal vectors of X c for each sentence, as different sentences may have disparate alignments with the corpus. For each sentence, {d 1 , ..., d K } are re-ranked in descending order of their correlation with sentence matrix S. The correlation is defined as 

o i = σ i S T d i 2 , 1 ≤ i ≤ K. Next,

d = [σ t1 , ..., σ t h ] ∈ R h .

Finally, a word w i with new semantic meaning vector q i in this sentence will be assigned a corpuswise uniqueness score:

EQUATION

This ensures that common stop words will have their effect diminished since their embeddings are closely aligned with the corpus' principal directions. The sentence to encode, with words w 1 to w n . And the contextual window of word w i is inside the dashed frame. Bottom middle: Form S i for w i , compute q i and novelty score α n (Section 2.1 and Section 2.2). Bottom left: Compute the SVD of S i and significance score α s (Section 2.3). Bottom right: Re-rank and select from principal components (orange blocks) and compute uniqueness score α u (Section 2.4).

2.5. SENTENCE VECTOR:

A sentence vector c s is computed as a weighted sum of its word embeddings, where the weights come from three scores: a novelty score (α n ), a significance score (α s ) and a corpus-wise uniqueness score (α u ).

EQUATION

We provide a theoretical explanation of Equation ( 9) in Appendix.

Sentence-Dependent Removal of Principal Components. Arora et al. (2017) shows that given a set of sentence vectors, removing projections onto the principal components of the spanned subspace can significantly enhance the performance on semantic similarity task. However, as each sentence may have a different semantic meaning, it could be sub-optimal to remove the same set of principal components from all sentences.

Therefore, we propose the sentence-dependent principal component removal (SDR), where we rerank top principal vectors based on correlation with each sentence. Using the method from Section 2.4.2, we obtain D = {d t1 , ..., d tr } for a sentence s. The final embedding of this sentence is then computed as:

EQUATION

Ablation experiments show that sentence-dependent principal component removal can achieve better result. The complete algorithm is summarized in Algorithm 1 with an illustration in Figure 1 .

3. EXPERIMENTS:



3.1. SEMANTIC SIMILARITY TASKS: STS BENCHMARK:

We evaluate our model on the STS Benchmark (Cer et al., 2017) , a sentence-level semantic similarity dataset from SemEval and SEM STS. The goal for a model is to predict a similarity score of two sentences given a sentence pair. The evaluation is by the Pearson's coefficient r between humanlabeled similarity (0 -5 points) and predictions.

Experimental settings. We report two versions of our model, one only using GloVe word vectors (GEM + GloVe), and the other using word vectors concatenated from LexVec, fastText and PSL (Wieting et al., 2015b) Form matrix S ∈ R d×n , S :,j = v wj and w j is the jth word in s 5:

The SVD is S = U ΣV T

6::

The ith column of the coarse-grained sentence embedding matrix X c :,i is U (σ(S)) 3 7: end for 8: Take first K singular vectors {d 1 , ..., d K } and singular values σ 1 ≥ σ 2 ≥ ... ≥ σ K of X c 9: for sentence s in S do 10:

Re-rank {d 1 , ..., d K } in descending order by for word w i in s do 13:

o i = σ i S T d i 2 , 1 ≤ i ≤ K.

S i = [v wi-m , ..., v wi-1 , v wi+1 , ..., v wi+m , v wi ]

is the contextual window matrix of w i .

14::

Do QR decomposition S i = Q i R i , let q i and r denote the last column of Q i and R i 15:

α n = exp(r -1 / r 2 ), α s = r -1 /(2m + 1), α u = exp (-σ d (q T i D) 2 /h)

16: (Wieting & Gimpel, 2017) 81.8 76.4 InferSent (Conneau et al., 2017) 80.1 75.8 Sent2Vec (Pagliardini et al., 2018) 78.7 75.5 Paragram-Phrase (Wieting et al., 2015a) 

α i = α n + α s +

3.2. SEMANTIC SIMILARITY TASKS: CQA:

We evaluate our model on subtask B of the SemEval Community Question Answering (CQA) task, another semantic similarity dataset. Given an original question Q o and a set of the first ten related questions (Q 1 , ..., Q 10 ) retrieved by a search engine, the model is expected to re-rank the related questions according to their similarity with respect to the original question. Each retrieved question Q i is labelled "PerfectMatch", "Relevant" or "Irrelevant", with respect to Q o . Mean average precision (MAP) is used as the evaluation measure.

We encode each question text into a unit vector u. Retrieved questions {Q i } 10 i=1 are ranked according to their cosine similarity with Q o . Results are shown in Table 2 . For comparison, we include results from the best models in 2017 competition: SimBow (Charlet & Damnati, 2017) , KeLP (Filice et al., 2017) , and Reddit + SNLI tuned. Note that all three benchmark models require training, and SimBow and KeLP leverage optional features including usage of comments and user profiles. In comparison, our model only uses the question text without any training. Our model clearly outperforms both Reddit + SNLI tuned and SimBow-primary, and on par with KeLP model.

3.3. SUPERVISED TASKS:

We further test our model on nine supervised tasks, including seven classification tasks: movie review (MR) (Pang & Lee, 2005) , Stanford Sentiment Treebank (SST) (Socher et al., 2013) , questiontype classification (TREC) (Voorhees & Dang, 2003) , opinion polarity (MPQA) (Wiebe et al., 2005) , product reviews (CR) (Hu & Liu, 2004) , subjectivity/objectivity classification (SUBJ) (Pang & Lee, 2004) and paraphrase identification (MRPC) (Dolan et al., 2004) . We also evaluate on SICK similarity (SICK-R), the SICK entailment (SICK-E) (Marelli et al., 2014) . The sentence embeddings generated are fixed and only the downstream task-specific neural structure is learned. The four hyper-parameters are chosen the same as in STS benchmark experiment. Results are in Table 3 .

GEM outperforms all non-parameterized sentence embedding models, including SIF, p-mean (Rücklé et al., 2018) , and BOW on GloVe. It also compares favorably with most of parameterized models, including à la carte (Khodak et al., 2018) , FastSent (Hill et al., 2016) , InferSent, QT, Sent2Vec, SkipThought-LN (with layer normalization) (Kiros et al., 2015) , SDAE (Hill et al., 2016) , STN (Subramanian et al., 2018) and USE (Yang et al., 2018) . Note that sentence representations generated by GEM have much smaller dimension compared to most of benchmark models, and the subsequent neural structure has fewer learnable parameters. The fact that GEM does well on several classification tasks (e.g. TREC and SUBJ) indicates that the proposed weight scheme is able to recognize important words in the sentence. Also, GEM's competitive performance on sentiment tasks shows that exploiting the geometric structures of two sentence subspaces is beneficial. q q q q q q q q q q q q q q q 80 85 90 5 6 7 8 9

window size Performance q q q q q q q q q q q q q q q 80 85 90 35 45 55 65 75 K q q q q q q q q q q q q q q q 80 85 90 8 11 14 17 20 h q q q q q q q q q q q q q q q 80 85 90 1 2 3 4 5 power of σ q q q MPQA STSB SUBJ

Figure 2 : Sensitivity tests on four hyper-parameters, the window size m in contextual window matrix, the number of candidate principal components K, the number of principal components to remove h, and the exponential power of singular value in coarse sentence embedding.

4. DISCUSSION:

Ablation Study. As shown in in Sensitivity Study. We evaluate the effect of all four hyper-parameters in our model: the window size m in the contextual window matrix, the number of candidate principal components K, the number of principal components to remove h, and the power of the singular value in coarse sentence embedding, i.e. the power t in f (σ j ) = σ t j in Equation ( 7). We sweep the hyper-parameters and test on STSB dev set, SUBJ, and MPQA. Unspecified parameters are fixed at m = 7, K = 45, h = 17 and t = 3. As shown in Figure 2 , our model is quite robust with respect to hyper-parameters.

Inference speed. We also compare the inference speed of our algorithm on the STSB test set with the benchmark models SkipThought and InferSent. SkipThought and InferSent are run on a NVIDIA Tesla P100, and our model is run on a CPU (Intel R Xeon R CPU E5-2690 v4 @2.60GHz). For fair comparison, batch size in InferSent and SkipThought is set to be 1. The results are shown in Table 5 . It shows that without acceleration from GPU, our model is still faster than InferSent and is 54% faster than SkipThought. 

A PROOF:

The novelty score (α n ), significance score (α s ) and corpus-wise uniqueness score (α u ) are larger when a word w has relatively rare appearance in the corpus and can bring in new and important semantic meaning to the sentence.

Following the section 3 in Arora et al. (2017) , we can use the probability of a word w emitted from sentence s in a dynamic process to explain eq. ( 9) and put this as following Theorem with its proof provided below.

Theorem 1. Suppose the probability that word w i is emitted from sentence s is2 :

EQUATION

where c s is the sentence embedding, Z = wi∈V exp( c s , v wi ) and V denotes the vocabulary. Then when Z is sufficiently large, the MLE for c s is:

EQUATION

Proof: According to Equation (11),

EQUATION

Where N and Z are two partition functions defined as

EQUATION

The joint probability of sentence s is then Again by Taylor expansion on Z,

EQUATION

)

The approximation is based on the assumption that Z is sufficiently large. It follows that,

EQUATION

Then the maximum log likelihood estimation of c s is:

EQUATION

B EXPERIMENTAL SETTINGS For all experiments, sentences are tokenized using the NLTK tokenizer (Bird et al., 2009) wordpunct tokenize, and all punctuation is skipped. f (σ j ) = σ t j in Equation ( 7). In the STS benchmark dataset, our hyper-parameters are chosen by conducting parameters search on STSB dev set at m = 7, h = 17, K = 45, and t = 3. And we use the same values for all supervised tasks. The integer interval of parameters search are m ∈ [5, 9], h ∈ [8, 20], L ∈ [35, 75] (at stride of 5), and t ∈ [1, 5]. In CQA dataset, m and h are changed to 6 and 15, the correlation term in section 2.4.2 is changed to o i = S T d i 2 empirically. In supervised tasks, same as Arora et al. (2017) , we do not perform principal components in supervised tasks.

C ENCODE A LONG SEQUENCE OF WORDS:

We would like to give a clarification on encoding a long sequence of words, for example, a paragraph or a article. Specifically, the length n of the sequence is larger than the dimension d of pre-trained word vectors in this case. The only part in GEM relevant to the the length of the sequence n is the coarse embedding in Equation ( 7). The SVD of the sentence matrix of the ith sentence is still S ∈ R d×n = [v w1 , . . . , v wn ] = U ΣV T , where now U ∈ R d×d , Σ ∈ R d×n , and V ∈ R n×n . Note that the d + 1th column to nth column in Σ are all zero. And Equation (7) becomes g i = d j=1 f (σ j )U :,j . The rest of the algorithm works as usual.

:

The code of GEM will be published soon.

The first term is adapted fromArora et al. (2017), where words near the sentence vector cs has higher probability to be generated. The second term is introduced so that words similar to the context in the sentence or close to common words in the corpus are also likely to occur.

Figures and Tables:

Figure1: An illustration of GEM algorithm. Top middle: The sentence to encode, with words w 1 to w n . And the contextual window of word w i is inside the dashed frame. Bottom middle: Form S i for w i , compute q i and novelty score α n (Section 2.1 and Section 2.2). Bottom left: Compute the SVD of S i and significance score α s (Section 2.3). Bottom right: Re-rank and select from principal components (orange blocks) and compute uniqueness score α u (Section 2.4).

h principal vectors as D = [d t1 , ..., d t h ], with singular values σ d = [σ t1 ...., σ t h ].12:

notation, let α = α n + α s + α u . It follows that the log likelihood f (w i ) of word w i emitted from sentence s is given byf wi (c s ) = log( exp( c s , v wi ) Z + e -α ) -log(N ) (16) ∇f wi (c s ) = exp( c s , v wi )v wi exp( c s , v wi ) + Ze -α(17)By Taylor expansion, we havef wi (c s ) ≈ f wi (0) + ∇f wi (0) T c s = constant + c s , v wi Ze -α + 1 (18)

the top h principal vectors after re-ranking based on o i are selected: D = {d t1 , ..., d t h }, with o t1 ≥ o t2 ≥ ... ≥ o t h and their singular values in X c are σ

Table content: <table/>

Inputs: A set of sentences S, vocabulary V, word embeddings {v w ∈ R d | w ∈ V} 2: Outputs: Sentence embeddings {c s ∈ R d | s ∈ S} 3: for ith sentence s in S do

Table content: <table><tr><td>4:</td></tr></table>

MAP on CQA subtask B normalized sentence vectors. Since our model is non-parameterized, it does not utilize any information from the dev set when evaluating on the test set and vice versa. Hyper-parameters are chosen at m = 7, h = 17, K = 45, and t = 3 by conducing hyper-parameters search on dev set. Results on

Table content: <table><tr><td/><td>GEM + L.F.P</td><td>48.97</td></tr><tr><td/><td colspan="2">Reddit + SNLI tuned 47.44</td></tr><tr><td/><td>KeLP-contrastive1</td><td>49.00</td></tr><tr><td/><td colspan="2">SimBow-contrastive2 47.87</td></tr><tr><td/><td>SimBow-primary</td><td>47.22</td></tr><tr><td>73.9</td><td>73.2</td></tr><tr><td>Table 1: Pearson's r× 100 on STSB</td><td/></tr></table>

Results on supervised tasks. Sentence embeddings are fixed for downstream supervised tasks. Best results for each task are underlined, best results from models in the same category are in bold. SIF results are extracted fromArora et al. (2017) andRücklé et al. (2018), and some training time is collected fromLogeswaran & Lee (2018).

Table content: <table/>

Table4, every GEM weight (α n , α s , α u ) and proposed principal components removal methods contribute to the performance. As listed on the left, adding GEM weights improves the score by 8.6% on STS dataset compared with averaging three concatenated word vectors. The sentence-dependent principal component removal (SDR) proposed in GEM improves 0.3% compared to directly removing the top h corpus principal components (SIR). Using GEM weights and SDR together yields an overall improvement of 19.7%. As shown on the right in Table4, every weight contributes to the performance of our model. For example, three weights altogether improve the score in SUBJ task by 0.38% compared with only using α n . Comparison of different configurations demonstrates the effectiveness of our model on STSB dev set and SUBJ. SDR stands for sentence-dependent principal component removal in Section 2.4.2. SIR stands for sentence-independent principal component removal, i.e. directly removing top h corpus principal components from the sentence embedding.

Table content: <table><tr><td>Configurations</td><td>STSB dev</td><td>Configurations</td><td colspan="2">STSB dev SUBJ</td></tr><tr><td>Mean of L.F.P GEM weights GEM weights + SIR GEM weights + SDR</td><td>62.4 71.0 81.8 82.1</td><td>α n + SDR α n , α s + SDR α n , α s , α u + SDR</td><td>81.6 81.9 82.1</td><td>93.42 93.6 93.8</td></tr></table>

Running time of GEM, InferSent and SkipThought on encoding sentences in STSB test set. GEM is run on CPU, InferSent and SkipThought is run on GPU. Data are collected from 5 trials.

Table content: <table/>

Peer review (iclr2019):
 This is a paper about sentence embedding based on orthogonal decomposition of the spanned space by word embeddings. Via Gram-Schmidt process, the sequence of words in a sentence is regarded as a sequence of incoming vectors to be orthogonalized. Each word is then assigned 3 scores: novelty score, significance score, and uniqueness score. Eventually, the sentence embedding is achieved as weighted average of word embeddings based on those scores. The authors conduct extensive experiments to demonstrate the performance of the proposed embedding. I think the idea of the paper is novel and inspiring. But there are several issues and possible areas to improve: 1. What if the length of the sentence is larger than the dimension of the word embedding? Some of the 3 scores will not be well-defined. 2. Gram-Schmidt process is sensitive to the order of the incoming vectors. A well-defined sentence embedding algorithm should not. I suggest the authors to evaluate whether this is an issue. For example, if by simply removing a non-important stop word at the begging of the sentence and then the sentence embedding changes drastically, then it indicates that the embedding is problematic. 3. I’m confused by the classification between training-free sentence embedding and unsupervised sentence embedding? Don’t both of them require training word2vec-type embedding? 4. The definition of the three scores seems reasonable, but requires further evidence to justify. For example, by the definition of the scores, do we have any proof that the value of .alpha indeed demonstrated the related importance level?