Zero-training Sentence Embedding via Orthogonal Basis

1 INTRODUCTION . The concept of word embeddings has been prevalent in NLP community in recent years , as they can characterize semantic similarity between any pair of words , achieving promising results in a large number of NLP tasks ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ; Salle et al. , 2016 ) . However , due to the hierarchical nature of human language , it is not sufficient to comprehend text solely based on isolated understanding of each word . This has prompted a recent rise in search for semantically robust embeddings for longer pieces of text , such as sentences and paragraphs . Based on learning paradigms , the existing approaches to sentence embeddings can be categorized into two categories : i ) parameterized methods and ii ) non-parameterized methods . Parameterized sentence embeddings . These models are parameterized and require training to optimize their parameters . SkipThought ( Kiros et al. , 2015 ) is an encoder-decoder model that predicts adjacent sentences . Pagliardini et al . ( 2018 ) proposes an unsupervised model , Sent2Vec , to learn an n-gram feature in a sentence to predict the center word from the surrounding context . Quick thoughts ( QT ) ( Logeswaran & Lee , 2018 ) replaces the encoder with a classifier to predict context sentences from candidate sequences . Khodak et al . ( 2018 ) proposes à la carte to learn a linear mapping to reconstruct the center word from its context . Conneau et al . ( 2017 ) generates the sentence encoder InferSent using Natural Language Inference ( NLI ) dataset . Universal Sentence Encoder ( Yang et al. , 2018 ; Cer et al. , 2018 ) utilizes the transformer ( Vaswani et al. , 2017 ) for sentence embeddings . The model is first trained on large scale of unsupervised data from Wikipedia and forums , and then trained on the Stanford Natural Language Inference ( SNLI ) dataset . Wieting & Gimpel ( 2017 ) propose the gated recurrent averaging network ( GRAN ) , which is trained on Paraphrase Database ( PPDB ) and English Wikipedia . Subramanian et al . ( 2018 ) leverages a multi-task learning framework to generate sentence embeddings . Wieting et al . ( 2015a ) learns the paraphrastic sentence representations as the simple average of updated word embeddings . Non-parameterized sentence embedding . Recent work ( Arora et al. , 2017 ) shows that , surprisingly , a weighted sum or transformation of word representations can outperform many sophisticated neural network structures in sentence embedding tasks . These methods are parameter-free and require no further training upon pre-trained word vectors . Arora et al . ( 2017 ) constructs a sentence embedding called SIF as a sum of pre-trained word embeddings , weighted by reverse document frequency . Rücklé et al . ( 2018 ) concatenates different power mean word embeddings as a sentence vector in p-mean . As these methods do not have a parameterized model , they can be easily adapted to novel text domains with both fast inference speed and high-quality sentence embeddings . In view of this trend , our work aims to further advance the frontier of this group and make its new state-of-the-art . In this paper , we propose a novel sentence embedding algorithm , Geometric Embedding ( GEM ) , based entirely on the geometric structure of word embedding space . Given a d-dim word embedding matrix A ∈ Rd×n for a sentence with n words , any linear combination of the sentence ’ s word embeddings lies in the subspace spanned by the n word vectors . We analyze the geometric structure of this subspace in Rd . When we consider the words in a sentence one-by-one in order , each word may bring in a novel orthogonal basis to the existing subspace . This new basis can be considered as the new semantic meaning brought in by this word , while the length of projection in this direction can indicate the intensity of this new meaning . It follows that a word with a strong intensity should have a larger influence in the sentence ’ s meaning . Thus , these intensities can be converted into weights to linearly combine all word embeddings to obtain the sentence embedding . In this paper , we theoretically frame the above approach in a QR factorization of the word embedding matrix A . Furthermore , since the meaning and importance of a word largely depends on its close neighborhood , we propose the sliding-window QR factorization method to capture the context of a word and characterize its significance within the context . In the last step , we adapt a similar approach as Arora et al . ( 2017 ) to remove top principal vectors before generating the final sentence embedding . This step is to ensure commonly shared background components , e.g . stop words , do not bias sentence similarity comparison . As we build a new orthogonal basis for each sentence , we propose to have disparate background components for each sentence . This motivates us to put forward a sentence-specific principal vector removal method , leading to better empirical results . We evaluate our algorithm on 11 NLP tasks . In all of these tasks , our algorithm outperforms all non-parameterized methods and many parameterized approaches . For example , compared to SIF ( Arora et al. , 2017 ) , the performance is boosted by 5.5 % on STS benchmark dataset , and by 2.5 % on SST dataset . Plus , the running time of our model compares favorably with existing models . The rest of this paper is organized as following . In Section 2 , we describe our sentence embedding algorithm GEM . We evaluate our model on various tasks in Section 3 and Section 4 . Finally , we summarize our work in Section 5 . 2 APPROACH . 2.1 QUANTIFY NEW SEMANTIC MEANING . Let us consider the idea of word embeddings ( Mikolov et al. , 2013 ) , where a word wi is projected as a vector vwi ∈ Rd . Any sequence of words can be viewed as a subspace in Rd spanned by its word vectors . Before the appearance of the ith word , S is a subspace in Rd spanned by { vw1 , vw2 , ... , vwi−1 } . Its orthonormal basis is { q1 , q2 , ... , qi−1 } . The embedding vwi of the ith word wi can be decomposed into vwi = i−1∑ j=1 rjqj + riqi rj = q T j vwi ri = ‖vwi − i−1∑ j=1 rjqj‖2 ( 1 ) where ∑i−1 j=1 rjqj is the part in vwi that resides in subspace S , and qi is orthogonal to S and is to be added to S. The above algorithm is also known as Gram-Schmidt Process . In the case of rank deficiency , i.e. , vwi is already a linear combination of { q1 , q2 , ... qi−1 } , qi is a zero vector and ri = 0 . In matrix form , this process is also known as QR factorization , defined as follows . QR factorization . Define an embedding matrix of n words as A = [ A : ,1 , A : ,2 , ... , A : ,n ] ∈ Rd×n , where A : ,i is the embedding of the ith word wi in a word sequence ( w1 , . . . , wi , . . . , wn ) . A ∈ Rd×n can be factorized intoA = QR , where the non-zero columns inQ ∈ Rd×n are the orthonormal basis , andR ∈ Rn×n is an upper triangular matrix . The process above computes the novel semantic meaning of a word w.r.t all preceding words . As the meaning of a word influences and is influenced by its close neighbors , we now calculate the novel orthogonal basis vector qi of each word wi in its neighborhood , rather than only w.r.t the preceding words . Definition 1 ( Contextual Window Matrix ) Given a word wi , and its m-neighborhood window inside the sentence ( wi−m , . . . , wi−1 , wi , wi+1 , . . . , wi+m ) , define the contextual window matrix of word wi as : Si = [ vwi−m , . . . , vwi−1 , vwi+1 , . . . , vwi+m , vwi ] ∈ Rd× ( 2m+1 ) ( 2 ) Here we shuffle vwi to the end of S i to compute its novel semantic information compared with its context . Now the QR factorization of Si is Si = QiRi ( 3 ) Note that qi is the last column ofQi , which is also the new orthogonal basis vector to this contextual window matrix . Next , in order to generate the embedding for a sentence , we will assign a weight to each of its words . This weight should characterize how much new and important information a word brings to the sentence . The previous process yields the orthogonal basis vector qi . We propose that qi represents the novel semantic meaning brought by word wi . We will now discuss how to quantify i ) the novelty of qi to other meanings in wi , ii ) the significance of qi to its context , and iii ) the corpus-wise uniqueness of qi w.r.t the whole corpus . 2.2 NOVELTY . We propose that a word wi is more important to a sentence if its novel orthogonal basis vector qi is a large component in vwi . This can be quantified as a novelty score : αn = exp ( r−1/‖vwi‖2 ) = exp ( r−1/‖r‖2 ) ( 4 ) where r is the last column ofRi , and r−1 is the last element of r. Connection to least square . From QR factorization theory , the novel orthogonal basis qi is also the normalized residual in the least square problem min ‖Cx − vwi‖2 , i.e . qTi vwi = r−1 = min ‖Cx − vwi‖2 , where C = Si : ,1:2m . And qTi vwi = r−1 is the minimum distance from word vector vwi to the hyper-plane spanned by wi ’ s context words wi−m , . . . , wi−1 , wi+1 , . . . , wi+m . It follows that αn is the exponential of the normalized distance between vwi and the subspace spanned by its context . 2.3 SIGNIFICANCE . The significance of a word is related to how semantically aligned it is to the meaning of its context . To identify principal directions , i.e . meanings , in the contextual window matrix Si , we employ Singular Value Decomposition . Singular Value Decomposition . Given a matrix A ∈ Rd×n , there exists U ∈ Rd×n with orthogonal columns , diagonal matrix Σ = diag ( σ1 , ... , σn ) , σ1 ≥ σ2 ≥ ... ≥ σn ≥ 0 , and orthogonal matrix V ∈ Rn×n , such thatA = UΣV T . The columns of U , { U : ,j } nj=1 , are an orthonormal basis of A ’ s columns subspace and we propose that they represent a set of semantic meanings from the context . Their corresponding singular values { σj } nj=1 , denoted by σ ( A ) , represent the importance associated with { U : ,j } nj=1 . The SVD of wi ’ s contextual window matrix is Si = U iΣiV iT ∈ Rd× ( 2m+1 ) . It follows that qTi U i is the coordinate of qi in the basis of { U i : ,j } 2m+1 j=1 . Intuitively , a word is more important if its novel semantic meaning has a better alignment with more principal meanings in its contextual window . This can be quantified as ‖σ ( Si ) ( qTi U i ) ‖2 , where denotes element-wise product . Therefore , we define the significance of wi in its context to be : αs = ‖σ ( Si ) ( qTi U i ) ‖2 2m+ 1 ( 5 ) It turns out αs can be rewritten as αs = ‖qTi U iΣi‖2 2m+ 1 = ‖qTi U iΣiV i‖2 2m+ 1 = ‖qTi Si‖2 2m+ 1 = qTi vwi 2m+ 1 = r−1 2m+ 1 ( 6 ) We use the fact that V i is an orthogonal matrix and qi is orthogonal to all but the last column of Si , vwi . Therefore , αs is essentially the distance between wi and the context hyper-plane , normalized by the context size .

Summary: Paper overview: This paper proposes a new geometry-based method for sentence embedding from word embedding vectors, inspired by Arora et al (2017). The idea is to quantify the novelty,significance and corpus-wise uniqueness of each word. In order to do so, they analyze geometrically how the word vector of the target word relates to 1) the subspace created by the word-vectors in its context 2) its alignment with the meanings in its context (using SVD) 3) its presence in the all the corpus. For each of these aspects, they output a score or weight. The final sentence representation is a weighted average, using these scores, of the word vectors of the sentence.