Poincare Glove: Hyperbolic Word Embeddings

1 INTRODUCTION & MOTIVATION . Word embeddings are ubiquitous nowadays as first layers in neural network and deep learning models for natural language processing . They are essential in order to move from the discrete word space to the continuous space where differentiable loss functions can be optimized . The popular models of Glove ( Pennington et al. , 2014 ) , Word2Vec ( Mikolov et al. , 2013b ) or FastText ( Bojanowski et al. , 2016 ) , provide efficient ways to learn word vectors fully unsupervised from raw text corpora , solely based on word co-occurrence statistics . These models are then successfully applied to word similarity and other downstream tasks and , surprisingly ( or not ( Arora et al. , 2016 ) ) , exhibit a linear algebraic structure that is also useful to solve word analogy . However , unsupervised word embeddings still largely suffer from revealing asymmetric word relations including the latent hierarchical structure of words . This is currently one of the key limitations in automatic text understanding , e.g . for tasks such as textual entailment ( Bowman et al. , 2015 ) . To address this issue , ( Vilnis & McCallum , 2015 ; Muzellec & Cuturi , 2018 ) propose to move from point embeddings to probability density functions , the simplest being Gaussian or Elliptical distributions . Their intuition is that the variance of such a distribution should encode the generality/specificity of the respective word . However , this method results in losing the arithmetic properties of point embeddings ( e.g . for analogy reasoning ) and becomes unclear how to properly use them in downstream tasks . To this end , we propose to take the best from both worlds : we embed words as points in a Cartesian product of hyperbolic spaces and , additionally , explain how they are bijectively mapped to Gaussian embeddings with diagonal covariance matrices , where the hyperbolic distance between two points becomes the Fisher distance between the corresponding probability distribution functions ( PDFs ) . This allows us to derive a novel principled is-a score on top of word embeddings that can be leveraged for hypernymy detection . We learn these word embeddings unsupervised from raw text by generalizing the Glove method . Moreover , the linear arithmetic property used for solving word analogy has a mathematical grounded correspondence in this new space based on the established notion of parallel transport in Riemannian manifolds . In addition , these hyperbolic embeddings outperform Euclidean Glove on word similarity benchmarks . We thus describe , to our knowledge , ∗All authors contributed equally . the first word embedding model that competitively addresses the above three tasks simultaneously . Finally , these word vectors can also be used in downstream tasks as explained by Ganea et al . ( 2018b ) . We provide additional reasons for choosing the hyperbolic geometry to embed words . We explain the notion of average δ-hyperbolicity of a graph , a geometric quantity that measures its ” democracy ” ( Borassi et al. , 2015 ) . A small hyperbolicity constant implies ” aristocracy ” , namely the existence of a small set of nodes that ” influence ” most of the paths in the graph . It is known that real-world graphs are mainly complex networks ( e.g . scale-free exhibiting power-law node degree distributions ) which in turn are better embedded in a tree-like space , i.e . hyperbolic ( Krioukov et al. , 2010 ) . Since , intuitively , words form an ” aristocratic ” community ( few generic ones from different topics and many more specific ones ) and since a significant subset of them exhibits a hierarchical structure ( e.g . WordNet ( Miller et al. , 1990 ) ) , it is naturally to learn hyperbolic word embeddings . Moreover , we empirically measure very low average δ-hyperbolicity constants of some variants of the word log-co-occurrence graph ( used by the Glove method ) , providing additional quantitative reasons for why spaces of negative curvature ( i.e . hyperbolic ) are better suited for word representations . 2 RELATED WORK . Recent supervised methods can be applied to embed any tree or directed acyclic graph in a low dimensional space with the aim of improving link prediction either by imposing a partial order in the embedding space ( Vendrov et al. , 2015 ; Vilnis et al. , 2018 ; Athiwaratkun & Wilson , 2018 ) , by using hyperbolic geometry ( Nickel & Kiela , 2017 ; 2018 ) , or both ( Ganea et al. , 2018a ) . To learn word embeddings that exhibit hypernymy or hierarchical information , supervised methods ( Vulić & Mrkšić , 2018 ; Nguyen et al. , 2017 ) leverage external information ( e.g . WordNet ) together with raw text corpora . However , the same goal is also targeted by more ambitious fully unsupervised models which move away from the ” point ” assumption and learn various probability densities for each word ( Vilnis & McCallum , 2015 ; Muzellec & Cuturi , 2018 ; Athiwaratkun & Wilson , 2017 ; Singh et al. , 2018 ) . There have been two very recent attempts at learning unsupervised word embeddings in the hyperbolic space ( Leimeister & Wilson , 2018 ; Dhingra et al. , 2018 ) . However , they suffer from either not being competitive on standard tasks in high dimensions , not showing the benefit of using hyperbolic spaces to model asymmetric relations , or not being trained on realistically large corpora . We address these problems and , moreover , the connection with density based methods is made explicit and leveraged to improve hypernymy detection . 3 HYPERBOLIC SPACES AND THEIR CARTESIAN PRODUCT . In order to work in the hyperbolic space , we have to choose one model , among the five isometric models that exist . We choose to embed words in the Poincaré ball Dn = { x ∈ Rn | ‖x‖2 < 1 } . This is illustrated in Figure 1a for n = 2 , where dark lines represent geodesics . The distance function in this space is given by dDn ( x , y ) = cosh −1 ( 1 + λxλy‖x− y‖22/2 ) , λx : = 2/ ( 1 − ‖x‖22 ) being the conformal factor . We will also embed words in products of hyperbolic spaces , and explain why later on . A product of p balls ( Dn ) p , with the induced product geometry , is known to have distance function d ( Dn ) p ( x , y ) 2 = ∑p i=1 dDn ( xi , yi ) 2 . Finally , another model of interest for us is the Poincaré half-plane H2 = R×R∗+ illustrated in Figure 1d , with distance function dH2 ( x , y ) = cosh −1 ( 1 + ‖x− y‖22/ ( 2y1y2 ) ) . Figure 1 shows an isometry ϕ from D2 to H2 mapping the vertical segment { 0 } × ( −1 , 1 ) to R∗+ and fixing ( 0 , 1 ) , sending the radius to∞ . 4 ADAPTING GLOVE . Euclidean GLOVE . The GLOVE ( Pennington et al. , 2014 ) algorithm is an unsupervised method for learning word representations in the Euclidean space from statistics of word co-occurrences in a text corpus , with the aim to geometrically capture the words ’ meaning and relations . We use the notations : Xij is the number of times word j occurs in the same window context as word i ; Xi = ∑ kXik ; Pij = Xij/Xi is the probability that word j appears in the context of word i . An embedding of a ( target ) word i is written wi , while an embedding of a context word k is written w̃k . The initial formulation of the GLOVE model suggests to learn embeddings as to satisfy the equation wTi w̃k = log ( Pik ) = log ( Xik ) − log ( Xi ) . Since Xik is symmetric in ( i , k ) but Pik is not , ( Pennington et al. , 2014 ) propose to restore the symmetry by introducing biases for each word , absorbing log ( Xi ) into i ’ s bias : wTi w̃k + bi + b̃k = log ( Xik ) . ( 1 ) Finally , the authors suggest to enforce this equality by optimizing a weighted least-square loss : J = V∑ i , j=1 f ( Xij ) ( wTi w̃j + bi + b̃j − logXij ) 2 , ( 2 ) where V is the size of the vocabulary and f down-weights the signal coming from frequent words ( it is typically chosen to be f ( x ) = min { 1 , ( x/xm ) α } , with α = 3/4 and xm = 100 ) . GLOVE in metric spaces . Note that there is no clear correspondence of the Euclidean inner-product in a hyperbolic space . However , we are provided with a distance function . Further notice that one could rewrite Eq . ( 1 ) with the Euclidean distance as− 12‖wi− w̃k‖ 2 + bi + b̃k = log ( Xik ) , where we absorbed the squared norms of the embeddings into the biases . We thus replace the GLOVE loss by : J = V∑ i , j=1 f ( Xij ) ( −h ( d ( wi , w̃j ) ) + bi + b̃j − logXij ) 2 , ( 3 ) where h is a function to be chosen as a hyperparameter of the model , and d can be any differentiable distance function . Although the most direct correspondence with GLOVE would suggest h ( x ) = x2/2 , we sometimes obtained better results with other functions , such as h = cosh2 ( see sections 8 & 9 ) . Note that De Sa et al . ( 2018 ) also apply cosh to their distance matrix for hyperbolic MDS before applying PCA . Understanding why h = cosh2 is a good choice would be interesting future work . 5 CONNECTING GAUSSIAN EMBEDDINGS & HYPERBOLIC EMBEDDINGS . In order to endow Euclidean word embeddings with richer information , Vilnis & McCallum ( 2015 ) proposed to represent words as Gaussians , i.e . with a mean vector and a covariance matrix1 , expecting the variance parameters to capture how generic/specific a word is , and , hopefully , entailment relations . On the other hand , Nickel & Kiela ( 2017 ) proposed to embed words of the WordNet hierarchy ( Miller et al. , 1990 ) in hyperbolic space , because this space is mathematically known to be better suited to embed tree-like graphs . It is hence natural to wonder : is there a connection between the two ? The Fisher geometry of Gaussians is hyperbolic . It turns out that there exists a striking connection ( Costa et al. , 2015 ) . Note that a 1D Gaussian N ( µ , σ2 ) can be represented as a point ( µ , σ ) in R×R∗+ . Then , the Fisher distance between two distributions relates to the hyperbolic distance in H2 : dF ( N ( µ , σ2 ) , N ( µ′ , σ′2 ) ) = √ 2dH2 ( ( µ/ √ 2 , σ ) , ( µ′/ √ 2 , σ′ ) ) . ( 4 ) For n-dimensional Gaussians with diagonal covariance matrices written Σ = diag ( σ ) 2 , it becomes : dF ( N ( µ , Σ ) , N ( µ′ , Σ′ ) ) = √√√√ n∑ i=1 2dH2 ( ( µi/ √ 2 , σi ) , ( µ′i/ √ 2 , σ′i ) ) 2 . ( 5 ) Hence there is a direct correspondence between diagonal Gaussians and the product space ( H2 ) n. 1diagonal or even spherical , for simplicity . Fisher distance , KL & Gaussian embeddings . The above paragraph lets us relate the WORD2GAUSS algorithm ( Vilnis & McCallum , 2015 ) to hyperbolic word embeddings . Although one could object that WORD2GAUSS is trained using a KL divergence , while hyperbolic embeddings relate to Gaussian distributions via the Fisher distance dF , let us remind that KL and dF define the same local geometry . Indeed , the KL is known to be related to dF , as its local approximation ( Jeffreys , 1946 ) . In short , if P ( θ + dθ ) and P ( θ ) denote two closeby probability distributions for a small dθ , then KL ( P ( θ + dθ ) ||P ( θ ) ) = ( 1/2 ) ∑ ij gijdθ idθj + O ( ‖dθ‖3 ) , where ( gij ) ij is the Fisher information metric , inducing dF . Riemannian optimization . A benefit of representing words in ( products of ) hyperbolic spaces , as opposed to ( diagonal ) Gaussian distributions , is that one can use recent Riemannian adaptive optimization tools such as RADAGRAD ( Bécigneul & Ganea , 2018 ) . Note that without this connection , it would be unclear how to define a variant of ADAGRAD ( Duchi et al. , 2011 ) intrinsic to the statistical manifold of Gaussians . Empirically , we indeed noticed better results using RADAGRAD , rather than simply Riemannian SGD ( Bonnabel , 2013 ) . Similarly , note that GLOVE trains with ADAGRAD .

Summary: Words have implicit hierarchy among themselves in a text. Hyperbolic geometry due to the negative curvature and the delta-hyperbolicity is more suitable for representing hierarchical data in the continuous space. As a result it is natural to learn word representations/embeddings in the hyperbolic space. This paper proposes a promising approach that extends the approach presented in [1] to implement a GLOVE based hyperbolic word embedding model. The embeddings are optimized by using the Riemannian Optimization methods presented in [2]. Authors provide results on word-similarity and word-analogy tasks.