Poincare Glove: Hyperbolic Word Embeddings

1 INTRODUCTION & MOTIVATION . Word embeddings are ubiquitous nowadays as first layers in neural network and deep learning models for natural language processing . They are essential in order to move from the discrete word space to the continuous space where differentiable loss functions can be optimized . The popular models of Glove ( Pennington et al. , 2014 ) , Word2Vec ( Mikolov et al. , 2013b ) or FastText ( Bojanowski et al. , 2016 ) , provide efficient ways to learn word vectors fully unsupervised from raw text corpora , solely based on word co-occurrence statistics . These models are then successfully applied to word similarity and other downstream tasks and , surprisingly ( or not ( Arora et al. , 2016 ) ) , exhibit a linear algebraic structure that is also useful to solve word analogy . However , unsupervised word embeddings still largely suffer from revealing asymmetric word relations including the latent hierarchical structure of words . This is currently one of the key limitations in automatic text understanding , e.g . for tasks such as textual entailment ( Bowman et al. , 2015 ) . To address this issue , ( Vilnis & McCallum , 2015 ; Muzellec & Cuturi , 2018 ) propose to move from point embeddings to probability density functions , the simplest being Gaussian or Elliptical distributions . Their intuition is that the variance of such a distribution should encode the generality/specificity of the respective word . However , this method results in losing the arithmetic properties of point embeddings ( e.g . for analogy reasoning ) and becomes unclear how to properly use them in downstream tasks . To this end , we propose to take the best from both worlds : we embed words as points in a Cartesian product of hyperbolic spaces and , additionally , explain how they are bijectively mapped to Gaussian embeddings with diagonal covariance matrices , where the hyperbolic distance between two points becomes the Fisher distance between the corresponding probability distribution functions ( PDFs ) . This allows us to derive a novel principled is-a score on top of word embeddings that can be leveraged for hypernymy detection . We learn these word embeddings unsupervised from raw text by generalizing the Glove method . Moreover , the linear arithmetic property used for solving word analogy has a mathematical grounded correspondence in this new space based on the established notion of parallel transport in Riemannian manifolds . In addition , these hyperbolic embeddings outperform Euclidean Glove on word similarity benchmarks . We thus describe , to our knowledge , ∗All authors contributed equally . the first word embedding model that competitively addresses the above three tasks simultaneously . Finally , these word vectors can also be used in downstream tasks as explained by Ganea et al . ( 2018b ) . We provide additional reasons for choosing the hyperbolic geometry to embed words . We explain the notion of average δ-hyperbolicity of a graph , a geometric quantity that measures its ” democracy ” ( Borassi et al. , 2015 ) . A small hyperbolicity constant implies ” aristocracy ” , namely the existence of a small set of nodes that ” influence ” most of the paths in the graph . It is known that real-world graphs are mainly complex networks ( e.g . scale-free exhibiting power-law node degree distributions ) which in turn are better embedded in a tree-like space , i.e . hyperbolic ( Krioukov et al. , 2010 ) . Since , intuitively , words form an ” aristocratic ” community ( few generic ones from different topics and many more specific ones ) and since a significant subset of them exhibits a hierarchical structure ( e.g . WordNet ( Miller et al. , 1990 ) ) , it is naturally to learn hyperbolic word embeddings . Moreover , we empirically measure very low average δ-hyperbolicity constants of some variants of the word log-co-occurrence graph ( used by the Glove method ) , providing additional quantitative reasons for why spaces of negative curvature ( i.e . hyperbolic ) are better suited for word representations . 2 RELATED WORK . Recent supervised methods can be applied to embed any tree or directed acyclic graph in a low dimensional space with the aim of improving link prediction either by imposing a partial order in the embedding space ( Vendrov et al. , 2015 ; Vilnis et al. , 2018 ; Athiwaratkun & Wilson , 2018 ) , by using hyperbolic geometry ( Nickel & Kiela , 2017 ; 2018 ) , or both ( Ganea et al. , 2018a ) . To learn word embeddings that exhibit hypernymy or hierarchical information , supervised methods ( Vulić & Mrkšić , 2018 ; Nguyen et al. , 2017 ) leverage external information ( e.g . WordNet ) together with raw text corpora . However , the same goal is also targeted by more ambitious fully unsupervised models which move away from the ” point ” assumption and learn various probability densities for each word ( Vilnis & McCallum , 2015 ; Muzellec & Cuturi , 2018 ; Athiwaratkun & Wilson , 2017 ; Singh et al. , 2018 ) . There have been two very recent attempts at learning unsupervised word embeddings in the hyperbolic space ( Leimeister & Wilson , 2018 ; Dhingra et al. , 2018 ) . However , they suffer from either not being competitive on standard tasks in high dimensions , not showing the benefit of using hyperbolic spaces to model asymmetric relations , or not being trained on realistically large corpora . We address these problems and , moreover , the connection with density based methods is made explicit and leveraged to improve hypernymy detection . 3 HYPERBOLIC SPACES AND THEIR CARTESIAN PRODUCT . In order to work in the hyperbolic space , we have to choose one model , among the five isometric models that exist . We choose to embed words in the Poincaré ball Dn = { x ∈ Rn | ‖x‖2 < 1 } . This is illustrated in Figure 1a for n = 2 , where dark lines represent geodesics . The distance function in this space is given by dDn ( x , y ) = cosh −1 ( 1 + λxλy‖x− y‖22/2 ) , λx : = 2/ ( 1 − ‖x‖22 ) being the conformal factor . We will also embed words in products of hyperbolic spaces , and explain why later on . A product of p balls ( Dn ) p , with the induced product geometry , is known to have distance function d ( Dn ) p ( x , y ) 2 = ∑p i=1 dDn ( xi , yi ) 2 . Finally , another model of interest for us is the Poincaré half-plane H2 = R×R∗+ illustrated in Figure 1d , with distance function dH2 ( x , y ) = cosh −1 ( 1 + ‖x− y‖22/ ( 2y1y2 ) ) . Figure 1 shows an isometry ϕ from D2 to H2 mapping the vertical segment { 0 } × ( −1 , 1 ) to R∗+ and fixing ( 0 , 1 ) , sending the radius to∞ . 4 ADAPTING GLOVE . Euclidean GLOVE . The GLOVE ( Pennington et al. , 2014 ) algorithm is an unsupervised method for learning word representations in the Euclidean space from statistics of word co-occurrences in a text corpus , with the aim to geometrically capture the words ’ meaning and relations . We use the notations : Xij is the number of times word j occurs in the same window context as word i ; Xi = ∑ kXik ; Pij = Xij/Xi is the probability that word j appears in the context of word i . An embedding of a ( target ) word i is written wi , while an embedding of a context word k is written w̃k . The initial formulation of the GLOVE model suggests to learn embeddings as to satisfy the equation wTi w̃k = log ( Pik ) = log ( Xik ) − log ( Xi ) . Since Xik is symmetric in ( i , k ) but Pik is not , ( Pennington et al. , 2014 ) propose to restore the symmetry by introducing biases for each word , absorbing log ( Xi ) into i ’ s bias : wTi w̃k + bi + b̃k = log ( Xik ) . ( 1 ) Finally , the authors suggest to enforce this equality by optimizing a weighted least-square loss : J = V∑ i , j=1 f ( Xij ) ( wTi w̃j + bi + b̃j − logXij ) 2 , ( 2 ) where V is the size of the vocabulary and f down-weights the signal coming from frequent words ( it is typically chosen to be f ( x ) = min { 1 , ( x/xm ) α } , with α = 3/4 and xm = 100 ) . GLOVE in metric spaces . Note that there is no clear correspondence of the Euclidean inner-product in a hyperbolic space . However , we are provided with a distance function . Further notice that one could rewrite Eq . ( 1 ) with the Euclidean distance as− 12‖wi− w̃k‖ 2 + bi + b̃k = log ( Xik ) , where we absorbed the squared norms of the embeddings into the biases . We thus replace the GLOVE loss by : J = V∑ i , j=1 f ( Xij ) ( −h ( d ( wi , w̃j ) ) + bi + b̃j − logXij ) 2 , ( 3 ) where h is a function to be chosen as a hyperparameter of the model , and d can be any differentiable distance function . Although the most direct correspondence with GLOVE would suggest h ( x ) = x2/2 , we sometimes obtained better results with other functions , such as h = cosh2 ( see sections 8 & 9 ) . Note that De Sa et al . ( 2018 ) also apply cosh to their distance matrix for hyperbolic MDS before applying PCA . Understanding why h = cosh2 is a good choice would be interesting future work . 5 CONNECTING GAUSSIAN EMBEDDINGS & HYPERBOLIC EMBEDDINGS . In order to endow Euclidean word embeddings with richer information , Vilnis & McCallum ( 2015 ) proposed to represent words as Gaussians , i.e . with a mean vector and a covariance matrix1 , expecting the variance parameters to capture how generic/specific a word is , and , hopefully , entailment relations . On the other hand , Nickel & Kiela ( 2017 ) proposed to embed words of the WordNet hierarchy ( Miller et al. , 1990 ) in hyperbolic space , because this space is mathematically known to be better suited to embed tree-like graphs . It is hence natural to wonder : is there a connection between the two ? The Fisher geometry of Gaussians is hyperbolic . It turns out that there exists a striking connection ( Costa et al. , 2015 ) . Note that a 1D Gaussian N ( µ , σ2 ) can be represented as a point ( µ , σ ) in R×R∗+ . Then , the Fisher distance between two distributions relates to the hyperbolic distance in H2 : dF ( N ( µ , σ2 ) , N ( µ′ , σ′2 ) ) = √ 2dH2 ( ( µ/ √ 2 , σ ) , ( µ′/ √ 2 , σ′ ) ) . ( 4 ) For n-dimensional Gaussians with diagonal covariance matrices written Σ = diag ( σ ) 2 , it becomes : dF ( N ( µ , Σ ) , N ( µ′ , Σ′ ) ) = √√√√ n∑ i=1 2dH2 ( ( µi/ √ 2 , σi ) , ( µ′i/ √ 2 , σ′i ) ) 2 . ( 5 ) Hence there is a direct correspondence between diagonal Gaussians and the product space ( H2 ) n. 1diagonal or even spherical , for simplicity . Fisher distance , KL & Gaussian embeddings . The above paragraph lets us relate the WORD2GAUSS algorithm ( Vilnis & McCallum , 2015 ) to hyperbolic word embeddings . Although one could object that WORD2GAUSS is trained using a KL divergence , while hyperbolic embeddings relate to Gaussian distributions via the Fisher distance dF , let us remind that KL and dF define the same local geometry . Indeed , the KL is known to be related to dF , as its local approximation ( Jeffreys , 1946 ) . In short , if P ( θ + dθ ) and P ( θ ) denote two closeby probability distributions for a small dθ , then KL ( P ( θ + dθ ) ||P ( θ ) ) = ( 1/2 ) ∑ ij gijdθ idθj + O ( ‖dθ‖3 ) , where ( gij ) ij is the Fisher information metric , inducing dF . Riemannian optimization . A benefit of representing words in ( products of ) hyperbolic spaces , as opposed to ( diagonal ) Gaussian distributions , is that one can use recent Riemannian adaptive optimization tools such as RADAGRAD ( Bécigneul & Ganea , 2018 ) . Note that without this connection , it would be unclear how to define a variant of ADAGRAD ( Duchi et al. , 2011 ) intrinsic to the statistical manifold of Gaussians . Empirically , we indeed noticed better results using RADAGRAD , rather than simply Riemannian SGD ( Bonnabel , 2013 ) . Similarly , note that GLOVE trains with ADAGRAD .

Summary: This paper adapts the Glove word embedding (Pennington et al 2014) to a hyperbolic space given by the Poincare half-plane model.  The embedding objective function is given by equation (3), where h=cosh^2 so that it corresponds to a hyperbolic geometry. The author(s) showed that their hyperbolic version of Glove is better than the original Glove. Besides that,  based on (Costa et al 2015), the author provided theoretical insights on the connection between hyperbolic embeddings with Gaussian word embeddings. Besides, the author(s) proposed a measure called "delta-hyperbolicity", that is based on (Gromov 1987) to study the model selection problem of using hyperbolic embeddings vs. traditional Euclidean embeddings.