Unit Ball Model for Embedding Hierarchical Structures in the Complex Hyperbolic Space

1 INTRODUCTION . Representation learning of data with hierarchical structures is an important machine learning task with many applications , such as taxonomy induction ( Fu et al. , 2014 ) and hypernymy detection ( Shwartz et al. , 2016 ) . In recent years , the hyperbolic embeddings ( Nickel and Kiela , 2017 ; 2018 ) have been proposed to improve the traditional Euclidean embedding models . The constant negative curvature of the hyperbolic space produces a manifestation that the hyperbolic space can be regarded as a continuous approximation to trees ( Krioukov et al. , 2010 ) . The hyperbolic space is capable of embedding any finite tree while preserving the distances approximately ( Gromov , 1987 ) . However , most real-world hierarchical data do not belong to tree structures since they can have varying local structures while being tree-like globally . For example , the taxonomies such as WordNet ( Miller , 1995 ) and YAGO ( Suchanek et al. , 2007 ) contain many 1-N ( 1 child links to multiple parents ) cases and multitree structures ( Griggs et al. , 2012 ) , which are much more complicated than the tree structure . In consequence , the hyperbolic space which resembles tree metrics has limitations on capturing the general hierarchically structured data . To address the challenge , in this paper , we propose a new approach to learning the embeddings of hierarchically structured data . Specifically , we embed the hierarchical data into the unit ball model of the complex hyperbolic space . The unit ball model is a projective geometry based model to identify the complex hyperbolic space . One of the main differences between the complex and the real hyperbolic space is that the curvature is non-constant in the complex hyperbolic space . In practice , the variable negative curvature makes the complex hyperbolic space more flexible in handling varying structures while the tree-like properties are still retained . For empirical evaluation , we evaluate different geometrical embedding models on various hierarchically structured data , including synthetic graphs and real-world data . The experimental results demonstrate the advantages of our approach . In addition , we investigate two specific structures in which complex hyperbolic geometry shows outstanding performances , namely the multitree structure and 1-N structure , which are highly common and typical in real-world data . To summarize , our work has the following main contributions : 1 . We present a novel embedding approach based on the complex hyperbolic geometry to handle data with complicated and various hierarchical structures . To the best of our knowledge , our work is the first to propose complex hyperbolic embeddings . 2 . We introduce the embedding algorithm in the unit ball model of the complex hyperbolic space . We formulate the learning and Riemannian optimization in the unit ball model . 3 . We evaluate our approach with experiments on an extensive range of synthetic and real-world data and show the remarkable improvements of our approach . 2 RELATED WORK . Hyperbolic embeddings . Hyperbolic embedding methods have become the leading approach for representation learning of hierarchical structures . Nickel and Kiela ( 2017 ) learned the representations of hierarchical graphs in the Poincaré ball model of the hyperbolic space and outperformed the Euclidean embedding methods for taxonomies . The Poincaré embedding model was then improved by follow-up works on hyperbolic emebddings ( Ganea et al. , 2018a ; Nickel and Kiela , 2018 ) . These methods learned the hyperbolic embeddings by Riemannian optimization ( Bonnabel , 2013 ) , which was further improved by the Riemannian adaptive optimization ( Bécigneul and Ganea , 2019 ) . Another branch of study ( Sala et al. , 2018 ; Sonthalia and Gilbert , 2020 ) learned the hyperbolic embeddings through combinatorial construction . Instead of optimizing the soft-ranking loss by Riemannian optimization as in ( Nickel and Kiela , 2017 ; 2018 ) , the construction-based methods minimize the reconstruction distortion by combinatorial construction . However , both the optimizationbased and construction-based hyperbolic embeddings suffer from the limitation in hierarchical graphs with varying local structures . To tackle the challenge , Gu et al . ( 2019 ) extended the construction-based method by jointly learning the curvature and the embeddings of data in a product manifold . Although it can provide a better representation than a single space with constant curvature , it is impractical to search for the best manifold combination among enormous combinations for each new structure . Motivated by the promising results of previous works , extensions to the multi-relational graph hyperbolic embeddings ( Balazevic et al. , 2019 ; Chami et al. , 2020 ; Sun et al. , 2020 ) and hyperbolic neural networks ( Ganea et al. , 2018b ; Gülçehre et al. , 2019 ; Liu et al. , 2019 ; Chami et al. , 2019 ; Zhu et al. , 2020 ; Dai et al. , 2021a ; Shimizu et al. , 2021 ) were explored . Notably , ( Chami et al. , 2019 ; 2020 ) leverages the trainable curvature to compensate for the disparity between the actual data structures and the constant-curvature hyperbolic space , where each layer in the graph neural network or each relation in the multi-relational graph has its own curvature parameterization . Since we only focus on the single-relation graph embeddings and taxonomy embeddings in this work , we do not evaluate the multi-relational knowledge graph embedding models or the neural networks in our tasks . The hyperbolic learning also inspired other research tasks and applications , such as classification ( Cho et al. , 2019 ) , image reconstruction ( Skopek et al. , 2020 ) , text generation ( Dai et al. , 2021b ) , etc . Complex embeddings . The traditional knowledge graph embeddings were learned in the real Euclidean space ( Nickel et al. , 2011 ; Bordes et al. , 2013 ; Yang et al. , 2015 ) and were used for knowledge graph inference and reasoning . In recent years , several works suggested utilizing the complex Euclidean space for inferring more relation patterns , such as ComplEx ( Trouillon et al. , 2016 ) and RotatE ( Sun et al. , 2019 ) . These models have been demonstrated to be effective in knowledge graph embeddings . The success of the complex embeddings reveals the potential of the complex space and inspires us to explore the complex hyperbolic space . 3 PRELIMINARIES . 3.1 HYPERBOLIC GEOMETRY . Hyperbolic space1 is a homogeneous space with constant negative curvature.2 In the hyperbolic space HnR ( K ) of dimension n and curvature K , the volume of a ball grows exponentially with its radius ρ : vol ( BHnR ( K ) ( ρ ) ) ∼ e √ −K ( n−1 ) ρ . ( 1 ) 1In this paper , we use hyperbolic space to refer to real hyperbolic space and hyperbolic embeddings to refer to real hyperbolic embeddings for avoiding wordiness . 2In this paper , curvature refers to the sectional curvature . Please see Appendix A.1 for definition . Contrastively , in the Euclidean space En , the curvature is 0 and the volume of a ball grows polynomially with its radius : vol ( BEn ( ρ ) ) = πn/2 Γ ( n/2 ) ρn ∼ ρn . ( 2 ) The exponential volume growth rate enables the hyperbolic space to have powerful representation capability for tree structures since the number of nodes grows exponentially with the depth in a tree , while the Euclidean space is too flat and narrow to embed trees . 3.2 COMPLEX HYPERBOLIC GEOMETRY . Complex hyperbolic space is a homogeneous space of variable negative curvature . Its ambient Hermitian vector space Cn,1 is the complex Euclidean space Cn+1 endowed with some Hermitian form 〈〈z , w〉〉 , where z , w ∈ Cn+1 . Then the Hermitian space Cn,1 can be divided into three subsets : V− = { z ∈ Cn,1|〈〈z , z〉〉 < 0 } , V0 = { z ∈ Cn,1 − { 0 } |〈〈z , z〉〉 = 0 } , and V+ = { z ∈ Cn,1|〈〈z , z〉〉 > 0 } . Let P be a projection map P : Cn,1 − { zn+1 = 0 } → Cn , i.e. , P : z1 ... zn+1 7→ z1/zn+1 ... zn/zn+1 , where zn+1 6= 0 . ( 3 ) Then the complex hyperbolic space HnC and its boundary ∂HnC are defined using the projectivization : HnC = PV− , ∂HnC = PV0 . ( 4 ) The curvature of the complex hyperbolic space is summarized by ( Goldman , 1999 ) as follows : Theorem 1 . The curvature is not constant in HnC . It is pinched between −1 ( in the directions of complex projective lines ) and −1/4 ( in the directions of totally real planes ) . We leave the full proof in Appendix B . The non-constant curvature , which we expect to be favorable for embedding various hierarchical structures , is one of the main differences between HnC and HnR . The complex hyperbolic space also has the tree-like exponential volume growth property . The volume of a ball with radius ρ in HnC is given by vol ( BHnC ( ρ ) ) = 8nσ2n−1 2n sinh2n ( ρ/2 ) ∼ enρ , ( 5 ) where σ2n−1 = 2πn/n ! is the Euclidean volume of the unit sphere S2n−1 ∈ Cn . From the properties of the complex hyperbolic geometry , we expect that the complex hyperbolic space can naturally handle data with diverse local structures in virtue of the variable curvature as presented in Theorem 1 while preserving the tree-like properties as shown in Eq . ( 5 ) . From this section , we see that complex hyperbolic geometry and hyperbolic geometry are typically of different characteristics . The n-dimensional ( n-d ) complex hyperbolic space is not simply the 2n-d hyperbolic space or the product of two n-d hyperbolic spaces . This implies that our complex hyperbolic embedding model is intrinsically different from the hyperbolic embedding methods ( Nickel and Kiela , 2017 ; 2018 ) or the product manifold embeddings ( Gu et al. , 2019 ) . 4 UNIT BALL EMBEDDINGS . We propose to embed the hierarchically structured data into the unit ball model of the complex hyperbolic space . In this section , we introduce our approach in detail . 4.1 THE UNIT BALL MODEL . The unit ball model is one model used to identify the complex hyperbolic space , which can be derived via the projective geometry ( Goldman , 1999 ) . We now provide the necessary derivation . Take the Hermitian form of Cn,1 in Section 3.2 to be a standard Hermitian form : 〈〈z , w〉〉 = z1w1 + · · ·+ znwn − zn+1wn+1 , ( 6 ) where w is the conjugate of w. Take zn+1 = 1 in the projection map P in Eq . ( 3 ) . Then from Eq . ( 4 ) , we can derive the formula of the unit ball model : BnC = P ( { z ∈ Cn,1|〈〈z , z〉〉 < 0 } ) = { ( z1 , · · · , zn , 1 ) ||z1|2 + · · ·+ |zn|2 < 1 } . ( 7 ) The metric on BnC is Bergman metric , which takes the formula below in 2-d case : ds2 = −4 〈〈z , z〉〉2 det [ 〈〈z , z〉〉 〈〈dz , z〉〉 〈〈z , dz〉〉 〈〈dz , dz〉〉 ] . ( 8 ) The distance function on BnC is given by dBnC ( z , w ) = arcosh ( 2 〈〈z , w〉〉〈〈w , z〉〉 〈〈z , z〉〉〈〈w , w〉〉 − 1 ) , ( 9 ) Note that there are other choices of the Hermitian form 〈〈z , w〉〉 , which corresponds to other models of complex hyperbolic geometry , such as the Siegel domain model . We choose the unit ball model for the relatively simple formula as well as convenient computations of the metric and distance function .

Summary: This paper proposes a method to learn embeddings in complex hyperbolic spaces. In contrast with real hyperbolic spaces, complex hyperbolic spaces have a variable negative curvature, which can handle more flexible graph structures. Experiments show that complex hyperbolic embeddings outperform real hyperbolic embeddings on synthetic and real-world graphs.