Unit Ball Model for Embedding Hierarchical Structures in the Complex Hyperbolic Space

1 INTRODUCTION . Representation learning of data with hierarchical structures is an important machine learning task with many applications , such as taxonomy induction ( Fu et al. , 2014 ) and hypernymy detection ( Shwartz et al. , 2016 ) . In recent years , the hyperbolic embeddings ( Nickel and Kiela , 2017 ; 2018 ) have been proposed to improve the traditional Euclidean embedding models . The constant negative curvature of the hyperbolic space produces a manifestation that the hyperbolic space can be regarded as a continuous approximation to trees ( Krioukov et al. , 2010 ) . The hyperbolic space is capable of embedding any finite tree while preserving the distances approximately ( Gromov , 1987 ) . However , most real-world hierarchical data do not belong to tree structures since they can have varying local structures while being tree-like globally . For example , the taxonomies such as WordNet ( Miller , 1995 ) and YAGO ( Suchanek et al. , 2007 ) contain many 1-N ( 1 child links to multiple parents ) cases and multitree structures ( Griggs et al. , 2012 ) , which are much more complicated than the tree structure . In consequence , the hyperbolic space which resembles tree metrics has limitations on capturing the general hierarchically structured data . To address the challenge , in this paper , we propose a new approach to learning the embeddings of hierarchically structured data . Specifically , we embed the hierarchical data into the unit ball model of the complex hyperbolic space . The unit ball model is a projective geometry based model to identify the complex hyperbolic space . One of the main differences between the complex and the real hyperbolic space is that the curvature is non-constant in the complex hyperbolic space . In practice , the variable negative curvature makes the complex hyperbolic space more flexible in handling varying structures while the tree-like properties are still retained . For empirical evaluation , we evaluate different geometrical embedding models on various hierarchically structured data , including synthetic graphs and real-world data . The experimental results demonstrate the advantages of our approach . In addition , we investigate two specific structures in which complex hyperbolic geometry shows outstanding performances , namely the multitree structure and 1-N structure , which are highly common and typical in real-world data . To summarize , our work has the following main contributions : 1 . We present a novel embedding approach based on the complex hyperbolic geometry to handle data with complicated and various hierarchical structures . To the best of our knowledge , our work is the first to propose complex hyperbolic embeddings . 2 . We introduce the embedding algorithm in the unit ball model of the complex hyperbolic space . We formulate the learning and Riemannian optimization in the unit ball model . 3 . We evaluate our approach with experiments on an extensive range of synthetic and real-world data and show the remarkable improvements of our approach . 2 RELATED WORK . Hyperbolic embeddings . Hyperbolic embedding methods have become the leading approach for representation learning of hierarchical structures . Nickel and Kiela ( 2017 ) learned the representations of hierarchical graphs in the Poincaré ball model of the hyperbolic space and outperformed the Euclidean embedding methods for taxonomies . The Poincaré embedding model was then improved by follow-up works on hyperbolic emebddings ( Ganea et al. , 2018a ; Nickel and Kiela , 2018 ) . These methods learned the hyperbolic embeddings by Riemannian optimization ( Bonnabel , 2013 ) , which was further improved by the Riemannian adaptive optimization ( Bécigneul and Ganea , 2019 ) . Another branch of study ( Sala et al. , 2018 ; Sonthalia and Gilbert , 2020 ) learned the hyperbolic embeddings through combinatorial construction . Instead of optimizing the soft-ranking loss by Riemannian optimization as in ( Nickel and Kiela , 2017 ; 2018 ) , the construction-based methods minimize the reconstruction distortion by combinatorial construction . However , both the optimizationbased and construction-based hyperbolic embeddings suffer from the limitation in hierarchical graphs with varying local structures . To tackle the challenge , Gu et al . ( 2019 ) extended the construction-based method by jointly learning the curvature and the embeddings of data in a product manifold . Although it can provide a better representation than a single space with constant curvature , it is impractical to search for the best manifold combination among enormous combinations for each new structure . Motivated by the promising results of previous works , extensions to the multi-relational graph hyperbolic embeddings ( Balazevic et al. , 2019 ; Chami et al. , 2020 ; Sun et al. , 2020 ) and hyperbolic neural networks ( Ganea et al. , 2018b ; Gülçehre et al. , 2019 ; Liu et al. , 2019 ; Chami et al. , 2019 ; Zhu et al. , 2020 ; Dai et al. , 2021a ; Shimizu et al. , 2021 ) were explored . Notably , ( Chami et al. , 2019 ; 2020 ) leverages the trainable curvature to compensate for the disparity between the actual data structures and the constant-curvature hyperbolic space , where each layer in the graph neural network or each relation in the multi-relational graph has its own curvature parameterization . Since we only focus on the single-relation graph embeddings and taxonomy embeddings in this work , we do not evaluate the multi-relational knowledge graph embedding models or the neural networks in our tasks . The hyperbolic learning also inspired other research tasks and applications , such as classification ( Cho et al. , 2019 ) , image reconstruction ( Skopek et al. , 2020 ) , text generation ( Dai et al. , 2021b ) , etc . Complex embeddings . The traditional knowledge graph embeddings were learned in the real Euclidean space ( Nickel et al. , 2011 ; Bordes et al. , 2013 ; Yang et al. , 2015 ) and were used for knowledge graph inference and reasoning . In recent years , several works suggested utilizing the complex Euclidean space for inferring more relation patterns , such as ComplEx ( Trouillon et al. , 2016 ) and RotatE ( Sun et al. , 2019 ) . These models have been demonstrated to be effective in knowledge graph embeddings . The success of the complex embeddings reveals the potential of the complex space and inspires us to explore the complex hyperbolic space . 3 PRELIMINARIES . 3.1 HYPERBOLIC GEOMETRY . Hyperbolic space1 is a homogeneous space with constant negative curvature.2 In the hyperbolic space HnR ( K ) of dimension n and curvature K , the volume of a ball grows exponentially with its radius ρ : vol ( BHnR ( K ) ( ρ ) ) ∼ e √ −K ( n−1 ) ρ . ( 1 ) 1In this paper , we use hyperbolic space to refer to real hyperbolic space and hyperbolic embeddings to refer to real hyperbolic embeddings for avoiding wordiness . 2In this paper , curvature refers to the sectional curvature . Please see Appendix A.1 for definition . Contrastively , in the Euclidean space En , the curvature is 0 and the volume of a ball grows polynomially with its radius : vol ( BEn ( ρ ) ) = πn/2 Γ ( n/2 ) ρn ∼ ρn . ( 2 ) The exponential volume growth rate enables the hyperbolic space to have powerful representation capability for tree structures since the number of nodes grows exponentially with the depth in a tree , while the Euclidean space is too flat and narrow to embed trees . 3.2 COMPLEX HYPERBOLIC GEOMETRY . Complex hyperbolic space is a homogeneous space of variable negative curvature . Its ambient Hermitian vector space Cn,1 is the complex Euclidean space Cn+1 endowed with some Hermitian form 〈〈z , w〉〉 , where z , w ∈ Cn+1 . Then the Hermitian space Cn,1 can be divided into three subsets : V− = { z ∈ Cn,1|〈〈z , z〉〉 < 0 } , V0 = { z ∈ Cn,1 − { 0 } |〈〈z , z〉〉 = 0 } , and V+ = { z ∈ Cn,1|〈〈z , z〉〉 > 0 } . Let P be a projection map P : Cn,1 − { zn+1 = 0 } → Cn , i.e. , P : z1 ... zn+1 7→ z1/zn+1 ... zn/zn+1 , where zn+1 6= 0 . ( 3 ) Then the complex hyperbolic space HnC and its boundary ∂HnC are defined using the projectivization : HnC = PV− , ∂HnC = PV0 . ( 4 ) The curvature of the complex hyperbolic space is summarized by ( Goldman , 1999 ) as follows : Theorem 1 . The curvature is not constant in HnC . It is pinched between −1 ( in the directions of complex projective lines ) and −1/4 ( in the directions of totally real planes ) . We leave the full proof in Appendix B . The non-constant curvature , which we expect to be favorable for embedding various hierarchical structures , is one of the main differences between HnC and HnR . The complex hyperbolic space also has the tree-like exponential volume growth property . The volume of a ball with radius ρ in HnC is given by vol ( BHnC ( ρ ) ) = 8nσ2n−1 2n sinh2n ( ρ/2 ) ∼ enρ , ( 5 ) where σ2n−1 = 2πn/n ! is the Euclidean volume of the unit sphere S2n−1 ∈ Cn . From the properties of the complex hyperbolic geometry , we expect that the complex hyperbolic space can naturally handle data with diverse local structures in virtue of the variable curvature as presented in Theorem 1 while preserving the tree-like properties as shown in Eq . ( 5 ) . From this section , we see that complex hyperbolic geometry and hyperbolic geometry are typically of different characteristics . The n-dimensional ( n-d ) complex hyperbolic space is not simply the 2n-d hyperbolic space or the product of two n-d hyperbolic spaces . This implies that our complex hyperbolic embedding model is intrinsically different from the hyperbolic embedding methods ( Nickel and Kiela , 2017 ; 2018 ) or the product manifold embeddings ( Gu et al. , 2019 ) . 4 UNIT BALL EMBEDDINGS . We propose to embed the hierarchically structured data into the unit ball model of the complex hyperbolic space . In this section , we introduce our approach in detail . 4.1 THE UNIT BALL MODEL . The unit ball model is one model used to identify the complex hyperbolic space , which can be derived via the projective geometry ( Goldman , 1999 ) . We now provide the necessary derivation . Take the Hermitian form of Cn,1 in Section 3.2 to be a standard Hermitian form : 〈〈z , w〉〉 = z1w1 + · · ·+ znwn − zn+1wn+1 , ( 6 ) where w is the conjugate of w. Take zn+1 = 1 in the projection map P in Eq . ( 3 ) . Then from Eq . ( 4 ) , we can derive the formula of the unit ball model : BnC = P ( { z ∈ Cn,1|〈〈z , z〉〉 < 0 } ) = { ( z1 , · · · , zn , 1 ) ||z1|2 + · · ·+ |zn|2 < 1 } . ( 7 ) The metric on BnC is Bergman metric , which takes the formula below in 2-d case : ds2 = −4 〈〈z , z〉〉2 det [ 〈〈z , z〉〉 〈〈dz , z〉〉 〈〈z , dz〉〉 〈〈dz , dz〉〉 ] . ( 8 ) The distance function on BnC is given by dBnC ( z , w ) = arcosh ( 2 〈〈z , w〉〉〈〈w , z〉〉 〈〈z , z〉〉〈〈w , w〉〉 − 1 ) , ( 9 ) Note that there are other choices of the Hermitian form 〈〈z , w〉〉 , which corresponds to other models of complex hyperbolic geometry , such as the Siegel domain model . We choose the unit ball model for the relatively simple formula as well as convenient computations of the metric and distance function .

Summary: The paper introduces an extension of real hyperbolic embeddings to the complex hyperbolic space [A]. The exploited geometry is an extension of the Poincaré ball that contains complex vectors (instead of real vectors) whose norm is smaller than 1. The resulting manifold is of nonconstant negative curvature, which the authors expect to be favorable for embedding various hierarchical structures.  Following the optimization framework of [B] and since the complex hyperbolic space is a Riemannian manifold, Section 4.3 presents a standard Riemannian optimization framework to learn nonparametric embeddings. The proposed manifold shows (slightly) superior results compared to real hyperbolic embeddings proposed in [B,C] in the graph reconstruction and link prediction tasks.