Learning Mixed-Curvature Representations in Product Spaces

The quality of the representations achieved by embeddings is determined by how well the geometry of the embedding space matches the structure of the data . Euclidean space has been the workhorse for embeddings ; recently hyperbolic and spherical spaces have gained popularity due to their ability to better embed new types of structured data—such as hierarchical data—but most data is not structured so uniformly . We address this problem by proposing learning embeddings in a product manifold combining multiple copies of these model spaces ( spherical , hyperbolic , Euclidean ) , providing a space of heterogeneous curvature suitable for a wide variety of structures . We introduce a heuristic to estimate the sectional curvature of graph data and directly determine an appropriate signature—the number of component spaces and their dimensions—of the product manifold . Empirically , we jointly learn the curvature and the embedding in the product space via Riemannian optimization . We discuss how to define and compute intrinsic quantities such as means—a challenging notion for product manifolds—and provably learnable optimization functions . On a range of datasets and reconstruction tasks , our product space embeddings outperform single Euclidean or hyperbolic spaces used in previous works , reducing distortion by 32.55 % on a Facebook social network dataset . We learn word embeddings and find that a product of hyperbolic spaces in 50 dimensions consistently improves on baseline Euclidean and hyperbolic embeddings , by 2.6 points in Spearman rank correlation on similarity tasks and 3.4 points on analogy accuracy . 1 INTRODUCTION . With four decades of use , Euclidean space is the venerable elder of embedding spaces . Recently , non-Euclidean spaces—hyperbolic ( Nickel & Kiela , 2017 ; Sala et al. , 2018 ) and spherical ( Wilson et al. , 2014 ; Liu et al. , 2017 ) —have gained attention by providing better representations for certain types of structured data . The resulting embeddings offer better reconstruction metrics : higher mean average precision ( mAP ) and lower distortion compared to their Euclidean counterparts . These three spaces are the model spaces of constant curvature ( Lee , 1997 ) , and this improvement in representation fidelity arises from the correspondence between the structure of the data ( hierarchical , cyclical ) and the geometry of non-Euclidean space ( hyperbolic : negatively curved , spherical : positively curved ) . The notion of curvature plays the key role . To improve representations for a variety of types of data—beyond hierarchical or cyclical—we seek spaces with heterogeneous curvature . The motivation for such mixed spaces is intuitive : our data may have complicated , varying structure , in some regions tree-like , in others cyclical , and we seek the best of all worlds . We expect mixed spaces to match the geometry of the data and thus provide higher quality representations . However , to employ these spaces , we face several key obstacles . We must perform a challenging manifold optimization to learn both the curvature and the embedding . Afterwards , we also wish to operate on the embedded points . For example , analogy operations for word embeddings in Euclidean vector space ( e.g. , a− b+ c ) must be lifted to manifolds . We propose embedding into product spaces in which each component has constant curvature . As we show , this allows us to capture a wider range of curvatures than traditional embeddings , while retaining the ability to globally optimize and operate on the resulting embeddings . Specifically , we form a Riemannian product manifold combining hyperbolic , spherical , and Euclidean components and equip it with a decomposable Riemannian metric . While each component space in the product has constant curvature ( positive for spherical , negative for hyperbolic , and zero for Euclidean ) , the resulting mixed space has non-constant curvature . However , selecting appropriate curvatures for the embedding space is a potential challenge . We directly learn the curvature for each component space along with the embedding ( via Riemannian optimization ) , recovering the correct curvature , and thus the matching geometry , directly from data . We show empirically that we can indeed recover non-uniform curvatures and improve performance on reconstruction metrics . Another technical challenge is to select the underlying number of components and dimensions of the product space ; we call this the signature . This concept is vacuous in Euclidean space : the product of Er1 , . . . , Ern is identical to the single space Er1+ ... +rn . However , this is not the case with spherical and hyperbolic spaces . For example , the product of the spherical space S1 ( the circle ) with itself is the torus S1 × S1 , which is topologically distinct from the sphere S2 . We address this challenge by introducing a theory-guided heuristic estimator for the signature . We do so by matching an empirical notion of discrete curvature in our data with the theoretical distribution of the sectional curvature , a fine-grained measure of curvature on Riemannian manifolds that is amenable to analysis in products . We verify that this approach recovers the correct signature on reconstruction tasks . Standard techniques such as PCA require centering so that the embedded directions capture components of variation . Centering in turn needs an appropriate generalization of the mean . We develop a formulation of mean for embedded points that exploits the decomposability of the distance and has theoretical guarantees . For T = { p1 , . . . , pn } in a manifoldM with dimension r , the mean is µ ( T ) : = argminp ∑ i d 2 M ( p , pi ) . We give a global existence result : under symmetry conditions on the distribution of the points in T on the spherical components , gradient descent recovers µ ( T ) with error ε in time O ( nr log ε−1 ) . We demonstrate the advantages of product space embeddings through a variety of experiments ; products are at least as good as single spaces , but can offer significant improvements when applied to structures not suitable for single spaces . We measure reconstruction quality ( via mAP and distortion ) for synthetic and real datasets over various allocations of embedding spaces . We observe a 32.55 % improvement in distortion versus any single space on a Facebook social network graph . Beyond reconstruction , we apply product spaces to skip-gram word embeddings , a popular technique with numerous downstream applications , which crucially require the use of the manifold structure . We find that products of hyperbolic spaces improve performance on benchmark evaluations—suggesting that words form multiple smaller hierarchies rather than one larger one . We see an improvement of 3.4 points over baseline single spaces on the Google word analogy benchmark and of 2.6 points in Spearman rank correlation on a word similarity task using the WS-353 corpus . Our results and initial exploration suggest that mixed product spaces are a promising area for future study . 2 PRELIMINARIES & BACKGROUND . Embeddings For metric spaces1 U , V equipped with distances dU , dV , an embedding is a mapping f : U → V . The quality of an embedding is measured by various fidelity measures . A standard measure is average distortion Davg . The distortion of a pair of points a , b is ( |dV ( f ( a ) , f ( b ) ) − dU ( a , b ) | ) /dU ( a , b ) , and Davg is the average over all pairs of points . Distortion is a global metric ; it considers the explicit value of all distances . At the other end of the global-local spectrum of fidelity measures is mean average precision ( mAP ) , which applies to unweighted graphs . Let G = ( V , E ) be a graph and node a ∈ V have neighborhood Na = { b1 , . . . , bdeg ( a ) } , where deg ( a ) is the degree of a . In the embedding f , defineRa , bi to be the smallest ball around f ( a ) that contains bi ( that is , Ra , bi is the smallest set of nearest points required to retrieve the ith neighbor of a in f ) . Then , mAP ( f ) = 1|V | ∑ a∈V 1 deg ( a ) ∑|Na| i=1 |Na∩Ra , bi |/|Ra , bi | . Note that mAP does not track explicit distances ; it is a ranking-based measure for local neighborhoods . Observe that mAP ( f ) ≤ 1 ( higher is better ) while davg ≥ 0 ( lower is better ) . Riemannian Manifolds We briefly review some notions from manifolds and Riemannian geometry . A more in-depth treatment can be found in standard texts ( Lee , 2012 ; do Carmo , 1992 ) . Let M be a smooth manifold , p ∈ M be a point , and TpM be the tangent space to the point p. If M is equipped with a Riemannian metric g , then the pair ( M , g ) is called a Riemannian manifold . The shortest-distance paths on manifolds are called geodesics . To compute distance functions on a Riemannian manifold , the metric tensor g is integrated along the geodesic . This is a smoothly varying function ( in p ) g : TpM × TpM → R that induces geometric notions such as length and angle by defining an inner product on the tangent space . For example , the norm of v ∈ TpM is defined as ‖v‖g : = gp ( v , v ) 1 2 . In Euclidean space Rd , each tangent space TpRd is canonically identified with Rd , and the metric tensor gE is simply the normal inner product . Product Manifolds Consider a sequence of smooth manifolds M1 , M2 , . . . , Mk . The product manifold is defined as the Cartesian product M = M1 ×M2 × . . . ×Mk . Notationally , we write points p ∈ M through their coordinates p = ( p1 , . . . , pk ) : pi ∈ Mi , and similarly a tangent vector v ∈ TpM can be written ( v1 , . . . , vk ) : vi ∈ TpiMi . If the Mi are equipped with metric tensor gi , then the product M is also Riemannian with metric tensor g ( u , v ) = ∑k i=1 gi ( ui , vi ) . That is , the product metric decomposes into the sum of the constituent metrics . Geodesics and Distances Optimization on manifolds requires a notion of taking a step . This step can be performed in the tangent space and transferred to the manifold via the exponential map Expp : TpM → M . In a product manifold P , for tangent vectors v = ( v1 , . . . , vk ) at p = ( p1 , . . . , pk ) ∈M , the exponential map simply decomposes , as do squared distances ( Ficken , 1939 ; Turaga & Srivastava , 2016 ) : Expp ( v ) = ( Expp1 ( v1 ) , . . . , Exppk ( vk ) ) , d 2 P ( x , y ) = k∑ i=1 d2i ( xi , yi ) . ( 1 ) In other words , the shortest path between points in the product travels along the shortest paths in each component simultaneously . Note the analogy to Euclidean products Rd ≡ ( R1 ) d. Hyperbolic and Spherical Model We use the hyperboloid model of hyperbolic space , with points in Rd+1 . Let J ∈ R ( d+1 ) × ( d+1 ) be the diagonal matrix with J00 = −1 and Jii = 1 : i > 0 . For p , q ∈ Rd+1 , the Minkowski inner product is 〈p , q〉∗ : = pTJq = −p0q0 + p1q1 + . . . + pdqd , and the corresponding norm is ‖p‖∗ = 〈p , p〉 1 2 ∗ . For any K > 0 , the hyperboloid HdK is defined on the 1In this paper , we use the language of graphs ; note that any discrete metric space can be identified with a weighted graph , and all of our algorithms operate on weighted graphs . subset { p ∈ Rd+1 : ‖p‖∗ = −K1/2 , p0 > 0 } . When the subscript K is omitted , it is taken to be 1 . The hyperbolic distance on Hd is dH ( p , q ) = acosh ( −〈p , q〉∗ ) . Similarly , spherical space SdK is most easily defined when embedded in Rd+1 . The manifold is defined on the subset { p ∈ Rd+1 : ‖p‖2 = K1/2 } , with metric gS induced by the Euclidean metric on Rd+1 . The spherical distance on Sd is dS ( p , q ) = arccos ( 〈p , q〉 ) .

Summary: This paper proposes a new method to embed a graph onto a product of spherical/Euclidean/hyperbolic manifolds. The key is to use sectional curvature estimations to determine proper signature, i.e., all component manifolds, and then optimize over these manifolds. The results are validated on various synthetic and real graphs. The proposed idea is new, nontrivial, and is well supported by experimental evidence.