Capacity and Bias of Learned Geometric Embeddings for Directed Graphs

1 Introduction . Embedding-based methods are a cornerstone of modern approaches to modeling graph data . Geometric embeddings , which represent elements using objects with richer geometric structure ( such as boxes [ 60 ] , cones [ 27 , 58 ] , discs [ 55 ] , densities [ 2 , 59 ] , and elements of hyperbolic space [ 39 , 40 ] ) have recently been proposed as a powerful alternative to standard vector embedding models for representing asymmetric relationships . Modeling such asymmetric relationships is central to discovering embedded representations of directed graphs , such as DAG-structured taxonomies , where geometric embeddings have been shown empirically to be more effective [ 39 , 58 , 60 ] . The success of geometric embeddings largely rests on exploiting inherent biases intrinsic to the particular geometry chosen . The negative curvature of hyperbolic space , for example , provides an inductive bias which is believed to better model the exponential increase of nodes in trees [ 39 , 61 , 62 ] . Region-based representations , on the other hand , model directional edges via containment relationships [ 58 , 60 ] . As containment is , itself , transitive , these models would seemingly have a natural bias toward representing transitive relationships . While previous work has empirically tested ⇤ Equal Contribution . † Now at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the generalization capability of these region-based geometric representations on directed graphs with missing edges [ 17 , 27 , 30 , 58 , 60 ] an understanding of their representational capability , both theoretically and experimentally , is lacking . There are several notable open questions regarding geometric embeddings . First , previous work has observed that the performance benefits of geometric embeddings are most strongly observed in low dimensions , and are often diminished or negated by vector models in higher dimensions , raising the question of whether there exists a geometric embedding model which can utilize higher dimensions more effectively . Second , previous work has provided theoretical results for vector [ 4 , 15 ] and hyperbolic [ 48 , 49 ] models , but has not addressed what region-based embeddings , such as order and box embeddings , can represent , theoretically . Third , there is not an agreed upon convention for how to empirically analyze the bias and representational capacity of models without confounding factors such as optimization , hyperparameters , and generalization performance . In this paper , we address each of these open questions . We introduce a novel box embedding model , extending the recently proposed Gumbel-box process [ 17 ] to learn from data a parametric model representing a continuous relaxation of box intersections . In this way , we can effectively rely on the structure of the graph to guide the fitting of the embedding model . Our model can trade-off between smoother , vector-like settings and more discrete , region-like settings , providing better performance across both lower and higher dimensionalities in a single model . We provide theoretical analysis of this approach as well as region-based approaches more generally . We lastly design a rigorous evaluation framework , which uses Bayesian hyperparameter optimization to fairly compare representational capacity independently from optimization details of each approach . This empirical framework allows us to characterize properties of graphs for which each method performs well . In summary , our contributions are as follows : Methodological Contributions ( §3 ) We introduce a novel continuous relaxation of probabilistic box embeddings , where each node has an associated trained variance or temperature . Theoretical Contributions ( §4 ) : We prove • Order Embeddings may produce cycles , even when pruning reverse edges ( Proposition 2 ) • Probabilistic Order Embeddings ( or Order Embeddings under ` 1 energy ) always produce acyclic graphs when pruning reverse edges ( Theorem 1 ) While it is known [ 60 ] that box embeddings also produce acyclic graphs when reverse edges are removed , we significantly strengthen this by proving • Any DAG can be represented by Probabilistic Box Embeddings in O ( ( + 2 ) log |V | ) dimensions , where is the maximum node degree , and |V | the number of nodes ( Theorem 2 ) • This can be accomplished using O ( D ( +2 ) log2 |V | ) bits of precision , where D is the max depth ( Proposition 3 ) Empirical Highlights ( §6 ) : We conduct a suite of experiments on synthetic and real-world graphs.1 We compare various vector , hyperbolic , cone , and box representations over a range of dimensions , using extensive Bayesian hyperparameter tuning , and find that • Geometric embedding models perform better in low dimensions • Our model outperforms all other geometric embeddings in high dimensions • Our model always outperforms the state-of-the-art box embeddings when using equal dimensions • Hyperbolic models do particularly well on trees , but struggle with transitivity • Region-based embeddings perform particularly well on transitively-closed graphs 2 Background : Embedding Directed Graphs . Given a simple2 directed graph G with nodes and edges ( V , E ) we seek to represent the adjacency matrix , that is , a function f : V ⇥ V ! { 0 , 1 } where f ( i , j ) = 1 if ( i , j ) 2 E and 0 otherwise . We parameterize the nodes , defining a function : V ! Rd , and define an energy function E : Rd ⇥ Rd ! R+ , which models the negative log probability of edge existence . We can view 1Our code and data are available at https : //github.com/iesl/geometric_graph_embedding . 2A simple directed graph has no multiple edges or self-loops , i.e . the adjacency matrix contains only 0s and 1s , with 0s on the diagonal . We remove self-loops from the learned representations . the probabilities provided as a weighted graph , however in practice it is often necessary to make a hard decision on edge existence , which is done by choosing a ( global ) threshold ⌧ and binarizing the output , yielding an adjacency matrix Ai , j = E ( i , j ) < ⌧ . 2.1 Vectors . A straightforward approach to this problem would be to parameterize each node using a vector , however typical score functions for vectors ( e.g. , inner-product , distance ) are symmetric , and thus would only be capable of embedding undirected graphs . Given a directed graph G , one can always first form an undirected graph G0 = ( V 0 , E0 ) with 2|V | nodes , where for each node i 2 V we associate two nodes iin , iout 2 V 0 , and for each edge ( i , j ) 2 E we have an edge ( iout , jin ) 2 E0 . We could embed this graph using symmetric score functions on vectors , for example E ( iout , jin ) : = log ( ( ( iout ) · ( jin ) ) ) . ( 1 ) Of course this can also be retracted to the original directed graph G , namely given a parameterization : V ! R2d we can define an energy function ESIM ( i , j ) : = log ✓ ✓ ( i ) T 0 I 0 0 ( j ) ◆◆ . ( 2 ) More generally , for ( learned ) weights W 2 R2d⇥2d we can define a bilinear score function [ 37 ] , EBILINEAR ( i , j ) : = log ( i ) TW ( j ) . ( 3 ) 2.2 Complex Vectors . The models discussed in §2.1 are often considered Euclidean vector embeddings , as they make use of the inner product and linear transformations on Euclidean space . Alternative spaces for representing nodes have been introduced , most notably complex space Cd [ 57 ] , where the parametrization can be thought of as : V ! Cd with energy function E ( i , j ) : = log ⇣ ⇣ Re ⇣Pd k=1 ( i ) kwk ( j ) k ⌘⌘⌘ ( 4 ) where w 2 Cd and Re projects to the real part . In order to compare this method to previous models , we can also think of this as parameterized by : V ! R2d , where ( i ) k : = Re ( i ) k and ( i ) d+k : = Im ( j ) k , in which case the energy function defined as ECOMPLEX ( i , j ) : = log ( i ) TW ( j ) with W : = diag ( Rew ) diag ( Imw ) diag ( Imw ) diag ( Rew ) ( 5 ) is equivalent to ( 4 ) . 2.3 Hyperbolic Vectors . While the use of bilinear products or complex vectors provide a naturally asymmetric scoring function , one might reasonably ask if there are spaces whose geometry lends itself to representing graphs . It is known , for example , that ( undirected ) trees can be embedded with arbitrarily low distortion into the Poincaré disk [ 49 ] , which is not possible in Euclidean space even using an unbounded number of dimensions [ 31 ] . These statements are related to undirected graphs , and while the in/out parameterization approach taken in the SIM model could be applied here we would lose any geometric benefit , as the undirected graph G0 corresponding to a tree is not , itself , a tree . Nickel and Kiela [ 39 ] suggest that edge direction can be decided by use of the Euclidean norm , and define a score function for an edge ( u , v ) as score ( u , v ) = ( 1 + ↵ ( kuk kvk ) ) d ( u , v ) , ( 6 ) where d is the Poincaré distance and ↵ is a hyperparameter indicating the strength of the penalty for edge directions which violate the order of their norms . An inspection of this score function shows that it is very sensitive to the choice of ↵ , which makes training unstable ( see Appendix A ) . Poincaré distance , in general , suffers from numeric stability issues which have been addressed in more recent work . Nickel and Kiela [ 40 ] consider the Lorentz model of hyperbolic space on the hyperboloid , and Law et al . [ 28 ] demonstrates that squared Lorentzian distance on the hyperboloid has similar geometric properties to Poincaré distance , while also providing numerically stable gradients . Using the notation of Law et al . [ 28 ] , the hyperboloid is defined as Hd , : = { a = ( a0 , . . . , ad ) 2 R+⇥Rd : kak2L = } , ( 7 ) where > 0 , kak2L = ha , aiL is the squared Lorentzian norm of a , and ha , biL is the Lorentzian inner product , and the squared Lorentzian distance d2L : Hd , ⇥Hd , ! R+ is defined as d2L ( a , b ) : = 2 2ha , biL , ha , biL : = a0b0 + Pd i=1 aibi . ( 8 ) Note that ha , biL for a , b 2 Hd , , with equality if and only if a = b . We also make use of the bijection f : Rd ! Hd , , where f ( x ) : = ⇣p kxkd + , x1 , . . . , xd ⌘ 2 Hd , . Given some parameterization : V ! Rd , we define a new hyperbolic energy function : EHYPERBOLIC ( i , j ) : = d 2 L ( f ( i ) , f ( j ) ) + ↵ log ( 1 + exp ( k ( i ) k k ( j ) k ) ) , ( 9 ) where the second term serves as a penalty on Euclidean norms to provide a measure of edge direction , i.e. , edges should go from nodes with smaller Euclidean norm to larger . ( See Appendix Figure 3 )

Summary: This work provides an empirical and theoretical foundation for the analysis of geometric representations of directed graphs. The paper introduces a novel box embedding model, extending the recent Gumbel-box process to learn a parametric model representing a continuous relaxation of box intersections. The model can trade-off between smoother, vector-like settings and more discrete, region-like settings, and provides better performance across a range of dimensionalities.  Directed graphs have received considerably less attention in the representation learning community than undirected graphs, making this paper particularly appealing.