Capacity and Bias of Learned Geometric Embeddings for Directed Graphs

1 Introduction . Embedding-based methods are a cornerstone of modern approaches to modeling graph data . Geometric embeddings , which represent elements using objects with richer geometric structure ( such as boxes [ 60 ] , cones [ 27 , 58 ] , discs [ 55 ] , densities [ 2 , 59 ] , and elements of hyperbolic space [ 39 , 40 ] ) have recently been proposed as a powerful alternative to standard vector embedding models for representing asymmetric relationships . Modeling such asymmetric relationships is central to discovering embedded representations of directed graphs , such as DAG-structured taxonomies , where geometric embeddings have been shown empirically to be more effective [ 39 , 58 , 60 ] . The success of geometric embeddings largely rests on exploiting inherent biases intrinsic to the particular geometry chosen . The negative curvature of hyperbolic space , for example , provides an inductive bias which is believed to better model the exponential increase of nodes in trees [ 39 , 61 , 62 ] . Region-based representations , on the other hand , model directional edges via containment relationships [ 58 , 60 ] . As containment is , itself , transitive , these models would seemingly have a natural bias toward representing transitive relationships . While previous work has empirically tested ⇤ Equal Contribution . † Now at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the generalization capability of these region-based geometric representations on directed graphs with missing edges [ 17 , 27 , 30 , 58 , 60 ] an understanding of their representational capability , both theoretically and experimentally , is lacking . There are several notable open questions regarding geometric embeddings . First , previous work has observed that the performance benefits of geometric embeddings are most strongly observed in low dimensions , and are often diminished or negated by vector models in higher dimensions , raising the question of whether there exists a geometric embedding model which can utilize higher dimensions more effectively . Second , previous work has provided theoretical results for vector [ 4 , 15 ] and hyperbolic [ 48 , 49 ] models , but has not addressed what region-based embeddings , such as order and box embeddings , can represent , theoretically . Third , there is not an agreed upon convention for how to empirically analyze the bias and representational capacity of models without confounding factors such as optimization , hyperparameters , and generalization performance . In this paper , we address each of these open questions . We introduce a novel box embedding model , extending the recently proposed Gumbel-box process [ 17 ] to learn from data a parametric model representing a continuous relaxation of box intersections . In this way , we can effectively rely on the structure of the graph to guide the fitting of the embedding model . Our model can trade-off between smoother , vector-like settings and more discrete , region-like settings , providing better performance across both lower and higher dimensionalities in a single model . We provide theoretical analysis of this approach as well as region-based approaches more generally . We lastly design a rigorous evaluation framework , which uses Bayesian hyperparameter optimization to fairly compare representational capacity independently from optimization details of each approach . This empirical framework allows us to characterize properties of graphs for which each method performs well . In summary , our contributions are as follows : Methodological Contributions ( §3 ) We introduce a novel continuous relaxation of probabilistic box embeddings , where each node has an associated trained variance or temperature . Theoretical Contributions ( §4 ) : We prove • Order Embeddings may produce cycles , even when pruning reverse edges ( Proposition 2 ) • Probabilistic Order Embeddings ( or Order Embeddings under ` 1 energy ) always produce acyclic graphs when pruning reverse edges ( Theorem 1 ) While it is known [ 60 ] that box embeddings also produce acyclic graphs when reverse edges are removed , we significantly strengthen this by proving • Any DAG can be represented by Probabilistic Box Embeddings in O ( ( + 2 ) log |V | ) dimensions , where is the maximum node degree , and |V | the number of nodes ( Theorem 2 ) • This can be accomplished using O ( D ( +2 ) log2 |V | ) bits of precision , where D is the max depth ( Proposition 3 ) Empirical Highlights ( §6 ) : We conduct a suite of experiments on synthetic and real-world graphs.1 We compare various vector , hyperbolic , cone , and box representations over a range of dimensions , using extensive Bayesian hyperparameter tuning , and find that • Geometric embedding models perform better in low dimensions • Our model outperforms all other geometric embeddings in high dimensions • Our model always outperforms the state-of-the-art box embeddings when using equal dimensions • Hyperbolic models do particularly well on trees , but struggle with transitivity • Region-based embeddings perform particularly well on transitively-closed graphs 2 Background : Embedding Directed Graphs . Given a simple2 directed graph G with nodes and edges ( V , E ) we seek to represent the adjacency matrix , that is , a function f : V ⇥ V ! { 0 , 1 } where f ( i , j ) = 1 if ( i , j ) 2 E and 0 otherwise . We parameterize the nodes , defining a function : V ! Rd , and define an energy function E : Rd ⇥ Rd ! R+ , which models the negative log probability of edge existence . We can view 1Our code and data are available at https : //github.com/iesl/geometric_graph_embedding . 2A simple directed graph has no multiple edges or self-loops , i.e . the adjacency matrix contains only 0s and 1s , with 0s on the diagonal . We remove self-loops from the learned representations . the probabilities provided as a weighted graph , however in practice it is often necessary to make a hard decision on edge existence , which is done by choosing a ( global ) threshold ⌧ and binarizing the output , yielding an adjacency matrix Ai , j = E ( i , j ) < ⌧ . 2.1 Vectors . A straightforward approach to this problem would be to parameterize each node using a vector , however typical score functions for vectors ( e.g. , inner-product , distance ) are symmetric , and thus would only be capable of embedding undirected graphs . Given a directed graph G , one can always first form an undirected graph G0 = ( V 0 , E0 ) with 2|V | nodes , where for each node i 2 V we associate two nodes iin , iout 2 V 0 , and for each edge ( i , j ) 2 E we have an edge ( iout , jin ) 2 E0 . We could embed this graph using symmetric score functions on vectors , for example E ( iout , jin ) : = log ( ( ( iout ) · ( jin ) ) ) . ( 1 ) Of course this can also be retracted to the original directed graph G , namely given a parameterization : V ! R2d we can define an energy function ESIM ( i , j ) : = log ✓ ✓ ( i ) T 0 I 0 0 ( j ) ◆◆ . ( 2 ) More generally , for ( learned ) weights W 2 R2d⇥2d we can define a bilinear score function [ 37 ] , EBILINEAR ( i , j ) : = log ( i ) TW ( j ) . ( 3 ) 2.2 Complex Vectors . The models discussed in §2.1 are often considered Euclidean vector embeddings , as they make use of the inner product and linear transformations on Euclidean space . Alternative spaces for representing nodes have been introduced , most notably complex space Cd [ 57 ] , where the parametrization can be thought of as : V ! Cd with energy function E ( i , j ) : = log ⇣ ⇣ Re ⇣Pd k=1 ( i ) kwk ( j ) k ⌘⌘⌘ ( 4 ) where w 2 Cd and Re projects to the real part . In order to compare this method to previous models , we can also think of this as parameterized by : V ! R2d , where ( i ) k : = Re ( i ) k and ( i ) d+k : = Im ( j ) k , in which case the energy function defined as ECOMPLEX ( i , j ) : = log ( i ) TW ( j ) with W : = diag ( Rew ) diag ( Imw ) diag ( Imw ) diag ( Rew ) ( 5 ) is equivalent to ( 4 ) . 2.3 Hyperbolic Vectors . While the use of bilinear products or complex vectors provide a naturally asymmetric scoring function , one might reasonably ask if there are spaces whose geometry lends itself to representing graphs . It is known , for example , that ( undirected ) trees can be embedded with arbitrarily low distortion into the Poincaré disk [ 49 ] , which is not possible in Euclidean space even using an unbounded number of dimensions [ 31 ] . These statements are related to undirected graphs , and while the in/out parameterization approach taken in the SIM model could be applied here we would lose any geometric benefit , as the undirected graph G0 corresponding to a tree is not , itself , a tree . Nickel and Kiela [ 39 ] suggest that edge direction can be decided by use of the Euclidean norm , and define a score function for an edge ( u , v ) as score ( u , v ) = ( 1 + ↵ ( kuk kvk ) ) d ( u , v ) , ( 6 ) where d is the Poincaré distance and ↵ is a hyperparameter indicating the strength of the penalty for edge directions which violate the order of their norms . An inspection of this score function shows that it is very sensitive to the choice of ↵ , which makes training unstable ( see Appendix A ) . Poincaré distance , in general , suffers from numeric stability issues which have been addressed in more recent work . Nickel and Kiela [ 40 ] consider the Lorentz model of hyperbolic space on the hyperboloid , and Law et al . [ 28 ] demonstrates that squared Lorentzian distance on the hyperboloid has similar geometric properties to Poincaré distance , while also providing numerically stable gradients . Using the notation of Law et al . [ 28 ] , the hyperboloid is defined as Hd , : = { a = ( a0 , . . . , ad ) 2 R+⇥Rd : kak2L = } , ( 7 ) where > 0 , kak2L = ha , aiL is the squared Lorentzian norm of a , and ha , biL is the Lorentzian inner product , and the squared Lorentzian distance d2L : Hd , ⇥Hd , ! R+ is defined as d2L ( a , b ) : = 2 2ha , biL , ha , biL : = a0b0 + Pd i=1 aibi . ( 8 ) Note that ha , biL for a , b 2 Hd , , with equality if and only if a = b . We also make use of the bijection f : Rd ! Hd , , where f ( x ) : = ⇣p kxkd + , x1 , . . . , xd ⌘ 2 Hd , . Given some parameterization : V ! Rd , we define a new hyperbolic energy function : EHYPERBOLIC ( i , j ) : = d 2 L ( f ( i ) , f ( j ) ) + ↵ log ( 1 + exp ( k ( i ) k k ( j ) k ) ) , ( 9 ) where the second term serves as a penalty on Euclidean norms to provide a measure of edge direction , i.e. , edges should go from nodes with smaller Euclidean norm to larger . ( See Appendix Figure 3 )

Summary: This paper analyses models for node embeddings of directed acyclic graphs (DAGs). It has 3 contributions:   i) it proposes a relatively straightforward extension of probabilistic box embeddings P-BOX [56], called t-BOX which is similar to the previous GumbleBox model [14], but has per-dimension and per-node intersection and volume temperature parameters (learnable) as opposed to just global parameters (as in GumbleBox). This extension is natural and the motivation is clear, but comes with the cost of doubling the number of parameters of the P-Box and GumbleBox models.  ii) theoretically, it extends the prior theory of [41] and [13] (which was analysing only undirected graphs) to work with DAGs and prove  that their t-Box model can represent any DAG (theorem 2, proof is an extension of [41]), giving an upper bound on number of dimensions and bits per box (proposition 3, proof is an extension of the result in [13]). Moreover, corollaries 1 and 2  are also the authors' contributions, being simple extensions of the result  in [56]. These theoretical results are important for understanding the representational capacity of existing DAG embedding models and I find this section nice and well written.  iii) Empirically, the authors evaluate the representational capacity (not generalization, just the "overfitting" ability) of several popular DAG embedding models on a variety of synthetic and real datasets. The choice of datasets is diverse and the empirical evaluation (e.g. removing the hyperparameters as confounding factors) is compelling, however, there are a few downsides that I will discuss below.