Generalization Bounds for Graph Embedding Using Negative Sampling: Linear vs Hyperbolic

1 Introduction . Graphs are a fundamental formulation of real-world entities and their relations , such as words in natural languages , people in social network , and objects in knowledge bases . Here , the vertices and edges of a graph correspond to the entities and the relations among them , respectively . Based on the formulation , graph embedding has enabled numerous applications for those data , such as machine translation and sentiment analysis for natural language [ 1 , 2 , 3 , 4 ] , and community detection and ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . link prediction for social network data [ 5 , 6 , 7 , 8 , 9 ] , pathway prediction of biochemical network [ 10 , 11 ] , and link prediction and triplet classification for knowledge base [ 12 , 13 , 14 , 15 , 16 , 17 ] . Graph embedding produces representations of a graph ’ s vertices in a space equipped with a function that defines the dissimilarity between two points . In this paper , we call a function that defines the dissimilarity a dissimilarity function and a space equipped with a dissimilarity function a dissimilarity space . For example , we can consider the squared distance as a dissimilarity function . Graph embedding aims to obtain representations such that the dissimilarity function reflects the relations defined by the edges . Specifically , we expect that the dissimilarity function returns a small value for the representations of a positive pair , a pair of vertices connected by an edge , and a large value for a negative pair , a pair not connected . Here , to reduce the computational cost , obtaining training data by the negative sampling strategy has been known to be effective [ 1 ] . In this strategy , we sample a positive pair for each iteration , followed by sampling negative pairs around the positive pair . As a dissimilarity space , many graph embedding methods have used linear space equipped with an inner product function [ 1 , 2 , 3 , 5 , 6 , 7 , 8 , 9 ] , which we call linear graph embedding ( LGE ) . However , linear space has limitations in representing data with a hierarchical tree-like structure [ 18 , 19 , 20 , 21 ] . These limitations are due to linear space ’ s polynomial growth property , which means that the volume or surface of a ball in linear space grows polynomially with respect to its radius . This linear space ’ s growth speed is significantly slower than embedding hierarchical data such as an r-ary tree ( r ≥ 2 ) requires , which is exponential [ 21 ] . To overcome this limitation , graph embedding in hyperbolic space has recently attracted much attention [ 20 , 22 , 23 , 24 , 25 , 26 , 4 , 27 ] , which we call hyperbolic graph embedding ( HGE ) in this paper . In contrast to linear space ’ s polynomial growth property , hyperbolic space has the exponential growth property , that is , the volume of any ball in hyperbolic space grows exponentially with respect to its radius [ 18 , 19 , 20 , 21 ] . As a result , hyperbolic space is almost tree-like in that it can be well approximated by a tree [ 28 ] , and we can embed any tree in hyperbolic space with arbitrarily low distortion [ 29 ] . Existing HGE papers have experimentally shown HGE ’ s ability to effectively represent hierarchical tree-like data such as taxonomies and social networks . However , the theoretical guarantee of HGE ’ s performance is limited to ideal noiseless settings [ 28 , 29 , 23 ] , and the comparison between LGE and HGE ’ s generalization performance in noisy settings has not been discussed , although HGE could have a much worse generalization error than LGE in compensation for hyperbolic space ’ s exponential growth property and cause overfitting for real data , which are often noisy . In this paper , we derive a generalization bound for graph embedding using the negative sampling strategy under noisy settings . To the best of our knowledge , this is the first work that derives a complete generalization error bound for both LGE and HGE . As discussed in [ 21 ] , since the generalization error of a learning model reflects the volume of its hypothesis space , we can conjecture that the generalization errors of LGE and HGE are polynomial and exponential with respect to the embedding space ’ s radius , reflecting inner product space and hyperbolic space ’ s polynomial and exponential growth property . Also , an imbalanced data distribution , which often appears in graph embedding reflecting the graph structure ’ s imbalance , may worsen the error . In this paper , we formally prove that the above conjectures are true , as well as clarify the dependency of the error bound on the number of entities and the size of training data . Based on the derived generalization error bounds , we also clarify the data size condition on which HGE outperforms LGE in embedding a tree , by discussing the bias-variance trade-off . To derive a generalization error bound in embedding problem , existing papers [ 30 , 21 ] deriving ordinal embedding ’ s bounds have converted the problem into a linear prediction problem to calculate its Rademacher complexity [ 31 , 32 , 33 ] . Also , for hyperbolic embedding , the decomposition of the Lorentz Gramian matrix [ 34 ] has been combined with the above technique [ 21 ] . For graph embedding using negative sampling , however , we can not straightforwardly apply these techniques , which have been effective for deriving ordinal embedding ’ s generalization error bound . Since the data distribution depends on the graph structure and positive sampling affects the distribution of negative sampling in the negative sampling structure , we can not apply the i.i.d.-uniform-distribution-based discussions in [ 30 , 21 ] to graph embedding using the negative sampling strategy . Although a recent unpublished paper [ 35 ] has attempted to derive a generalization error bound only for LGE , which is incomplete in that the bound still has an unevaluated part and can not be applied for noisy settings , this dependency between the distributions has been ignored . We solve this problem by decomposing the loss function into functions of each edge sampled by the negative sampling strategy . We achieved this decomposition by our novel multivariable version of the Ledoux-Talagrand contraction lemma [ 36 ] . By our approach , we can upper-bound the Rademacher complexity of a graph embedding loss function by the sum of the Rademacher complexities [ 31 , 32 , 33 ] of linear prediction models , which have been calculated in [ 30 , 21 ] . As a result , our generalization bound is valid for various negative sampling settings where the distribution of positive and negative edges are dependent . Our contributions are threefold : • We have derived the generalization error bound for negative-sampling-based graph embedding . Our theorem is applicable to various settings regarding the embedding space , data distribution and loss function , such as LGE and HGE , the dependency between the positive pairs and negative pairs ’ distribution , and sigmoid loss functions . Our upper bound shows that LGE and HGE cause a polynomial and exponential error with respect to the embedding space ’ s radius , respectively . Our bound also shows that imbalanced data distribution can worsen the error . • We have derived specific error bounds for practical negative sampling strategies . • We have derived an explicit training data size condition on which HGE can represent a tree better than LGE . 2 Preliminary . Notation In this paper , the symbol : = is used to state that its left hand side is defined by its right hand side . We denote by Z , Z > 0 , R , R≥0 the set of integers , the set of positive integers , the set of real numbers , and the set of non-negative real numbers , respectively . Suppose that D , V ∈ Z > 0 . We denote by , RD the set of D-dimensional real vectors . For a matrixA ∈ RD , V , we denote by [ A ] d , v the element in the d-row and the v-th column . For a vector x ∈ RD , we denote by ‖x‖2 the 2-norm of x , defined by ‖x‖2 = √ x > x , and for a matrixA ∈ RD , V , we denote by ‖A‖op,2 , the operator norm ofA with respect to the 2-norm , defined by ‖A‖op,2 : = supx∈V , x 6=0 ‖Ax‖2 ‖x‖2 . By ( an ) N n=1 , we denote a sequence ( a1 , a2 , . . . , aN ) . 2.1 Graph Embedding . In this section , we first formulate the general embedding problem , before we specialize it into a graph embedding . Consider an entity set V and the true dissimilarity function V × V → R defined on the entity set V . For i , j ∈ V , we call δ∗ ( i , j ) the true dissimilarity between i and j , and a small δ∗ ( i , j ) value implies that entity i and j are similar or closely related to each other , and large δ∗ ( i , j ) value implies its converse . In this paper , we identify V with the integer set { 1 , 2 , . . . , |V| } . Embedding aims to get representations z1 , z2 , . . . , zV of the entity set V in a space ( Z , δZ ) equipped with a dissimilarity function δZ : Z × Z → R of Z so that the representations are consistent to the true dissimilarity among the entities in that if δ∗ ( i , j ) is small , then δ ( zi , zj ) is also small , and vice versa . We call Z the embedding space . Specifically , we aim to satisfy δ∗ ( i , j ) Q θ∗ ⇔ δZ ( zi , zj ) Q θZ , ( 1 ) as frequent as possible with respect to some distribution regarding ( i , j ) , which we discuss in the next subsection . Here , θ∗ , θZ ∈ RZ are fixed thresholds regarding the true dissimilarity δ∗ and the dissimilarity δZ in the embedding space . As a dissimilarity function of the embedding space Z , we mainly consider the square distance [ ∆Z ( zi , zj ) ] 2 if Z is equipped with a distance function ∆Z : Z × Z → R≥0 , and the negative inner product −〈zi , zj〉 if Z is equipped with an inner product 〈· , ·〉 : Z ×Z → R. In this paper , we deal with the following four dissimilarity spaces . Definition 1 . ( Dissimilarity spaces ) ( a ) The D-dimensional Euclidean space ( RD , ∆RD ) consists of the set of D-dimensional real vectors and the distance function ∆RD : RD × RD → R≥0 defined by ∆RD ( z , z′ ) : = ‖z − z′‖ 2 2 . The dissimilarity function is given by δRD ( z , z′ ) : = [ ∆RD ( z , z′ ) ] 2 . ( b ) The D-dimensional hyperbolic space ( LD , ∆LD ) consists of the D-dimensional hyperboloid LD ⊂ RD+1 and the distance function ∆LD defined by LD : = { x ∈ RD+1 ∣∣ 〈x , x〉M = −1 } , ∆LD ( x , x′ ) : = arcosh ( −〈x , x′〉M ) , ( 2 ) where 〈· , ·〉M : RD × RD → R is the Minkowski inner prodct defined by〈 [ z0 z1 · · · zD ] > , [ z′0 z′1 · · · z′D ] > 〉 M : = −z0z′0 + ∑D d=1 zdz ′ d. The dissimilarity function is given by δLD ( z , z′ ) : = [ ∆LD ( z , z′ ) ] 2 . ( c ) The D-dimensional sphere ( SD , ∆SD ) consists of the subset SD ⊂ RD+1 and the distance function ∆LD defined by SD : = { x ∈ RD+1 ∣∣ x > x = 1 } , ∆SD ( x , x′ ) : = arccos ( x > x′ ) . ( 3 ) The dissimilarity function is given by δSD ( z , z′ ) : = [ ∆SD ( z , z′ ) ] 2 . ( d ) The canonical D-dimensional inner product space ( ID , δID ) as a dissimilarity space consists of the set ID = RD of D-dimensional real vectors and the dissimilarity function δID : ID × ID → R defined by the negative canonical inner product δID ( z , z′ ) : = −z > z′ . Our main focus in this paper is RD and LD , although our bound is also applicable to SD and ID . Remark 1 . There are multiple models to represent the above spaces . For example , we can use the Poincaré ball model , upper half space model , Klein ball model to represent hyperbolic space , other than the hyperboloid model used in Definition 1 . While these are isometric to each other , we used the models in Definition 1 because we can formulate the dissimilarity function as a simple function of a linear combination of inner products z > z , z > z′ , and z′ > z′ , or 〈x , x′〉M , which makes it easy to apply techniques in [ 30 , 21 ] . Graph Embedding Consider a graph ( V , E ) , where V is the vertex set and E ∈ V × V is the edge set . Here , we only consider undirected graphs , and thus assume that if ( i , j ) ∈ E then ( j , i ) ∈ E holds . Graph embedding for a graph ( V , E ) is a special case of embedding problem defined above , where the vertices V are the entities , the true dissimilarity between two entities i , j is given by the graph distance defined by δ∗ ( i , j ) : = min { K ∣∣ ∃ ( v1 , v2 , . . . , vK−1 ) ∈ VK−1 , ( i , v1 ) , ( v1 , v2 ) , . . . , ( vK−1 , j ) ∈ E . } , ( 4 ) and θ∗ = 1 . Then , the objective defined by ( 1 ) is equivalent to ( i , j ) ∈ E ⇔ δZ ( zi , zj ) ≤ θz . ( 5 ) We say that a pair ( i , j ) is truly positive if ( i , j ) ∈ E and truly negative otherwise . In the following , we denote by deg ( i ) the degree of i ∈ V , defined by deg ( i ) : = { j| ( i , j ) ∈ E } .

Summary: This paper studies the question of the generalization error incurred when embedding graphs in linear vs. hyperbolic space. Recent literature (e.g. work of Nickel and Kiela, ref. [20]) has shown that graph data with hierarchical structure is often better embedded in hyperbolic space--due to its higher representation capacity--than in linear space (for which it becomes necessary to increase the dimensionality of the embedding). However, this choice may come with a tradeoff in terms of generalization error, and the current paper studies this phenomenon in the context of embeddings obtained via negative sampling. Specifically, the authors show that the generalization errors of such embedding into linear and hyperbolic space incur polynomial and exponential growth, respectively, and are also affected by imbalance in how edges are distributed (i.e. regular graphs are balanced, graphs with hubs are more imbalanced). Some of the difficulty of this problem arises due to dependency in the negative sampling procedure, such that iid assumptions and related results cannot be applied. Towards the end, the authors also show that for the special case of tree data, their theory suggests that the generalization error incurred by a hyperbolic embedding is better than that of a linear embedding. Finally, the authors discuss the limitation that their bound does not depend on the embedding dimension, so is likely not tight. The authors explain their setup through numerous examples of negative sampling strategies, notably the Skip-gram model (ref. [1]), as well as detailed explanations of the different terms appearing in their main result.