Generalization Bounds for Graph Embedding Using Negative Sampling: Linear vs Hyperbolic

1 Introduction . Graphs are a fundamental formulation of real-world entities and their relations , such as words in natural languages , people in social network , and objects in knowledge bases . Here , the vertices and edges of a graph correspond to the entities and the relations among them , respectively . Based on the formulation , graph embedding has enabled numerous applications for those data , such as machine translation and sentiment analysis for natural language [ 1 , 2 , 3 , 4 ] , and community detection and ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . link prediction for social network data [ 5 , 6 , 7 , 8 , 9 ] , pathway prediction of biochemical network [ 10 , 11 ] , and link prediction and triplet classification for knowledge base [ 12 , 13 , 14 , 15 , 16 , 17 ] . Graph embedding produces representations of a graph ’ s vertices in a space equipped with a function that defines the dissimilarity between two points . In this paper , we call a function that defines the dissimilarity a dissimilarity function and a space equipped with a dissimilarity function a dissimilarity space . For example , we can consider the squared distance as a dissimilarity function . Graph embedding aims to obtain representations such that the dissimilarity function reflects the relations defined by the edges . Specifically , we expect that the dissimilarity function returns a small value for the representations of a positive pair , a pair of vertices connected by an edge , and a large value for a negative pair , a pair not connected . Here , to reduce the computational cost , obtaining training data by the negative sampling strategy has been known to be effective [ 1 ] . In this strategy , we sample a positive pair for each iteration , followed by sampling negative pairs around the positive pair . As a dissimilarity space , many graph embedding methods have used linear space equipped with an inner product function [ 1 , 2 , 3 , 5 , 6 , 7 , 8 , 9 ] , which we call linear graph embedding ( LGE ) . However , linear space has limitations in representing data with a hierarchical tree-like structure [ 18 , 19 , 20 , 21 ] . These limitations are due to linear space ’ s polynomial growth property , which means that the volume or surface of a ball in linear space grows polynomially with respect to its radius . This linear space ’ s growth speed is significantly slower than embedding hierarchical data such as an r-ary tree ( r ≥ 2 ) requires , which is exponential [ 21 ] . To overcome this limitation , graph embedding in hyperbolic space has recently attracted much attention [ 20 , 22 , 23 , 24 , 25 , 26 , 4 , 27 ] , which we call hyperbolic graph embedding ( HGE ) in this paper . In contrast to linear space ’ s polynomial growth property , hyperbolic space has the exponential growth property , that is , the volume of any ball in hyperbolic space grows exponentially with respect to its radius [ 18 , 19 , 20 , 21 ] . As a result , hyperbolic space is almost tree-like in that it can be well approximated by a tree [ 28 ] , and we can embed any tree in hyperbolic space with arbitrarily low distortion [ 29 ] . Existing HGE papers have experimentally shown HGE ’ s ability to effectively represent hierarchical tree-like data such as taxonomies and social networks . However , the theoretical guarantee of HGE ’ s performance is limited to ideal noiseless settings [ 28 , 29 , 23 ] , and the comparison between LGE and HGE ’ s generalization performance in noisy settings has not been discussed , although HGE could have a much worse generalization error than LGE in compensation for hyperbolic space ’ s exponential growth property and cause overfitting for real data , which are often noisy . In this paper , we derive a generalization bound for graph embedding using the negative sampling strategy under noisy settings . To the best of our knowledge , this is the first work that derives a complete generalization error bound for both LGE and HGE . As discussed in [ 21 ] , since the generalization error of a learning model reflects the volume of its hypothesis space , we can conjecture that the generalization errors of LGE and HGE are polynomial and exponential with respect to the embedding space ’ s radius , reflecting inner product space and hyperbolic space ’ s polynomial and exponential growth property . Also , an imbalanced data distribution , which often appears in graph embedding reflecting the graph structure ’ s imbalance , may worsen the error . In this paper , we formally prove that the above conjectures are true , as well as clarify the dependency of the error bound on the number of entities and the size of training data . Based on the derived generalization error bounds , we also clarify the data size condition on which HGE outperforms LGE in embedding a tree , by discussing the bias-variance trade-off . To derive a generalization error bound in embedding problem , existing papers [ 30 , 21 ] deriving ordinal embedding ’ s bounds have converted the problem into a linear prediction problem to calculate its Rademacher complexity [ 31 , 32 , 33 ] . Also , for hyperbolic embedding , the decomposition of the Lorentz Gramian matrix [ 34 ] has been combined with the above technique [ 21 ] . For graph embedding using negative sampling , however , we can not straightforwardly apply these techniques , which have been effective for deriving ordinal embedding ’ s generalization error bound . Since the data distribution depends on the graph structure and positive sampling affects the distribution of negative sampling in the negative sampling structure , we can not apply the i.i.d.-uniform-distribution-based discussions in [ 30 , 21 ] to graph embedding using the negative sampling strategy . Although a recent unpublished paper [ 35 ] has attempted to derive a generalization error bound only for LGE , which is incomplete in that the bound still has an unevaluated part and can not be applied for noisy settings , this dependency between the distributions has been ignored . We solve this problem by decomposing the loss function into functions of each edge sampled by the negative sampling strategy . We achieved this decomposition by our novel multivariable version of the Ledoux-Talagrand contraction lemma [ 36 ] . By our approach , we can upper-bound the Rademacher complexity of a graph embedding loss function by the sum of the Rademacher complexities [ 31 , 32 , 33 ] of linear prediction models , which have been calculated in [ 30 , 21 ] . As a result , our generalization bound is valid for various negative sampling settings where the distribution of positive and negative edges are dependent . Our contributions are threefold : • We have derived the generalization error bound for negative-sampling-based graph embedding . Our theorem is applicable to various settings regarding the embedding space , data distribution and loss function , such as LGE and HGE , the dependency between the positive pairs and negative pairs ’ distribution , and sigmoid loss functions . Our upper bound shows that LGE and HGE cause a polynomial and exponential error with respect to the embedding space ’ s radius , respectively . Our bound also shows that imbalanced data distribution can worsen the error . • We have derived specific error bounds for practical negative sampling strategies . • We have derived an explicit training data size condition on which HGE can represent a tree better than LGE . 2 Preliminary . Notation In this paper , the symbol : = is used to state that its left hand side is defined by its right hand side . We denote by Z , Z > 0 , R , R≥0 the set of integers , the set of positive integers , the set of real numbers , and the set of non-negative real numbers , respectively . Suppose that D , V ∈ Z > 0 . We denote by , RD the set of D-dimensional real vectors . For a matrixA ∈ RD , V , we denote by [ A ] d , v the element in the d-row and the v-th column . For a vector x ∈ RD , we denote by ‖x‖2 the 2-norm of x , defined by ‖x‖2 = √ x > x , and for a matrixA ∈ RD , V , we denote by ‖A‖op,2 , the operator norm ofA with respect to the 2-norm , defined by ‖A‖op,2 : = supx∈V , x 6=0 ‖Ax‖2 ‖x‖2 . By ( an ) N n=1 , we denote a sequence ( a1 , a2 , . . . , aN ) . 2.1 Graph Embedding . In this section , we first formulate the general embedding problem , before we specialize it into a graph embedding . Consider an entity set V and the true dissimilarity function V × V → R defined on the entity set V . For i , j ∈ V , we call δ∗ ( i , j ) the true dissimilarity between i and j , and a small δ∗ ( i , j ) value implies that entity i and j are similar or closely related to each other , and large δ∗ ( i , j ) value implies its converse . In this paper , we identify V with the integer set { 1 , 2 , . . . , |V| } . Embedding aims to get representations z1 , z2 , . . . , zV of the entity set V in a space ( Z , δZ ) equipped with a dissimilarity function δZ : Z × Z → R of Z so that the representations are consistent to the true dissimilarity among the entities in that if δ∗ ( i , j ) is small , then δ ( zi , zj ) is also small , and vice versa . We call Z the embedding space . Specifically , we aim to satisfy δ∗ ( i , j ) Q θ∗ ⇔ δZ ( zi , zj ) Q θZ , ( 1 ) as frequent as possible with respect to some distribution regarding ( i , j ) , which we discuss in the next subsection . Here , θ∗ , θZ ∈ RZ are fixed thresholds regarding the true dissimilarity δ∗ and the dissimilarity δZ in the embedding space . As a dissimilarity function of the embedding space Z , we mainly consider the square distance [ ∆Z ( zi , zj ) ] 2 if Z is equipped with a distance function ∆Z : Z × Z → R≥0 , and the negative inner product −〈zi , zj〉 if Z is equipped with an inner product 〈· , ·〉 : Z ×Z → R. In this paper , we deal with the following four dissimilarity spaces . Definition 1 . ( Dissimilarity spaces ) ( a ) The D-dimensional Euclidean space ( RD , ∆RD ) consists of the set of D-dimensional real vectors and the distance function ∆RD : RD × RD → R≥0 defined by ∆RD ( z , z′ ) : = ‖z − z′‖ 2 2 . The dissimilarity function is given by δRD ( z , z′ ) : = [ ∆RD ( z , z′ ) ] 2 . ( b ) The D-dimensional hyperbolic space ( LD , ∆LD ) consists of the D-dimensional hyperboloid LD ⊂ RD+1 and the distance function ∆LD defined by LD : = { x ∈ RD+1 ∣∣ 〈x , x〉M = −1 } , ∆LD ( x , x′ ) : = arcosh ( −〈x , x′〉M ) , ( 2 ) where 〈· , ·〉M : RD × RD → R is the Minkowski inner prodct defined by〈 [ z0 z1 · · · zD ] > , [ z′0 z′1 · · · z′D ] > 〉 M : = −z0z′0 + ∑D d=1 zdz ′ d. The dissimilarity function is given by δLD ( z , z′ ) : = [ ∆LD ( z , z′ ) ] 2 . ( c ) The D-dimensional sphere ( SD , ∆SD ) consists of the subset SD ⊂ RD+1 and the distance function ∆LD defined by SD : = { x ∈ RD+1 ∣∣ x > x = 1 } , ∆SD ( x , x′ ) : = arccos ( x > x′ ) . ( 3 ) The dissimilarity function is given by δSD ( z , z′ ) : = [ ∆SD ( z , z′ ) ] 2 . ( d ) The canonical D-dimensional inner product space ( ID , δID ) as a dissimilarity space consists of the set ID = RD of D-dimensional real vectors and the dissimilarity function δID : ID × ID → R defined by the negative canonical inner product δID ( z , z′ ) : = −z > z′ . Our main focus in this paper is RD and LD , although our bound is also applicable to SD and ID . Remark 1 . There are multiple models to represent the above spaces . For example , we can use the Poincaré ball model , upper half space model , Klein ball model to represent hyperbolic space , other than the hyperboloid model used in Definition 1 . While these are isometric to each other , we used the models in Definition 1 because we can formulate the dissimilarity function as a simple function of a linear combination of inner products z > z , z > z′ , and z′ > z′ , or 〈x , x′〉M , which makes it easy to apply techniques in [ 30 , 21 ] . Graph Embedding Consider a graph ( V , E ) , where V is the vertex set and E ∈ V × V is the edge set . Here , we only consider undirected graphs , and thus assume that if ( i , j ) ∈ E then ( j , i ) ∈ E holds . Graph embedding for a graph ( V , E ) is a special case of embedding problem defined above , where the vertices V are the entities , the true dissimilarity between two entities i , j is given by the graph distance defined by δ∗ ( i , j ) : = min { K ∣∣ ∃ ( v1 , v2 , . . . , vK−1 ) ∈ VK−1 , ( i , v1 ) , ( v1 , v2 ) , . . . , ( vK−1 , j ) ∈ E . } , ( 4 ) and θ∗ = 1 . Then , the objective defined by ( 1 ) is equivalent to ( i , j ) ∈ E ⇔ δZ ( zi , zj ) ≤ θz . ( 5 ) We say that a pair ( i , j ) is truly positive if ( i , j ) ∈ E and truly negative otherwise . In the following , we denote by deg ( i ) the degree of i ∈ V , defined by deg ( i ) : = { j| ( i , j ) ∈ E } .

Summary: Graph embedding has already been a common and effective technique in many applications, while the corresponding theoretical analysis is limited to ideal noiseless settings. This work intends to derive the unified generalization error bound for negative-sampling-based graph embedding regarding the existence of noise. However, the assumption made in this work to drive the theoretical analysis is somewhat impractical making the derived bound a little bit vacuous.