The Importance of Norm Regularization in Linear Graph Embedding: Theoretical Analysis and Empirical Demonstration

1 INTRODUCTION . Graphs have long been considered as one of the most fundamental structures that can naturally represent interactions between numerous real-life objects ( e.g. , the Web , social networks , proteinprotein interaction networks ) . Graph embedding , whose goal is to learn distributed representations for nodes while preserving the structure of the given graph , is a fundamental problem in network analysis that underpins many applications . A handful of graph embedding techniques have been proposed in recent years ( Perozzi et al. , 2014 ; Tang et al. , 2015b ; Grover & Leskovec , 2016 ) , along with impressive results in applications like link prediction , text classification ( Tang et al. , 2015a ) , and gene function prediction ( Wang et al. , 2015 ) . Linear graph embedding methods preserve graph structures by converting the inner products of the node embeddings into probability distributions with a softmax function ( Perozzi et al. , 2014 ; Tang et al. , 2015b ; Grover & Leskovec , 2016 ) . Since the exact softmax objective is computationally expensive to optimize , the negative sampling technique ( Mikolov et al. , 2013 ) is often used in these methods : instead of optimizing the softmax objective function , we try to maximize the probability of positive instances while minimizing the probability of some randomly sampled negative instances . It has been shown that by using this negative sampling technique , these graph embedding methods are essentially computing a factorization of the adjacency ( or proximity ) matrix of graph ( Levy & Goldberg , 2014 ) . Hence , it is commonly believed that the key to the generalization performance of these methods is the dimensionality constraint . However , in this paper we argue that the key factor to the good generalization of these embedding methods is not the dimensionality constraint , but rather the small norm of embedding vectors . We provide both theoretical and empirical evidence to support this argument : • Theoretically , we analyze the generalization error of two linear graph embedding hypothesis spaces ( restricting embedding dimension/norm ) , and show that only the norm-restricted hypothesis class can theoretically guarantee good generalization in typical parameter settings . • Empirically , we show that the success of existing linear graph embedding methods ( Perozzi et al. , 2014 ; Tang et al. , 2015b ; Grover & Leskovec , 2016 ) are due to the early stopping of stochastic gradient descent ( SGD ) , which implicitly restricts the norm of embedding vectors . Furthermore , with prolonged SGD execution and no proper norm regularization , the embedding vectors can severely overfit the training data . PAPER OUTLINE . The rest of this paper is organized as follows . In Section 2 , we review the definition of graph embedding problem and the general framework of linear graph embedding . In Section 3 , we present both theoretical and empirical evidence to support our argument that the generalization of embedding vectors is determined by their norm . In Section 4 , we present additional experimental results for a hinge-loss linear graph embedding variant , which further support our argument . In Section 5 , we discuss the new insights that we gained from previous results . Finally in Section 6 , we conclude our paper . Details of the experiment settings , algorithm pseudo-codes , theorem proofs and the discussion of other related work can all be found in the appendix . 2 PRELIMINIARIES . 2.1 THE GRAPH EMBEDDING PROBLEM . We consider a graph G = ( V , E ) , where V is the set of nodes in G , and E is the set of edges between the nodes in V . For any two nodes u , v ∈ V , an edge ( u , v ) ∈ E if u and v are connected , and we assume all edges are unweighted and undirected for simplicity1 . The task of graph embedding is to learn a D-dimensional vector representation xu for each node u ∈ V such that the structure of G can be maximally preserved . These embedding vectors can then be used as features for subsequent applications ( e.g. , node label classification or link prediction ) . 2.2 THE LINEAR GRAPH EMBEDDING FRAMEWORK . Linear graph embedding ( Tang et al. , 2015b ; Grover & Leskovec , 2016 ) is one of the two major approaches for computing graph embeddings 2 . These methods use the inner products of embedding vectors to capture the likelihood of edge existence , and are appealing to practitioners due to their simplicity and good empirical performance . Formally , given a node u and its neighborhood N+ ( u ) 3 , the probability of observing node v being a neighbor of u is defined as : p ( v|u ) = exp ( x T uxv ) ∑ k∈V exp ( xTuxk ) . By minimizing the KL-divergence between the embedding-based distribution and the actual neighborhood distribution , the overall objective function is equivalent to : L = − ∑ u∈V ∑ v∈N+ ( u ) log p ( v|u ) Unfortunately , it is quite problematic to optimize this objective function directly , as the softmax term involves normalizing over all vertices . To address this issue , the negative sampling ( Mikolov et al. , 2013 ) technique is used to avoid computing gradients over the full softmax function . Intuitively , the negative sampling technique can be viewed as randomly selecting a set of nodes N− ( u ) that are not connected to each node u as its negative neighbors . The embedding vectors are then learned by minimizing the following objective function instead : 1All linear graph embedding methods discussed in this paper can be generalized to weighted case by multiplying the weight to the corresponding loss function of each edge . The directed case is usually handled by associating each node with two embedding vectors for incoming and outgoing edges respectively , which is equivalent as learning embedding on a transformed undirected bipartite graph . 2The other major approach is to use deep neural network structure to compute the embedding vectors , see the discussion of other related works in the appendix for details . 3Note that N+ ( u ) can be either the set of direct neighbors in the original graph G ( Tang et al. , 2015b ) , or an expanded neighborhood based on measures like random walk ( Grover & Leskovec , 2016 ) . L = − ∑ u ∑ v∈N+ ( u ) log σ ( xTuxv ) − ∑ u ∑ v∈N− ( u ) κ |N+ ( u ) | |N− ( u ) | log σ ( −xTuxv ) . ( 1 ) where σ ( x ) = 1/ ( 1 + e−x ) is the standard logistic function . 2.3 THE MATRIX FACTORIZATION INTERPRETATION . Although the embedding vectors learned through negative sampling do have good empirical performance , there is very few theoretical analysis of such technique that explains the good empirical performance . The most well-known analysis of negative sampling was done by Levy & Goldberg ( 2014 ) , which claims that the embedding vectors are approximating a low-rank factorization of the PMI ( Pointwise Mutual Information ) matrix . More specifically , the key discovery of Levy & Goldberg ( 2014 ) is that when the embedding dimension is large enough , the optimal solution to Eqn ( 1 ) recovers exactly the PMI matrix ( up to a shifted constant , assuming the asymptotic case where N− ( u ) = V for all u ∈ V ) : ∀u , v , xTuxv = log ( |E| · 1 ( u , v ) ∈E |N+ ( u ) ||N+ ( v ) | ) − log κ Based on this result , Levy & Goldberg ( 2014 ) suggest that optimizing Eqn ( 1 ) under the dimensionality constraint is equivalent as computing a low-rank factorization of the shifted PMI matrix . This is currently the mainstream opinion regarding the intuition behind negative sampling . Although Levy and Goldberg only analyzed negative sampling in the context of word embedding , it is commonly believed that the same conclusion also holds for graph embedding ( Qiu et al. , 2018 ) . 3 THE IMPORTANCE OF NORM REGULARIZATION . As explained in Section 2.3 , it is commonly believed that linear graph embedding methods are approximating a low-rank factorization of PMI matrices . As such , people often deem the dimensionality constraint of embedding vectors as the key factor to good generalization ( Tang et al. , 2015b ; Grover & Leskovec , 2016 ) . However , due to the sparsity of real-world networks , the explanation of Levy & Goldberg is actually very counter-intuitive in the graph embedding setting : the average node degree usually only ranges from 10 to 100 , which is much less than the typical value of embedding dimension ( usually in the range of 100 ∼ 400 ) . Essentially , this means that in the context of graph embedding , the total number of free parameters is larger than the total number of training data points , which makes it intuitively very unlikely that the negative sampling model ( i.e. , Eqn ( 1 ) ) can inherently guarantee the generalization of embedding vectors in such scenario , and it is much more plausible if the observed good empirical performance is due to some other reason . In this paper , we provide a different explanation to the good empirical performance of linear graph embedding methods : we argue that the good generalization of linear graph embedding vectors is due to their small norm , which is in turn caused by the vanishing gradients during the stochastic gradient descent ( SGD ) optimization procedure . We provide the following evidence to support this argument : • In Section 3.1 , we theoretically analyze the generalization error of two linear graph embedding variants : one has the standard dimensionality constraints , while the other restricts the vector norms . Our analysis shows that : – The embedding vectors can generalize well to unseen data if their average squared l2 norm is small , and this is always true regardless of the embedding dimension choice . – Without norm regularization , the embedding vectors can severely overfit the training data if the embedding dimension is larger than the average node degree . • In Section 3.2 , we provide empirical evidence that the generalization of linear graph embedding is determined by vector norm instead of embedding dimension . We show that : – In practice , the average norm of the embedding vectors is small due to the early stopping of SGD and the vanishing gradients . – The generalization performance of embedding vectors starts to drop when the average norm of embedding vectors gets large . – The dimensionality constraint is only helpful when the embedding dimension is very small ( around 5 ∼ 10 ) and there is no norm regularization . 3.1 GENERALIZATION ANALYSIS OF TWO LINEAR GRAPH EMBEDDING VARIANTS . In this section , we present a generalization error analysis of linear graph embedding based on the uniform convergence framework ( Bartlett & Mendelson , 2002 ) , which bounds the maximum difference between the training and generalization error over the entire hypothesis space . We assume the following statistical model for graph generation : there exists an unknown probability distribution Q over the Cartesian product V × U of two vertex sets V and U . Each sample ( a , b ) from Q denotes an edge connecting a ∈ V and b ∈ U .The set of ( positive ) training edges E+ consists of the first m i.i.d . samples from the distribution Q , and the negative edge set E− consists of i.i.d . samples from the uniform distribution U over V × U . The goal is to use these samples to learn a model that generalizes well to the underlying distribution Q . We allow either V = U for homogeneous graphs or V ∩ U = ∅ for bipartite graphs . Denote E± = { ( a , b , +1 ) : ( a , b ) ∈ E+ } ∪ { ( a , b , −1 ) : ( a , b ) ∈ E− } to be the collection of all training data , and we assume that data points inE± are actually sampled from a combined distribution P over V ×U × { ±1 } that generates both positive and negative edges . Using the above notations , the training error Lt ( x ) and generalization error Lg ( x ) of embedding x : ( U ∪ V ) → RD are defined as follows : Lt ( x ) = 1 |E±| ∑ ( a , b , y ) ∈E± − log σ ( yxTa xb ) Lg ( x ) = −E ( a , b , y ) ∼P log σ ( yxTa xb ) In the uniform convergence framework , we try to prove the following statement : Pr ( sup x∈H ( Lg ( x ) − Lt ( x ) ) ≤ ) ≥ 1− δ which bounds the maximum difference between Lt ( x ) and Lg ( x ) over all possible embeddings x in the hypothesis space H. If the above uniform convergence statement is true , then minimizing the training error Lt ( x ) would naturally lead to small generalization error Lg ( x ) with high probability . Now we present our first technical result , which follows the above framework and bounds the generalization error of linear graph embedding methods with norm constraints : Theorem 1 . [ Generalization of Linear Graph Embedding with Norm Constraints ] Let E± = { ( a1 , b1 , y1 ) , ( a2 , b2 , y2 ) , . . . , ( am+m′ , bm+m′ , ym+m′ ) } be i.i.d . samples from a distribution P over V × U × { ±1 } . Let x : ( U ∪ V ) → RD to be the embedding for nodes in the graph . Then for any bounded 1-Lipschitz loss function l : R→ [ 0 , B ] and CU , CV > 0 , with probability 1− δ ( over the sampling of E± ) , the following inequality holds E ( a , b , y ) ∼P l ( yxTa xb ) ≤ 1 m+m′ m+m′∑ i=1 l ( yix T aixbi ) + 2 √ CUCV m+m′ Eσ||Aσ||2 + 4B √ 2 ln ( 4/δ ) m+m′ ( 2 ) for all embeddings x satisfying ∑ u∈U ‖xu‖2 ≤ CU , ∑ v∈V ‖xv‖2 ≤ CV where ||Aσ||2 is the spectral norm of the randomized adjacency matrix Aσ defined as follows : Aσ ( i , j ) = { σij ∃y , ( ui , vj , y ) ∈ E± 0 ∀y , ( ui , vj , y ) /∈ E± in which σij are i.i.d . Rademacher random variables . The proof can be found in the appendix . Intuitively , Theeorem 1 states that with sufficient norm regularization , linear graph embedding can generalize well regardless of the embedding dimension ( note that term D does not appear in Eqn ( 2 ) at all ) . Theorem 1 also characterizes the importance of choosing proper regularization in lnorm restricted inear graph embedding : in Eqn ( 2 ) , both the training error term 1m+m′ ∑m+m′ i=1 l ( yix T aixbi ) and the gap term 2 √ CUCV m+m′ Eσ||Aσ||2 are dependent on the value of CU and CV . With larger C values ( i.e. , weak norm regularization ) , the training error would be smaller due to the less restrictive hypothesis space , but the gap term would larger , meaning that the model will likely overfit the training data . Meanwhile , smaller C values ( i.e. , strong norm regularization ) would lead to more restrictive models , which will not overfit but have larger training error as trade-off . Therefore , choosing the most proper norm regularization is the key to achieving optimal generalization performance . A rough estimate of Eσ||Aσ||2 can be found in the appendix for interested readers . On the other hand , if we restrict only the embedding dimension ( i.e. , no norm regularization on embedding vectors ) , and the embedding dimension is larger than the average degree of the graph , then it is possible for the embedding vectors to severely overfit the training data . The following example demonstrates this possibility on a d-regular graph , in which the embedding vectors can always achieve zero training error even when the edge labels are randomly placed : Claim 1 . Let G = ( V , E ) be a d-regular graph with n vertices and m = nd/2 labeled edges ( with labels yi ∈ { ±1 } ) : V = { v1 , . . . , vn } E = { ( a1 , b1 , y1 ) , . . . , ( am , bm , ym ) } Then for each of the 2m possible label combinations , there exists a embedding x : V → RD with dimension D = d that achieves perfect classification accuracy on the randomized training dataset : ∀ ( y1 , . . . , ym ) ∈ { ±1 } m , ∃x : V → RD s.t . ∀i ∈ { 1 , . . . , m } , yixTaixbi > 1 The proof can be found in the appendix . In other words , without norm regularization , the number of training samples required for learning D-dimensional embedding vectors is at least Ω ( nD ) . Considering the fact that many large-scale graphs are sparse ( with average degree < 20 ) and the default embedding dimension commonly ranges from 100 to 400 , it is highly unlikely that the the dimensionality constraint by itself could lead to good generalization performance .

Summary: In this paper, the authors proved that the generalization error of linear graph embedding methods is bounded by the norm of embedding vectors, rather than the dimensionality constraints. Interestingly, along with the analysis of Levy & Goldberg (2014), they found that linear graph embedding methods are probably computing a low-norm factorization of the PMI matrix. Correspondingly,  experimental results are provided to support their analysis.