Contrastive Laplacian Eigenmaps

1 Introduction . Celebrated graph embedding methods , including Laplacian Eigenmaps [ 5 ] and IsoMap [ 42 ] , reduce the dimensionality of the data by assuming that it lies on a low-dimensional manifold . The objective functions used in studies [ 5 , 42 ] model the pairwise node similarity [ 7 ] by encouraging the embeddings of nodes to lie close in the embedding space if the nodes are closely related . In other words , such penalties do not guarantee that unrelated graph nodes are separated from each other in the embedding space . For instance , Elastic Embedding [ 8 ] uses data-driven affinities for the so-called local distance term and the data-independent repulsion term . In contrast , modern graph embedding models , often unified under the Sampled Noise Contrastive Estimation ( SampledNCE ) framework [ 33 , 28 ] and extended to graph learning [ 41 , 15 , 50 ] , enjoy contrastive objectives . By maximizing the mutual information between patch representations and highlevel summaries of the graph , Deep Graph Infomax ( DGI ) [ 43 ] is a contrastive method . GraphSAGE [ 15 ] minimizes/maximizes distances between so-called positive/negative pairs , respectively . It relies on the inner product passed through the sigmoid non-linearity , which we argue below as suboptimal . Thus , we propose a new COntrastive Laplacian EigenmapS ( COLES ) framework for unsupervised network embedding . COLES , derived from SampledNCE framework [ 33 , 28 ] , realizes the negative sampling strategy for Laplacian Eigenmaps . Our general objective is given as : Θ∗ = argmax Θ Tr ( fΘ ( X ) > ∆WfΘ ( X ) ) + βΩ ( fΘ ( X ) ) . ( 1 ) X ∈ Rn×d in Eq . ( 1 ) is the node feature matrix with d feature dimensions given n nodes , fΘ ( X ) ∈ Rn×d′ is an output of a chosen Graph Neural Network backbone ( embeddings to optimize ) with the * The corresponding author . Code : https : //github.com/allenhaozhu/COLES . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . feature dimension d′ , Θ denotes network parameters , whereas ∆W ∈ Sn+ is the difference between the degree-normalized positive and negative adjacency matrices which represent the data graph and some negative graph capturing negative links for contrastive learning . Moreover , β ≥ 0 controls the regularization term Ω ( · ) whose role is to constrain the ` 2 norm of network outputs or encourage the so-called incoherence [ 36 ] between column vectors . Section 3.1 presents COLES for the Linear Graph Network ( LGN ) family , in which we take special interest due to their simplicity and agility . By building upon previous studies [ 28 , 2 , 48 ] , we show that COLES can be derived by reformulating SampledNCE into Wasserstein GAN using a GAN-inspired contrastive formulation . This result has a profound impact on the performance of COLES , as the standard contrastive approaches based on SampledNCE strategy ( i.e. , GraphSAGE [ 15 ] ) turn out to utilize the Jensen-Shannon divergence , which yields log 2 constant and vanishing gradients for disjoint distributions of positive and negative sampled pairs used for contrastive learning . Figure 1 shows two examples of such nearly disjoint distributions . In contrast , COLES by design avoids the sigmoid in favour of the Radial Basis Function ( RBF ) non-linearity . We show that such a choice coincides with a surrogate of Wasserstein distance , which is known for its robustness under poor overlap of distributions , leading to the good performance of COLES . Moreover , we also show that the loss of COLES belongs to the family of so-called blockcontrastive losses , which were shown to be superior compared to pair-wise losses [ 3 ] . In summary , our contributions are threefold : i . We derive COLES , a reformulation of the Laplacian Eigenmaps into a contrastive setting , based on the SampledNCE framework [ 33 , 28 ] . ii . By using a formulation inspired by GAN , we show that COLES essentially minimizes a surrogate of Wasserstein distance , as opposed to the Jensen-Shannon ( JS ) divergence emerging in traditional contrastive learning . Specifically , by showing the Lipschitz continuous nature of our formulation , we prove that our formulation enjoys the Kantorovich-Rubinstein duality for the Wasserstein distance . iii . We show COLES enjoys a block-contrastive loss known to outperform pair-wise losses [ 3 ] . Novelty . We propose a simple way to obtain contrastive parametric graph embeddings which works with numerous backbones . For instance , we obtain spectral graph embeddings by combining COLES with SGC [ 49 ] and S2GC [ 61 ] , which is solved by the SVD decomposition . 2 Preliminaries . Notations . Let G= ( V , E ) be a simple , connected and undirected graph with n= |V | nodes and m= |E| edges . Let i ∈ { 1 , · · · , n } be the node index of G , and dj be the degree of node j of G. Let W be the adjacency matrix , and D be the diagonal matrix containing degrees of nodes . Moreover , let X ∈ Rn×d denote the node feature matrix where each node v is associated with a feature vector xv ∈ Rd . Let the normalized graph Laplacian matrix be defined as L = I−D−1/2ŴD−1/2 ∈ Sn+ , a symmetric positive semi-definite matrix . Finally , scalars and vectors are denoted by lowercase regular and bold fonts , respectively . Matrices are denoted by uppercase bold fonts . 2.1 Negative Sampling . SampledNCE [ 14 , 33 , 28 ] , a contrastive learning framework , is used by numerous works [ 41 , 15 , 50 ] . Let pd ( u|v ) and pn ( u′|v ) be the so-called data and negative distributions given the so-called anchor node v , where u and u′ denote the node for a positive and negative sample , respectively . Let pd ( v ) be the anchor distribution . Given some loss components sΘ ( v , u ) and s̃Θ ( v , u′ ) whose role is to evaluate the similarity for pairs ( v , u ) and ( v , u′ ) , the contrastive loss is typically given as : J ( Θ ) = Ev∼pd ( v ) [ Eu∼pd ( u|v ) sΘ ( v , u ) + ηEu′∼pn ( u′|v ) s̃Θ ( v , u ′ ) ] , ( 2 ) where η > 0 controls the impact of negative sampling . Let u ∈ Rd′ be the embedding of the node u obtained with an encoder fΘ ( xu ) given parameters Θ , where xu ∈ Rd is the initial node feature vector . Let u′ ∈ Rd′ and v ∈ Rd′ be embeddings of nodes u′ and v , accordingly . Let sΘ ( u , v ) = log σ ( u > v ) and s̃Θ ( u′ , v ) = log ( 1−σ ( u′ > v ) ) , where σ ( · ) is the sigmoid function . Subsequently , one obtains the contrastive objective ( to be maximized ) , employed by LINE [ 41 ] , REFINE [ 60 ] , GraphSAGE [ 15 ] and many other methods according to Yang et al . [ 50 ] : J ( Θ ) = Ev∼pd ( v ) [ Eu∼pd ( u|v ) log σ ( u > v ) + ηEu′∼pn ( u′|v ) log σ ( −u ′ > v ) ] . ( 3 ) In what follows , we argue that the choice of sigmoid for σ ( · ) leads to negative consequences . Thus , we derive COLES under a different choice of sΘ ( v , u ) and s̃Θ ( v , u′ ) . 3 Methodology . In what follows , we depart from the above setting of ( typical ) contrastive sampling , which results in a derivation of our COntrastive Laplacian EigenmapS ( COLES ) . 3.1 Contrastive Laplacian Eigenmaps . Instead of log-sigmoid used in sΘ ( v , u ) and s̃Θ ( v , u′ ) of Eq . ( 3 ) , let us substitute sΘ ( v , u ) = log exp ( u > v ) = u > v and s̃Θ ( v , u′ ) = log exp ( −u′ > v ) = −u′ > v into Eq . ( 2 ) , which yields : J ( Θ ) =Ev∼pd ( v ) [ Eu∼pd ( u|v ) ( u > v ) + ηEu′∼pn ( u′|v ) ( −u′ > v ) ] . ( 4 ) We assume that variables of the above objective ( to maximize ) can be constrained ( e.g. , by the ` 2 norms to prevent ill-posed solutions ) and represented by degree-normalized adjacency matrices . Next , we cast Eq . ( 4 ) into the objective of COLES ( refer to our Suppl . Material for derivations ) : Y∗ = argmin Y , s.t . Y > Y=I Tr ( Y > LY ) − η ′ κ κ∑ k=1 Tr ( Y > L ( − ) k Y ) = argmax Y , s.t . Y > Y=I Tr ( Y > ∆WY ) where ∆W=W ( + ) − η ′ κ κ∑ k=1 W ( − ) k , ( 5 ) and the rows of matrix Y ∈ Rn×d′ contain the embedding vectors , L ( − ) k for k = 1 , · · · , κ are randomly generated degree-normalized Laplacian matrices capturing the negative sampling , L ( − ) k = I−W ( − ) k and L=I−W ( + ) . The scalar 0 ≤ η′ ≤ 1 ensures that L− η′ κ ∑η′ k=1 L ( − ) k ∈ Sn+ ( one could truncate the negative spectrum instead ) and controls the impact of W ( − ) k . We note that COLES minimizes over the standard Laplacian Eigenmap while maximizing over the randomized Laplacian Eigenmap , which alleviates the lack of negative sampling in the original Laplacian Eigenmaps . However , unlike Laplacian Eigenmaps , we do not optimize over free variables Y but over the network parameters , as in Eq . ( 1 ) and ( 6 ) . Clearly , if η′=0 and Y are free variables , Eq . ( 5 ) reduces to standard Laplacian Eigenmaps [ 5 ] : Y∗= argminY , s.t . Y > Y=I Tr ( Y > LY ) . COLES for Linear Graph Networks . In what follows , we are especially interested in the lightweight family of LGNs such as SGC [ 49 ] and S2GC [ 61 ] ( APPNP [ 23 ] with the linear activation could be another choice ) whose COLES-based objective can be reformulated as : P∗ = argmax P , s.t . PP > =I Tr ( PX > F > ∆WFXP > ) . ( 6 ) F ∈ Rn×n in Eq . ( 6 ) is the so-called spectral filter operating on the ( degree-normalized ) graph adjacency matrix , and P ∈ Rd′×d is a unitary projection matrix such that 0 < d′ < d. The solution to Eq . ( 6 ) can be readily obtained by solving the generalized eigenvalue problem X > F > ∆WFXp = λp ( an SVD on a small d × d matrix ( X > F > ∆WFX ) ∈ Sd+ ) . This step results in a matrix of embeddings fP ( X ) = FXP > ∈ Rn×d ′ for supervised training . Based on given a degree-normalized graph adjacency matrix W ∈ Rn×n , the spectral filters for SGC and S2GC are given as WK′ and αI + 1−αK′ ∑K′ k=1 W k. Here , integer K ′ ≥ 1 and scalar α ≥ 0 are the number of layers and the importance of self-loop . Note that Eq . ( 6 ) is related to Locality Preserving Projections [ 17 ] if η′ = 0 . Note also that enforcing the orthogonality constraints in Eq . ( 6 ) coincides with the SVD-based solution described above . In contrast , the more general form of COLES in Eq . ( 1 ) requires the regularization or constraints ( depending on the backbone ) imposed on minibatches i ∈ B e.g. , we used the soft penalty Ω ( fΘ ( Xi ) ) =‖f > Θ ( Xi ) fΘ ( Xi ) − I‖2F . COLES ( Stiefel ) . Inspired by the Locality Preserving Projections [ 17 ] and Eq . ( 6 ) , we also investigate : ( P∗ , Θ∗ ) = argmax P , Θ , s.t . PP > =I Tr ( Pf > Θ ( X ) ∆WfΘ ( X ) P > ) , ( 7 ) solved on the Stiefel manifold by GeoTorch [ 29 ] . The embed . is : fP ( X ) = fΘ ( X ) P > ∈ Rn×d ′ .

Summary: This paper provides a reformulation of the traditional Laplacian Eigenmaps in a contrastive setting, for graph nodes embedding. Theoretical results for its connection to Wasserstein distance and block-contrastive loss are presented. Overall, I think this is a good paper with substantial contributions in connecting manifold learning and constructive learning.