Contrastive Laplacian Eigenmaps

1 Introduction . Celebrated graph embedding methods , including Laplacian Eigenmaps [ 5 ] and IsoMap [ 42 ] , reduce the dimensionality of the data by assuming that it lies on a low-dimensional manifold . The objective functions used in studies [ 5 , 42 ] model the pairwise node similarity [ 7 ] by encouraging the embeddings of nodes to lie close in the embedding space if the nodes are closely related . In other words , such penalties do not guarantee that unrelated graph nodes are separated from each other in the embedding space . For instance , Elastic Embedding [ 8 ] uses data-driven affinities for the so-called local distance term and the data-independent repulsion term . In contrast , modern graph embedding models , often unified under the Sampled Noise Contrastive Estimation ( SampledNCE ) framework [ 33 , 28 ] and extended to graph learning [ 41 , 15 , 50 ] , enjoy contrastive objectives . By maximizing the mutual information between patch representations and highlevel summaries of the graph , Deep Graph Infomax ( DGI ) [ 43 ] is a contrastive method . GraphSAGE [ 15 ] minimizes/maximizes distances between so-called positive/negative pairs , respectively . It relies on the inner product passed through the sigmoid non-linearity , which we argue below as suboptimal . Thus , we propose a new COntrastive Laplacian EigenmapS ( COLES ) framework for unsupervised network embedding . COLES , derived from SampledNCE framework [ 33 , 28 ] , realizes the negative sampling strategy for Laplacian Eigenmaps . Our general objective is given as : Θ∗ = argmax Θ Tr ( fΘ ( X ) > ∆WfΘ ( X ) ) + βΩ ( fΘ ( X ) ) . ( 1 ) X ∈ Rn×d in Eq . ( 1 ) is the node feature matrix with d feature dimensions given n nodes , fΘ ( X ) ∈ Rn×d′ is an output of a chosen Graph Neural Network backbone ( embeddings to optimize ) with the * The corresponding author . Code : https : //github.com/allenhaozhu/COLES . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . feature dimension d′ , Θ denotes network parameters , whereas ∆W ∈ Sn+ is the difference between the degree-normalized positive and negative adjacency matrices which represent the data graph and some negative graph capturing negative links for contrastive learning . Moreover , β ≥ 0 controls the regularization term Ω ( · ) whose role is to constrain the ` 2 norm of network outputs or encourage the so-called incoherence [ 36 ] between column vectors . Section 3.1 presents COLES for the Linear Graph Network ( LGN ) family , in which we take special interest due to their simplicity and agility . By building upon previous studies [ 28 , 2 , 48 ] , we show that COLES can be derived by reformulating SampledNCE into Wasserstein GAN using a GAN-inspired contrastive formulation . This result has a profound impact on the performance of COLES , as the standard contrastive approaches based on SampledNCE strategy ( i.e. , GraphSAGE [ 15 ] ) turn out to utilize the Jensen-Shannon divergence , which yields log 2 constant and vanishing gradients for disjoint distributions of positive and negative sampled pairs used for contrastive learning . Figure 1 shows two examples of such nearly disjoint distributions . In contrast , COLES by design avoids the sigmoid in favour of the Radial Basis Function ( RBF ) non-linearity . We show that such a choice coincides with a surrogate of Wasserstein distance , which is known for its robustness under poor overlap of distributions , leading to the good performance of COLES . Moreover , we also show that the loss of COLES belongs to the family of so-called blockcontrastive losses , which were shown to be superior compared to pair-wise losses [ 3 ] . In summary , our contributions are threefold : i . We derive COLES , a reformulation of the Laplacian Eigenmaps into a contrastive setting , based on the SampledNCE framework [ 33 , 28 ] . ii . By using a formulation inspired by GAN , we show that COLES essentially minimizes a surrogate of Wasserstein distance , as opposed to the Jensen-Shannon ( JS ) divergence emerging in traditional contrastive learning . Specifically , by showing the Lipschitz continuous nature of our formulation , we prove that our formulation enjoys the Kantorovich-Rubinstein duality for the Wasserstein distance . iii . We show COLES enjoys a block-contrastive loss known to outperform pair-wise losses [ 3 ] . Novelty . We propose a simple way to obtain contrastive parametric graph embeddings which works with numerous backbones . For instance , we obtain spectral graph embeddings by combining COLES with SGC [ 49 ] and S2GC [ 61 ] , which is solved by the SVD decomposition . 2 Preliminaries . Notations . Let G= ( V , E ) be a simple , connected and undirected graph with n= |V | nodes and m= |E| edges . Let i ∈ { 1 , · · · , n } be the node index of G , and dj be the degree of node j of G. Let W be the adjacency matrix , and D be the diagonal matrix containing degrees of nodes . Moreover , let X ∈ Rn×d denote the node feature matrix where each node v is associated with a feature vector xv ∈ Rd . Let the normalized graph Laplacian matrix be defined as L = I−D−1/2ŴD−1/2 ∈ Sn+ , a symmetric positive semi-definite matrix . Finally , scalars and vectors are denoted by lowercase regular and bold fonts , respectively . Matrices are denoted by uppercase bold fonts . 2.1 Negative Sampling . SampledNCE [ 14 , 33 , 28 ] , a contrastive learning framework , is used by numerous works [ 41 , 15 , 50 ] . Let pd ( u|v ) and pn ( u′|v ) be the so-called data and negative distributions given the so-called anchor node v , where u and u′ denote the node for a positive and negative sample , respectively . Let pd ( v ) be the anchor distribution . Given some loss components sΘ ( v , u ) and s̃Θ ( v , u′ ) whose role is to evaluate the similarity for pairs ( v , u ) and ( v , u′ ) , the contrastive loss is typically given as : J ( Θ ) = Ev∼pd ( v ) [ Eu∼pd ( u|v ) sΘ ( v , u ) + ηEu′∼pn ( u′|v ) s̃Θ ( v , u ′ ) ] , ( 2 ) where η > 0 controls the impact of negative sampling . Let u ∈ Rd′ be the embedding of the node u obtained with an encoder fΘ ( xu ) given parameters Θ , where xu ∈ Rd is the initial node feature vector . Let u′ ∈ Rd′ and v ∈ Rd′ be embeddings of nodes u′ and v , accordingly . Let sΘ ( u , v ) = log σ ( u > v ) and s̃Θ ( u′ , v ) = log ( 1−σ ( u′ > v ) ) , where σ ( · ) is the sigmoid function . Subsequently , one obtains the contrastive objective ( to be maximized ) , employed by LINE [ 41 ] , REFINE [ 60 ] , GraphSAGE [ 15 ] and many other methods according to Yang et al . [ 50 ] : J ( Θ ) = Ev∼pd ( v ) [ Eu∼pd ( u|v ) log σ ( u > v ) + ηEu′∼pn ( u′|v ) log σ ( −u ′ > v ) ] . ( 3 ) In what follows , we argue that the choice of sigmoid for σ ( · ) leads to negative consequences . Thus , we derive COLES under a different choice of sΘ ( v , u ) and s̃Θ ( v , u′ ) . 3 Methodology . In what follows , we depart from the above setting of ( typical ) contrastive sampling , which results in a derivation of our COntrastive Laplacian EigenmapS ( COLES ) . 3.1 Contrastive Laplacian Eigenmaps . Instead of log-sigmoid used in sΘ ( v , u ) and s̃Θ ( v , u′ ) of Eq . ( 3 ) , let us substitute sΘ ( v , u ) = log exp ( u > v ) = u > v and s̃Θ ( v , u′ ) = log exp ( −u′ > v ) = −u′ > v into Eq . ( 2 ) , which yields : J ( Θ ) =Ev∼pd ( v ) [ Eu∼pd ( u|v ) ( u > v ) + ηEu′∼pn ( u′|v ) ( −u′ > v ) ] . ( 4 ) We assume that variables of the above objective ( to maximize ) can be constrained ( e.g. , by the ` 2 norms to prevent ill-posed solutions ) and represented by degree-normalized adjacency matrices . Next , we cast Eq . ( 4 ) into the objective of COLES ( refer to our Suppl . Material for derivations ) : Y∗ = argmin Y , s.t . Y > Y=I Tr ( Y > LY ) − η ′ κ κ∑ k=1 Tr ( Y > L ( − ) k Y ) = argmax Y , s.t . Y > Y=I Tr ( Y > ∆WY ) where ∆W=W ( + ) − η ′ κ κ∑ k=1 W ( − ) k , ( 5 ) and the rows of matrix Y ∈ Rn×d′ contain the embedding vectors , L ( − ) k for k = 1 , · · · , κ are randomly generated degree-normalized Laplacian matrices capturing the negative sampling , L ( − ) k = I−W ( − ) k and L=I−W ( + ) . The scalar 0 ≤ η′ ≤ 1 ensures that L− η′ κ ∑η′ k=1 L ( − ) k ∈ Sn+ ( one could truncate the negative spectrum instead ) and controls the impact of W ( − ) k . We note that COLES minimizes over the standard Laplacian Eigenmap while maximizing over the randomized Laplacian Eigenmap , which alleviates the lack of negative sampling in the original Laplacian Eigenmaps . However , unlike Laplacian Eigenmaps , we do not optimize over free variables Y but over the network parameters , as in Eq . ( 1 ) and ( 6 ) . Clearly , if η′=0 and Y are free variables , Eq . ( 5 ) reduces to standard Laplacian Eigenmaps [ 5 ] : Y∗= argminY , s.t . Y > Y=I Tr ( Y > LY ) . COLES for Linear Graph Networks . In what follows , we are especially interested in the lightweight family of LGNs such as SGC [ 49 ] and S2GC [ 61 ] ( APPNP [ 23 ] with the linear activation could be another choice ) whose COLES-based objective can be reformulated as : P∗ = argmax P , s.t . PP > =I Tr ( PX > F > ∆WFXP > ) . ( 6 ) F ∈ Rn×n in Eq . ( 6 ) is the so-called spectral filter operating on the ( degree-normalized ) graph adjacency matrix , and P ∈ Rd′×d is a unitary projection matrix such that 0 < d′ < d. The solution to Eq . ( 6 ) can be readily obtained by solving the generalized eigenvalue problem X > F > ∆WFXp = λp ( an SVD on a small d × d matrix ( X > F > ∆WFX ) ∈ Sd+ ) . This step results in a matrix of embeddings fP ( X ) = FXP > ∈ Rn×d ′ for supervised training . Based on given a degree-normalized graph adjacency matrix W ∈ Rn×n , the spectral filters for SGC and S2GC are given as WK′ and αI + 1−αK′ ∑K′ k=1 W k. Here , integer K ′ ≥ 1 and scalar α ≥ 0 are the number of layers and the importance of self-loop . Note that Eq . ( 6 ) is related to Locality Preserving Projections [ 17 ] if η′ = 0 . Note also that enforcing the orthogonality constraints in Eq . ( 6 ) coincides with the SVD-based solution described above . In contrast , the more general form of COLES in Eq . ( 1 ) requires the regularization or constraints ( depending on the backbone ) imposed on minibatches i ∈ B e.g. , we used the soft penalty Ω ( fΘ ( Xi ) ) =‖f > Θ ( Xi ) fΘ ( Xi ) − I‖2F . COLES ( Stiefel ) . Inspired by the Locality Preserving Projections [ 17 ] and Eq . ( 6 ) , we also investigate : ( P∗ , Θ∗ ) = argmax P , Θ , s.t . PP > =I Tr ( Pf > Θ ( X ) ∆WfΘ ( X ) P > ) , ( 7 ) solved on the Stiefel manifold by GeoTorch [ 29 ] . The embed . is : fP ( X ) = fΘ ( X ) P > ∈ Rn×d ′ .

Summary: The paper proposes a reformulation of Laplacian Eigenmaps in a log-linear contrastive setting. Inspired by GAN architecture, the authors show that the their contrastive laplacian eigenmaps (COLES) minimises a surrogate of Wasserstein distance as opposed to JS distance in traditional contrastive methods. Further, COLES in shown to enjoy block contrastive loss, which is upper bounded by pair-wise loss.   The authors show that the log-linear contrastive loss can be reformulated as a generalised eigenvalue (GEV) problem of a contrastive degree normalised adjacency matrix. They apply this formulation to line graph networks such as SGC and $S^{2}$GC and obtain a closed form solution to the matrix of node embeddings, which depends on a spectral filter and the solution to the GEV problem.   By casting the positive and negative samples as the real and generated data from a GAN, the authors show that traditional contrastive losses can be reformulated into JS divergence, while COLES can be cast into a surrogate of 1-Wasserstein  metric. They claim that the latter is preferable since, under poor overlap of the the two distributions JS divergence vanishes, leading to vanishing gradients, while 1-Wasserstein distance does not.   The authors also show that COLES is Lipschitz continuous and enjoys block contrastive loss, wherein one can leverage blocks of similar data as opposed individual pairs. Since from [Saunshi et al] block contrastive loss is smaller than pair-wise loss, the authors claim that COLES is expected to achieve a lower minimum.   The authors perform empirical evaluation on a host of open source datasets and multiple baseline backbones. In most tasks, COLES objective seems to outperform the methods included in the comparison.