From Canonical Correlation Analysis to Self-supervised Graph Neural Networks

1 Introduction . Self-supervised learning ( SSL ) has been a promising paradigm for learning useful representations without costly labels [ 7 , 46 , 5 ] . In general , it learns representations via a proxy objective between inputs and self-defined signals , among which contrastive methods [ 46 , 40 , 16 , 5 , 12 ] have achieved impressive performance on learning image representations by maximizing the mutual information of two views ( or augmentations ) of the same input . Such methods can be interpreted as a discrimination of a joint distribution ( positive pairs ) from the product of two marginal ones ( negative pairs ) [ 50 ] . Inspired by the success of contrastive learning in vision [ 17 , 46 , 40 , 5 , 16 , 12 , 6 ] , similar methods have been adapted to learning graph neural networks [ 48 , 15 , 33 , 57 , 58 ] . Although these models have achieved impressive performance , they require complex designs and architectures . For example , DGI [ 48 ] and MVGRL [ 15 ] rely on a parameterized mutual information estimator to discriminate positive node-graph pairs from negative ones ; GRACE [ 57 ] and GCA [ 58 ] harness an additional MLP-projector to guarantee sufficient capacity . Moreover , negative pairs sampled or constructed from data often play an indispensable role in providing effective contrastive signals and have a large impact on performance . Selecting proper negative samples is often nontrivial for graph-structured data , not to mention the extra storage cost for prohibitively large graphs . BGRL [ 39 ] is a recent endeavor on ⇤This work was done during the author ’ s internship at AWS Shanghai AI Lab . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . targeting a negative-sample-free approach for GNN learning through asymmetric architectures [ 12 , 6 ] . However , it requires additional components , e.g. , an exponential moving average ( EMA ) and StopGradient , to empirically avoid degenerated solutions , leading to a more intricate architecture . Deviating from the large body of previous works on contrastive learning , in this paper we take a new perspective to address SSL on graphs . We introduce Canonical Correlation Analysis inspired Self-Supervised Learning on Graphs ( CCA-SSG ) , a simple yet effective approach that opens the way to a new SSL objective and frees the model from intricate designs . It follows the common practice of prior arts , generating two views of an input graph through random augmentation and acquiring node representations through a shared GNN encoder . Differently , we propose to harness a non-contrastive and non-discriminative feature-level objective , which is inspired by the well-studied Canonical Correlation Analysis ( CCA ) methods [ 18 , 10 , 11 , 14 , 2 , 4 ] . More specifically , the new objective aims at maximizing the correlation between two augmented views of the same input and meanwhile decorrelating different ( feature ) dimensions of a single view ’ s representation . We show that the objective 1 ) essentially pursuits discarding augmentation-variant information and preserving augmentation-invariant information , and 2 ) can prevent dimensional collapse [ 19 ] ( i.e. , different dimensions capture the same information ) in nature . Furthermore , our theoretical analysis sheds more lights that under mild assumptions , our model is an instantiation of Information Bottleneck Principle [ 43 , 44 , 37 ] under SSL settings [ 53 , 9 , 45 ] . To sum up , as shown in Table 1 , our new objective induces a simple and light model without reliance on negative pairs [ 48 , 15 , 57 , 58 ] , a parameterized mutual information estimator [ 48 , 15 ] , an additional projector or predictor [ 57 , 58 , 39 ] or asymmetric architectures [ 39 , 15 ] . We provide a thorough evaluation for the model on seven node classification benchmarks . The empirical results demonstrate that despite its simplicity , CCA-SSG can achieve very competitive performance in general and even superior test accuracy in five datasets . It is worth noting that our approach is agnostic to the input data format , which means that it can potentially be applied to other scenarios beyond graph-structured data ( such as vision , language , etc. ) . We leave such a technical extension for future works . Our contributions are as follows : 1 ) We introduce a non-contrastive and non-discriminative objective for self-supervised learning , which is inspired by Canonical Correlation Analysis methods . It does not rely on negative samples , and can naturally remove the complicated components . Based on it we propose CCA-SSG , a simple yet effective framework for learning node representations without supervision ( see Section 3 ) . 2 ) We theoretically prove that the proposed objective aims at keeping augmentation-invariant information while discarding augmentation-variant one , and possesses an inherent relationship to an embodiment of Information Bottleneck Principle under self-supervised settings ( see Section 4 ) . 3 ) Experimental results show that without complex designs , our method outperforms state-of-the-art self-supervised methods MVGRL [ 15 ] and GCA [ 58 ] on 5 out of 7 benchmarks . We also provide thorough ablation studies on the effectiveness of the key components of CCA-SSG ( see Section 5 ) . 2 Related Works and Background . Contrastive Learning on Graphs . Contrastive methods [ 46 , 40 , 17 , 16 , 5 , 12 ] have been shown to be effective for unsupervised learning in vision , which have also been adapted to graphs . Inspired by the local-global mutual information maximization viewpoints [ 17 ] , DGI [ 48 ] and InfoGraph [ 38 ] put forward unsupervised schemes for node and graph representation learning , respectively . MVGRL [ 15 ] generalizes CMC [ 40 ] to graph-structured data by introducing graph diffusion [ 23 ] to create another view for a graph . GCC [ 33 ] adopts InfoNCE loss [ 46 ] and MoCo-based negative pool [ 16 ] for largescale GNN pretraining . GRACE [ 57 ] , GCA [ 58 ] and GraphCL [ 52 ] follow the spirit of SimCLR [ 5 ] and learn node/graph representations by directly treating other nodes/graphs as negative samples . BGRL [ 39 ] targets a negative-sample-free model , inspired by BYOL [ 12 ] , on node representation learning . But it still requires complex asymmetric architectures . Feature-level Self-supervised Objectives . The above-mentioned methods all focus on instancelevel contrastive learning . To address their drawbacks , some recent works have been turning to feature-level objectives . For example , Contrastive Clustering [ 25 ] regards different feature dimensions as different clusters , thus combining the cluster-level discrimination with instance-level discrimination . W-MSE [ 8 ] performs a differentiable whitening operation on learned embeddings , which implicitly scatters data points in embedding space . Barlow Twins [ 53 ] borrows the idea of redundancy reduction and adopts a soft decorrelation term that makes the cross-correlation matrix of two views ’ representations close to an identity matrix . By contrast , our method is based on the classical Canonical Correlation Analysis , working by correlating the representations of two views from data augmentation and meanwhile decorrelating different feature dimensions of each view ’ s representation . Canonical Correlation Analysis . CCA is a classical multivariate analysis method , which is first introduced in [ 18 ] . For two random variables X 2 Rm and Y 2 Rn , their covariance matrix is ⌃XY = Cov ( X , Y ) . CCA aims at seeking two vectors a 2 Rm and b 2 Rn such that the correlation ⇢ = corr ( a > X , b > Y ) = a > ⌃XY bp a > ⌃XXa p b > ⌃Y Y b is maximized . Formally , the objective is max a , b a > ⌃XY b , s.t . a > ⌃XXa = b > ⌃Y Y b = 1 . ( 1 ) For multi-dimensional cases , CCA seeks two sets of vectors maximizing their correlation and subjected to the constraint that they are uncorrelated with each other [ 10 ] . Later studies apply CCA to multi-view learning with deep models [ 2 , 11 , 14 ] , by replacing the linear transformation with neural networks . Concretely , assuming X1 , X2 as two views of an input data , it optimizes max ✓1 , ✓2 Tr P > ✓1 ( X1 ) P✓2 ( X2 ) s.t . P > ✓1 ( X1 ) P✓1 ( X1 ) = P > ✓2 ( X2 ) P✓2 ( X2 ) = I . ( 2 ) where P✓1 and P✓2 are two feedforward neural networks and I is an identity matrix . Despite its preciseness , such computation is really expensive [ 4 ] . Fortunately , soft CCA [ 4 ] removes the hard decorrelation constraint by adopting the following Lagrangian relaxation : min ✓1 , ✓2 Ldist ( P✓1 ( X1 ) , P✓2 ( X2 ) ) + ( LSDL ( P✓1 ( X1 ) ) + LSDL ( P✓2 ( X2 ) ) ) , ( 3 ) where Ldist measures correlation between two views ’ representations and LSDL ( called stochastic decorrelation loss ) computes an L1 distance between P✓i ( Xi ) and an identity matrix , for i = 1 , 2 . 3 Approach . 3.1 Model Framework . In this paper we focus on self-supervised node representation learning , where we consider a single graph G = ( X , A ) . X 2 RN⇥F and A 2 RN⇥N denote node features and adjacency matrix respectively . Here N is the number of nodes within the graph and F denotes feature dimension . Our model simply consists of three parts : 1 ) a random graph augmentation generator T . 2 ) a GNNbased graph encoder f✓ where ✓ denotes its parameters . 3 ) a novel feature-level objective function based on Canonical Correlation Analysis . Fig . 1 is an illustration of the proposed model . Algorithm 1 : PyTorch-style code for CCA-SSG # f : encoder network # lambda : trade-off # D : embedding dimension # g : input graph # feat : node features # generate two views through random augmentation g1 , feat1 = augment ( g , feat ) g2 , feat2 = augment ( g , feat ) z1 = f ( g1 , feat1 ) # embedding of the 1st view z2 = f ( g2 , feat2 ) # embedding of the 2st view # batch normalization z1_norm = ( ( z1 - z1.mean ( 0 ) ) / z1.std ( 0 ) ) / sqrt ( N ) z2_norm = ( ( z2 - z2.mean ( 0 ) ) / z2.std ( 0 ) ) / sqrt ( N ) # covariance matrix of each view c1 = torch.mm ( z1_norm.T ( ) , z1_norm ) c2 = torch.mm ( z2_norm.T ( ) , z2_norm ) iden = torch.eye ( D ) loss_inv = ( z1_norm - z2_norm ) .pow ( 2 ) .sum ( ) loss_dec_1 = ( c1 - iden ) .pow ( 2 ) .sum ( ) loss_dec_2 = ( c2 - iden ) .pow ( 2 ) .sum ( ) loss_dec = loss_dec_1 + loss_dec_2 loss = loss_inv + lambda * loss_dec Graph augmentations . We consider the standard pipeline for random graph augmentation that has been commonly used in previous works [ 57 , 39 ] . To be specific , we harness two ways for augmentation : edge dropping and node feature masking . Edge dropping randomly drops a fraction of edges from the original graph , while node feature masking randomly masks a fraction of features for all the nodes . In this way , T is composed of all the possible graph transformation operations and each t ⇠ T denotes a specific graph transformation for graph G. Note that we use commonly adopted augmentation methods to stay our focus on the design of objective function and conduct fair comparison with existing approaches . More complicated random augmentations [ 52 , 58 ] can also be readily plugged into our model . Details for the used augmentation functions are in Appendix E. Training . In each training iteration , we first randomly sample two graph transformations tA and tB from T , and then generate two views G̃A = ( X̃A , ÃA ) and G̃B = ( X̃B , ÃB ) according to the transformations . The two views are subsequently fed into a shared GNN encoder to generate the node embeddings of the two views : ZA = f✓ ( X̃A , ÃA ) , ZB = f✓ ( X̃B , ÃB ) , where ZA , ZB 2 RN⇥D and D denotes embedding dimension . We further normalize the node embeddings along instance dimension so that each feature dimension has a 0-mean and 1/ p N -standard deviation distribution : Z̃ = Z µ ( Z ) ( Z ) ⇤ p N ( 4 ) The normalized Z̃A , Z̃B will be used to compute a feature-level objective in Section 3.2 . To help better understand the proposed framework , we provide the PyTorch-style pseudocode for training CCA-SSG in Algorithm 1 . Inference . To generate node embeddings for downstream tasks , we put the original graph G = ( X , A ) into the trained graph neural network f✓ and obtain node embeddings Z = f✓ ( X , A ) .

Summary: This paper focuses on self-supervised graph neural networks, aiming at training graph neural networks without labeled data, and an approach called the CCA-SSG is proposed. CCA-SSG constructs two views of the given graph through edge dropping and node feature masking, and further uses CCA for learning node representations from the two views. The idea of formalizing unsupervised node representation learning as a multi-view learning task looks novel to me. Also, CCA-SSG has intuitive connections with the information bottleneck principle, which is attractive. The authors do experiment on multiple datasets, and CCA-SSG outperforms many competitive baseline methods.  Strengths: 1. Important problem. 2. Principled method with good theoretical connections. 3. Strong results.