From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
Year: 2022

Abstract:
We introduce a conceptually simple yet effective model for self-supervised representation learning with graph data. It follows the previous methods that generate two views of an input graph through data augmentation. However, unlike contrastive methods that focus on instance-level discrimination, we optimize an innovative feature-level objective inspired by classical Canonical Correlation Analysis. Compared with other works, our approach requires none of the parameterized mutual information estimator, additional projector, asymmetric structures, and most importantly, negative samples which can be costly. We show that the new objective essentially 1) aims at discarding augmentation-variant information by learning invariant representations, and 2) can prevent degenerated solutions by decorrelating features in different dimensions. Our theoretical analysis further provides an understanding for the new objective which can be equivalently seen as an instantiation of the Information Bottleneck Principle under the self-supervised setting. Despite its simplicity, our method performs competitively on seven public graph datasets. The code is available at: https://github.com/hengruizhang98/CCA-SSG.

heading: 1 Introduction
text: Self-supervised learning (SSL) has been a promising paradigm for learning useful representations without costly labels [7, 46, 5]. In general, it learns representations via a proxy objective between inputs and self-defined signals, among which contrastive methods [46, 40, 16, 5, 12] have achieved impressive performance on learning image representations by maximizing the mutual information of two views (or augmentations) of the same input. Such methods can be interpreted as a discrimination of a joint distribution (positive pairs) from the product of two marginal ones (negative pairs) [50].
Inspired by the success of contrastive learning in vision [17, 46, 40, 5, 16, 12, 6], similar methods have been adapted to learning graph neural networks [48, 15, 33, 57, 58]. Although these models have achieved impressive performance, they require complex designs and architectures. For example, DGI [48] and MVGRL [15] rely on a parameterized mutual information estimator to discriminate positive node-graph pairs from negative ones; GRACE [57] and GCA [58] harness an additional MLP-projector to guarantee sufficient capacity. Moreover, negative pairs sampled or constructed from data often play an indispensable role in providing effective contrastive signals and have a large impact on performance. Selecting proper negative samples is often nontrivial for graph-structured data, not to mention the extra storage cost for prohibitively large graphs. BGRL [39] is a recent endeavor on
⇤This work was done during the author’s internship at AWS Shanghai AI Lab. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
targeting a negative-sample-free approach for GNN learning through asymmetric architectures [12, 6]. However, it requires additional components, e.g., an exponential moving average (EMA) and StopGradient, to empirically avoid degenerated solutions, leading to a more intricate architecture.
Deviating from the large body of previous works on contrastive learning, in this paper we take a new perspective to address SSL on graphs. We introduce Canonical Correlation Analysis inspired Self-Supervised Learning on Graphs (CCA-SSG), a simple yet effective approach that opens the way to a new SSL objective and frees the model from intricate designs. It follows the common practice of prior arts, generating two views of an input graph through random augmentation and acquiring node representations through a shared GNN encoder. Differently, we propose to harness a non-contrastive and non-discriminative feature-level objective, which is inspired by the well-studied Canonical Correlation Analysis (CCA) methods [18, 10, 11, 14, 2, 4]. More specifically, the new objective aims at maximizing the correlation between two augmented views of the same input and meanwhile decorrelating different (feature) dimensions of a single view’s representation. We show that the objective 1) essentially pursuits discarding augmentation-variant information and preserving augmentation-invariant information, and 2) can prevent dimensional collapse [19] (i.e., different dimensions capture the same information) in nature. Furthermore, our theoretical analysis sheds more lights that under mild assumptions, our model is an instantiation of Information Bottleneck Principle [43, 44, 37] under SSL settings [53, 9, 45].
To sum up, as shown in Table 1, our new objective induces a simple and light model without reliance on negative pairs [48, 15, 57, 58], a parameterized mutual information estimator [48, 15], an additional projector or predictor [57, 58, 39] or asymmetric architectures [39, 15]. We provide a thorough evaluation for the model on seven node classification benchmarks. The empirical results demonstrate that despite its simplicity, CCA-SSG can achieve very competitive performance in general and even superior test accuracy in five datasets. It is worth noting that our approach is agnostic to the input data format, which means that it can potentially be applied to other scenarios beyond graph-structured data (such as vision, language, etc.). We leave such a technical extension for future works.
Our contributions are as follows: 1) We introduce a non-contrastive and non-discriminative objective for self-supervised learning, which is inspired by Canonical Correlation Analysis methods. It does not rely on negative samples, and can naturally remove the complicated components. Based on it we propose CCA-SSG, a simple yet effective framework for learning node representations without supervision (see Section 3). 2) We theoretically prove that the proposed objective aims at keeping augmentation-invariant information while discarding augmentation-variant one, and possesses an inherent relationship to an embodiment of Information Bottleneck Principle under self-supervised settings (see Section 4). 3) Experimental results show that without complex designs, our method outperforms state-of-the-art self-supervised methods MVGRL [15] and GCA [58] on 5 out of 7 benchmarks. We also provide thorough ablation studies on the effectiveness of the key components of CCA-SSG (see Section 5).

heading: 2 Related Works and Background
text: Contrastive Learning on Graphs. Contrastive methods [46, 40, 17, 16, 5, 12] have been shown to be effective for unsupervised learning in vision, which have also been adapted to graphs. Inspired by the local-global mutual information maximization viewpoints [17], DGI [48] and InfoGraph [38] put forward unsupervised schemes for node and graph representation learning, respectively. MVGRL [15] generalizes CMC [40] to graph-structured data by introducing graph diffusion [23] to create another view for a graph. GCC [33] adopts InfoNCE loss [46] and MoCo-based negative pool [16] for largescale GNN pretraining. GRACE [57], GCA [58] and GraphCL [52] follow the spirit of SimCLR [5] and learn node/graph representations by directly treating other nodes/graphs as negative samples. BGRL [39] targets a negative-sample-free model, inspired by BYOL [12], on node representation learning. But it still requires complex asymmetric architectures.
Feature-level Self-supervised Objectives. The above-mentioned methods all focus on instancelevel contrastive learning. To address their drawbacks, some recent works have been turning to feature-level objectives. For example, Contrastive Clustering [25] regards different feature dimensions as different clusters, thus combining the cluster-level discrimination with instance-level discrimination. W-MSE [8] performs a differentiable whitening operation on learned embeddings, which implicitly scatters data points in embedding space. Barlow Twins [53] borrows the idea of redundancy reduction and adopts a soft decorrelation term that makes the cross-correlation matrix of two views’ representations close to an identity matrix. By contrast, our method is based on the classical Canonical Correlation Analysis, working by correlating the representations of two views from data augmentation and meanwhile decorrelating different feature dimensions of each view’s representation.
Canonical Correlation Analysis. CCA is a classical multivariate analysis method, which is first introduced in [18]. For two random variables X 2 Rm and Y 2 Rn, their covariance matrix is ⌃XY = Cov(X,Y ). CCA aims at seeking two vectors a 2 Rm and b 2 Rn such that the correlation ⇢ = corr(a>X, b>Y ) = a
>⌃XY bp a>⌃XXa p b>⌃Y Y b is maximized. Formally, the objective is
max a,b
a>⌃XY b, s.t. a>⌃XXa = b>⌃Y Y b = 1. (1)
For multi-dimensional cases, CCA seeks two sets of vectors maximizing their correlation and subjected to the constraint that they are uncorrelated with each other [10]. Later studies apply CCA to multi-view learning with deep models [2, 11, 14], by replacing the linear transformation with neural networks. Concretely, assuming X1, X2 as two views of an input data, it optimizes
max ✓1,✓2
Tr P>✓1(X1)P✓2(X2) s.t. P>✓1(X1)P✓1(X1) = P > ✓2(X2)P✓2(X2) = I. (2)
where P✓1 and P✓2 are two feedforward neural networks and I is an identity matrix. Despite its preciseness, such computation is really expensive [4]. Fortunately, soft CCA [4] removes the hard decorrelation constraint by adopting the following Lagrangian relaxation:
min ✓1,✓2 Ldist (P✓1(X1), P✓2(X2)) + (LSDL(P✓1(X1)) + LSDL(P✓2(X2))) , (3)
where Ldist measures correlation between two views’ representations and LSDL (called stochastic decorrelation loss) computes an L1 distance between P✓i(Xi) and an identity matrix, for i = 1, 2.

heading: 3 Approach

heading: 3.1 Model Framework
text: In this paper we focus on self-supervised node representation learning, where we consider a single graph G = (X,A). X 2 RN⇥F and A 2 RN⇥N denote node features and adjacency matrix respectively. Here N is the number of nodes within the graph and F denotes feature dimension.
Our model simply consists of three parts: 1) a random graph augmentation generator T . 2) a GNNbased graph encoder f✓ where ✓ denotes its parameters. 3) a novel feature-level objective function based on Canonical Correlation Analysis. Fig. 1 is an illustration of the proposed model.
Algorithm 1: PyTorch-style code for CCA-SSG
# f: encoder network # lambda: trade-off # D: embedding dimension # g: input graph # feat: node features
# generate two views through random augmentation g1, feat1 = augment(g, feat) g2, feat2 = augment(g, feat) z1 = f(g1, feat1) # embedding of the 1st view z2 = f(g2, feat2) # embedding of the 2st view
# batch normalization z1_norm = ((z1 - z1.mean(0)) / z1.std(0))/ sqrt(N) z2_norm = ((z2 - z2.mean(0)) / z2.std(0))/ sqrt(N)
# covariance matrix of each view c1 = torch.mm(z1_norm.T(), z1_norm) c2 = torch.mm(z2_norm.T(), z2_norm)
iden = torch.eye(D) loss_inv = (z1_norm - z2_norm).pow(2).sum() loss_dec_1 = (c1 - iden).pow(2).sum() loss_dec_2 = (c2 - iden).pow(2).sum() loss_dec = loss_dec_1 + loss_dec_2 loss = loss_inv + lambda * loss_dec
Graph augmentations. We consider the standard pipeline for random graph augmentation that has been commonly used in previous works [57, 39]. To be specific, we harness two ways for augmentation: edge dropping and node feature masking. Edge dropping randomly drops a fraction of edges from the original graph, while node feature masking randomly masks a fraction of features for all the nodes. In this way, T is composed of all the possible graph transformation operations and each t ⇠ T denotes a specific graph transformation for graph G. Note that we use commonly adopted augmentation methods to stay our focus on the design of objective function and conduct fair comparison with existing approaches. More complicated random augmentations [52, 58] can also be readily plugged into our model. Details for the used augmentation functions are in Appendix E.
Training. In each training iteration, we first randomly sample two graph transformations tA and tB from T , and then generate two views G̃A = (X̃A, ÃA) and G̃B = (X̃B , ÃB) according to the transformations. The two views are subsequently fed into a shared GNN encoder to generate the node embeddings of the two views: ZA = f✓(X̃A, ÃA), ZB = f✓(X̃B , ÃB), where ZA,ZB 2 RN⇥D and D denotes embedding dimension. We further normalize the node embeddings along instance dimension so that each feature dimension has a 0-mean and 1/ p N -standard deviation distribution:
Z̃ = Z µ(Z) (Z) ⇤ p N
(4)
The normalized Z̃A, Z̃B will be used to compute a feature-level objective in Section 3.2. To help better understand the proposed framework, we provide the PyTorch-style pseudocode for training CCA-SSG in Algorithm 1.
Inference. To generate node embeddings for downstream tasks, we put the original graph G = (X,A) into the trained graph neural network f✓ and obtain node embeddings Z = f✓(X,A).

heading: 3.2 Learning Objective
text: Canonical Correlation Analysis has shown its great power in multi-view learning like instance recognition [4]. However, it still remains unexplored to leverage CCA for self-supervised learning. Note that in SSL, one generates two sets of data from the same input through transformation or random data augmentation, which could be regraded as two views of the input data. This inspires us to introduce the following objective for self-supervised representation learning:
L = Z̃A Z̃B
2
F| {z } invariance term
+ ✓ Z̃>AZ̃A I 2
F +
Z̃>BZ̃B I 2
F
◆
| {z } decorrelation term
(5)
where is a non-negative hyperparameter trading off two terms. Note that minimizing the invariance term is essentially maximizing the correlation between two views as their representations are already normalized. In SSL, as the two augmented views come randomly from the same distribution, we can adopt one encoder f✓ that is shared across two branches and seek for a regularization that encourages different feature dimensions to capture distinct semantics via the decorrelation term.
We next provide a variance-covariance perspective to the new objective, following similar lines of reasoning in [41, 42]. Assume that input data come from a distribution x ⇠ p(x) and s is a view of x through random augmentation s ⇠ paug(·|x). Denote zs as the representation of s, then minimizing the invariance term, by expectation, is to minimize the variance of the normalized representation z̃s, conditioned on x. Also, minimizing the decorrelation term is to push the off-diagonal elements of the covariance matrix (given by two z̃s’s) close to 0. Formally, we have
Linv = Z̃A Z̃B
2 F =
NX
i=1
DX
k=1
(z̃Ai,j z̃Bi,j)2 ⇠= Ex
" DX
k=1
Vs|x[z̃s,k] # ⇤ 2N, (6)
Ldec = Z̃>S Z̃S I
2 F = kCovs[z̃] Ik2F ⇠=
X
i 6=j
⇢zsi,j 2 , for Z̃S 2 {Z̃A, Z̃B}, (7)
where ⇢ is the Pearson correlation coefficient.

heading: 3.3 Advantages over Contrastive Methods
text: In this subsection we provide a systematic comparison with previous self-supervised methods for node representation learning, including DGI [48], MVGRL [15], GRACE [57], GCA [58] and BGRL [39], and highlight the merits of CCA-SSG. A quick overview is presented in Table 1.
No reliance on negative samples. Most of previous works highly rely on negative pairs to avoid collapse or interchangeable, trivial/degenerated solutions [48, 15, 57, 58]. E.g., DGI and MVGRL generate negative examples by corrupting the graph structure severely, and GRACE/GCA treats all the other nodes within a graph as negative examples. However, for self-supervised learning on graphs, it is non-trivial to construct informative negative examples since nodes are structurally connected, and selecting negative examples in an arbitrary manner may lead to large variance for stochastic gradients and slow training convergence [51]. The recently proposed BGRL model adopts asymmetric encoder architectures for SSL on graphs without the use of negative samples. However, though BGRL could avoid collapse empirically, it still remains as an open problem concerning its theoretical guarantee for preventing trivial solutions [41]. Compared with these methods, our model does not rely on negative pairs and asymmetric encoders. The feature decorrelation term can naturally prevent trivial solutions caused by the invariance term. We discuss the collapse issue detailedly in Appendix B.
No MI estimator, projector network nor asymmetric architectures. Most previous works rely on additional components besides the GNN encoder to estimate some score functions in final objectives. DGI and MVGRL require a parameterized estimator to approximate mutual information between two views, and GRACE leverages a MLP projector followed by an InfoNCE estimator. BGRL harnesses asymmetric encoder architecture which consists of EMA (Exponential Moving Average), Stop-Gradient and an additional projector. MVGRL also induces asymmetric architectures as it adopts two different GNNs for the input graph and the diffusion graph respectively. In contrast, our approach requires no additional components except a single GNN encoder.
Better efficiency and scalability to large graphs. Consider a graph with N nodes. DGI and MVGRL contrast node embeddings with graph embedding, which would require O(N) space cost.
GRACE treats two views of the same node as positive pairs and treat views of different nodes as negative pairs, which would take O(N2) space. BGRL focuses only on positive pairs, which will also take O(N) space. By contrast, our method works on feature dimension. If we embed each node into a D-dimensional vector, the computation of the loss function would require O(D2) space. This indicates that the memory cost does not grow consistently as the size of graph increases. As a result, our method is promising for handling large-scale graphs without prohibitively large space costs.

heading: 4 Theoretical Insights with Connection to Information Theory
text: In this section we provide some analysis of the proposed objective function: 1) Interpretation of the loss function with entropy and mutual information. 2) The connection between the proposed objective and the Information Bottleneck principle. 3) Why the learned representations would be informative to downstream tasks. The proofs of propositions, theorems and corollaries are in Appendix D.
Notations. Denote the random variable of input data as X and the downstream task as T (it could be the label Y if the downstream task is classification). Note that in SSL, we have no access to T in training and here we introduce the notation for our analysis. Define S as the self-supervised signal (i.e., an augmented view of X), and S shares the same space as X . Our model learns a representation for the input, denoted by ZX and its views, denoted by ZS . ZX = f✓(X), ZS = f✓(S), f✓(·) is a encoder shared by the original data and its views, which is parameterized by ✓. The target of representation learning is to learn a optimal encoder parameter ✓. Furthermore, for random variable A,B,C, we use I(A,B) to denote the mutual information between A and B, I(A,B|C) to denote conditional mutual information of A and B on a given C, H(A) for the entropy, and H(A|B) for conditional entropy. The proofs of propositions, theorems and corollaries are in Appendix D.

heading: 4.1 An Entropy and Mutual Information Interpretation of the Objective
text: We first introduce an assumption about the distributions of P (ZS) and P (ZS |X). Assumption 1. (Gaussian assumption of P (ZS |X) and P (ZS)):
P (ZS |X) = N (µX ,⌃X), P (ZS) = N (µ,⌃). (8)
With Assumption 1, we can arrive at the following propositions: Proposition 1. In expectation, minimizing Eq. (6) is equivalent to minimizing the entropy of ZS conditioned on input X , i.e.,
min ✓ Linv ⇠= min ✓ H(ZS |X). (9)
Proposition 2. Minimizing Eq. (7) is equivalent to maximizing the entropy of ZS , i.e., min ✓ Ldec ⇠= max ✓ H(ZS). (10)
The two propositions unveil the effects of two terms in our objective. Combining two propositions, we can further interpret Eq. (5) from an information-theoretic perspective. Theorem 1. By optimizing Eq (5), we maximize the mutual information between the augmented view’s embedding ZS and the input data X , and minimize the mutual information between ZS and the view itself S, conditioned on the input data X . Formally we have
min ✓ L ) max ✓ I(ZS , X) and min ✓ I(ZS , S|X). (11)
The proof is based on the facts I(ZS , X) = H(ZS) H(ZS |X) and I(ZS , S|X) = H(ZS |X) + H(ZS |S) = H(ZS |X). Theorem 1 indicates that our objective Eq. (5) learns representations that maximize the information of the input data, i.e., I(ZS , X), and meanwhile minimize the lost information during augmentation, i.e., I(ZS , S|X).

heading: 4.2 Connection with the Information Bottleneck Principle
text: The analysis in Section 4.1 enables us to further build a connection between our objective Eq. (5) and the well-studied Information Bottleneck Principle [43, 44, 37, 1] under SSL settings. Recall that the supervised Information Bottleneck (IB) is defined as follows:
Definition 1. The supervised IB aims at maximizing an Information Bottleneck Lagrangian:
IBsup = I(Y, ZX) I(X,ZX), where > 0. (12)
As we can see, IBsup attempts to maximize the information between the data representation ZX and its corresponding label Y , and concurrently minimize the information between ZX and the input data X (i.e., exploiting compression of ZX from X). The intuition of IB principle is that ZX is expected to contain only the information that is useful for predicting Y .
Several recent works [9, 45, 53] propose various forms of IB under self-supervised settings. The most relevant one names Self-supervised Information Bottleneck: Definition 2. (Self-supervised Information Bottleneck [53]). The Self-supervised IB aims at maximizing the following Lagrangian:
IBssl = I(X,ZS) I(S,ZS), where > 0. (13)
Intuitively, IBssl posits that a desirable representation is expected to be informative to augmentation invariant features, and to be a maximally compressed representation of the input.
Our objective Eq. (5) is essentially an embodiment of IBssl: Theorem 2. Assume 0 <  1, then by minimizing Eq. (5), the self-supervised Information Bottleneck objective is maximized, formally:
min ✓ L ) max ✓ IBssl (14)
Theorem 2 also shows that Eq. (5) implicitly follows the same spirit of IB principle under selfsupervised settings. As further enlightenment, we can relate Eq. (5) with the multi-view Information Bottleneck [9] and the minimal and sufficient representations for self-supervision [45]: Corollary 1. Let X1 = S, X2 = X and assume 0 <  1, then minimizing Eq. (5) is equivalent to minimizing the Multi-view Information Bottleneck Loss in [9]:
LMIB = I(Z1, X1|X2) I(X2, Z1), where 0 <  1. (15)
Corollary 2. When the data augmentation process is reversible, minimizing Eq. (5) is equivalent to learning the Minimal and Sufficient Representations for Self-supervision in [45]:
ZsslX = argmax ZX I(ZX , S), Z sslmin X = argmin ZX H(ZX |S) s.t. I(ZX , S) is maximized. (16)

heading: 4.3 Influence on Downstream Tasks
text: We have provided a principled understanding for our new objective. Next, we discuss its effect on downstream tasks T . The rationality of data augmentations in SSL is rooted in a conjecture that an ideal data augmentation approach would not change the information related to its label. We formulate this hypothesis as a building block for analysis on downstream tasks [36, 9]. Assumption 2. (Task-relevant information and data augmentation). All the task-relevant information is shared across the input data X and its augmentations S, i.e., I(X,T ) = I(S, T ) = I(X,S, T ), or equivalently, I(X,T |S) = I(S, T |X) = 0.
This indicates that all the task-relevant information is contained in augmentation invariant features. We proceed to derive the following theorem which reveals the efficacy of the learned representations by our objective with respect to downstream tasks. Theorem 3. (Task-relevant/irrelevant information). By optimizing Eq. (5), the task-relevant information I(ZS , T ) is maximized, and the task-irrelevant information H(ZS |T ) is minimized. Formally,
min ✓ L ) max ✓ I(ZS , T ) and min ✓ H(ZS |T ). (17)
Therefore, the learned representation ZS is expected to contain minimal and sufficient information about downstream tasks [45, 9], which further illuminates the reason why the embeddings given by SSL approaches have superior performance on various downstream tasks.

heading: 5 Experiments
text: We assess the quality of representations after self-supervised pretraining on seven node classification benchmarks: Cora, Citeseer, Pubmed, Coauthor CS, Coauthor Physics and Amazon Computer, Amazon-Photo. We adopt the public splits for Cora, Citeseer, Pubmed, and a 1:1:9 training/validation/testing splits for the other 4 datasets. Details of the datasets are in Appendix E.
Evaluation protocol. We follow the linear evaluation scheme as introduced in [48]: i) We first train the model on all the nodes in a graph without supervision, by optimizing the objective in Eq. (5). ii) After that, we freeze the parameters of the encoder and obtain all the nodes’ embeddings, which are subsequently fed into a linear classifier (i.e., a logistic regression model) to generate a predicted label for each node. In the second stage, only nodes in training set are used for training the classifier, and we report the classification accuracy on testing nodes.
We implement the model with PyTorch. All experiments are conducted on a NVIDIA V100 GPU with 16 GB memory. We use the Adam optimizer [20] for both stages. The graph encoder f✓ is specified as a standard two-layer GCN model [22] for all the datasets except citeseer (where we empirically find that a one-layer GCN is better). We report the mean accuracy with a standard deviation through 20 random initialization (on Coauthor CS, Coauthor Physics and Amazon Computer, Amazon-Photo, the split is also randomly generated). Detailed hyperparameter settings are in Appendix E.

heading: 5.1 Comparison with Peer Methods
text: We compare CCA-SSG with classical unsupervised models, Deepwalk [32] and GAE [21], and self-supervised models, DGI [48], MVGRL [15], GRACE [57] and GCA [58]. We also compare with supervised learning models, including MLP, Label Propagation (LP) [56], and supervised baselines GCN [22] and GAT [47]3. The results of baselines are quoted from [15, 57, 58] if not specified.
We report the node classification results of citation networks and other datasets in Table 2 and Table 3 respectively. As we can see, CCA-SSG outperforms both the unsupervised competitors and the fully supervised baselines on Cora and Pubmed, despite its simple architecture. On Citeseer, CCA-SSG achieves competitive results as of the most powerful baseline MVGRL. On four larger benchmarks, CCA-SSG also achieves the best performance in four datasets except Coauther-Physics. It is worth mentioning that we empirically find that on Coauthor-CS a pure 2-layer-MLP encoder is better than GNN models. This might because the graph-structured information is much less informative than the node features, presumably providing harmful signals for classification (in fact, on Coauthor-CS, linear models using merely node features can greatly outperform DeepWalk/DeepWalk+features).
3The BGRL [39] is not compared as its source code has not been released.

heading: 5.2 Ablation Study and Scalability Comparison
text: Effectiveness of invariance/decorrelation terms. We alter our loss by removing the invariance/decorrelation term respectively to study the effects of each component, with results reported in Table 4. We find that only using the invariance term will lead to merely performance drop instead of completely collapsed solutions. This is because node embeddings are normalized along the instance dimension to have a zero-mean and fixed-standard deviation, and the worst solution is no worse than dimensional collapse (i.e., all the embeddings lie in an line, and our decorrelation term can help to prevent it) instead of complete collapse (i.e., all the embeddings degenerate into a single point). As expected, only optimizing the decorrelation term will lead to poor result, as the model learns nothing meaningful but disentangled representation. In Appendix B we discuss the relationship between complete/dimensional collapse, when the two cases happen and how to avoid them.
Effect of decorrelation intensity. We study how the intensity of feature decorrelation improves/degrades the performance by increasing the trade-off hyper-parameter . Fig. 2 shows test accuracy w.r.t. different ’s on Cora, Citeseer and Pubmed. The performance benefits from a proper selection of (from 0.0005 to 0.001 in our experiments). When is too small, the decorrelation term does not work; if it is too large, the invariance term would be neglected, leading to serious performance degrade. An interesting finding is that even when is very small or even equals to 0 (w/o Ldec in Table 4), the test accuracy on Citeseer does not degrade as much as that on Cora and Citeseer. The reason is that node embeddings of Citeseer is already highly uncorrelated even without the decorrelation term. Appendix F visualizes the correlation matrices without/with decorrelations.
Effect of embedding dimension. Fig. 3 shows the effect of the embedding dimension. Similar to contrastive methods [48, 15, 57, 58], CCA-SSG benefits from a large embedding dimension (compared with supervised learning), while the optimal embedding dimension of CCA-SSG (512 on most benchmarks) is a bit larger than other methods (usually 128 or 256). Yet, we notice a performance drop as the embedding dimension increases. We conjecture that the CCA is essentially a dimension-reduction method, the ideal embedding dimension ought to be smaller than the dimension of input. Hence we do not apply it on well-compressed datasets (e.g. ogbn-arXiv and ogbn-product).
Scalability Comparison. Table 5 compares model size, training time (till the epoch that gives the highest evaluation accuracy) and memory cost of CCA-SSG with other methods, on Cora, Pubmed and Amazon-Computers. Overall, our method has fewer parameters, shorter training time, and fewer memory cost than MVGRL, GRACE and GCA in most cases. DGI is another simple and efficient model, but it yields much poorer performance. The results show that despite its simplicity and efficiency, our method achieves even better (or competitive) performance.
Table 4: Ablation study of node classification accuracy (%) on the key components of CCA-SSG.
Variants Cora Citeseer Pubmed
Baseline 84.2 73.1 81.6
w/o Ldec 79.1 72.2 75.3 w/o Linv 40.1 28.9 46.5
Figure 2: Effect of . Figure 3: Effect of D.

heading: 6 Conclusion and Discussions
text: In this paper, we have introduced CCA-SSG, a conceptually simple, efficient yet effective method for self-supervised representation learning on graphs, based on the idea of Canonical Correlation Analysis. Compared with contrastive methods, our model does not require additional components except random augmentations and a GNN encoder, whose effectiveness is justified in experiments.
Limitations of the work. Despite the theoretical grounds and the promising experimental justifications, our method would suffer from several limitations. 1) The objective Eq. (5) is essentially performing dimension reduction, while SSL approach usually requires a large embedding dimension. As a result, our method might not work well on datasets where input data does not have a large feature dimension. 2) Like other augmentation based methods, CCA-SSG highly relies on a high-quality, informative and especially, label-invariant augmentations. However, the augmentations used in our model might not perfectly meet these requirements, and it remains an open problem how to generate informative graph augmentations that have non-negative impacts on the downstream tasks.
Potential negative societal impacts. This work explores a simple pipeline for representation learning without large amount of labeled data. However, in industry there are many career workers whose responsibility is to label or annotate data. The proposed method might reduce the need for labeling data manually, and thus makes a few individuals unemployed (especially for developing countries and remote areas). Furthermore, our model might be biased, as it tends to pay more attention to the majority and dominant features (shared information across most of the data). The minority group whose features are scare are likely to be downplayed by the algorithm.
Acknowledgments and Disclosure of Funding
This work was supported in part by NSF under grants III-1763325, III-1909323, III-2106758, and SaTC-1930941. Qitian Wu and Junchi Yan were partly supported by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102). We thank Amazon Web Services for sponsoring computation resources for this work.

Review summaries:
The authors propose CCA-SSG for self-supervised representation learning on graph. Tee method is simple and efficient. Experimental results show high performance of the method on the node classification task.  

This work proposes an alternative self-supervised objective inspired by canonical correlation analysis. Theoretical proof shows its connection to mutual information (MI) estimator and information bottleneck principles. Empirical experimental results demonstrate the effectiveness on node classification. 

This paper focuses on self-supervised graph neural networks, aiming at training graph neural networks without labeled data, and an approach called the CCA-SSG is proposed. CCA-SSG constructs two views of the given graph through edge dropping and node feature masking, and further uses CCA for learning node representations from the two views. The idea of formalizing unsupervised node representation learning as a multi-view learning task looks novel to me. Also, CCA-SSG has intuitive connections with the information bottleneck principle, which is attractive. The authors do experiment on multiple datasets, and CCA-SSG outperforms many competitive baseline methods.  Strengths: 1. Important problem. 2. Principled method with good theoretical connections. 3. Strong results.