From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
Authors: Hengrui Zhang
Qitian Wu
Junchi Yan
David Wipf
Philip S. Yu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Neural Network
Self-supervised Learning
Canonical Correlation Analysis

Abstract: We introduce a conceptually simple yet effective model for self-supervised representation learning with graph data. It follows the previous methods that generate two views of an input graph through data augmentation. However, unlike contrastive methods that focus on instance-level discrimination, we optimize an innovative feature-level objective inspired by classical Canonical Correlation Analysis. Compared with other works, our approach requires none of the parameterized mutual information estimator, additional projector, asymmetric structures, and most importantly, negative samples which can be costly. We show that the new objective essentially 1) aims at discarding augmentation-variant information by learning invariant representations, and 2) can prevent degenerated solutions by decorrelating features in different dimensions. Our theoretical analysis further provides an understanding for the new objective which can be equivalently seen as an instantiation of the Information Bottleneck Principle under the self-supervised setting. Despite its simplicity, our method performs competitively on seven public graph datasets.

From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
Hengrui Zhang', Qitian Wu?, Junchi Yan', David Wipf Philip S. Yu
Department of Computer Science, University of Illinois at Chicago
2 Department of Computer Science and Engineering, Shanghai Jiao Tong University 3 AWS Shanghai Lab
bzhan55@uic edu, fecho740, hanjunchi@@sjtee edu. cn davivipf@amazon com, psyuduic. edu
Abstract
We introduce a conceptually simple yet effective model for self-super represen- tation learning with graph data. It follows the previous methods that generate two views of an input graph through data augmentation. However, unlike contrastive methods that focus instance-level discrimination, we optimize innovative feature-level objective inspired by classical Canonical Correlation Analysis. Com- pared with other works, our approach requires none of the parameterized mutual information estimator, additional projector, asymmetric structures, and most im- portantly, negative samples which can be costly. We show that the new objective essentially 1) aims at discarding augmentation-variint information by learning invariant representations, and 2) can prevent degenerated solutions by decorrelating features in different dimensions. Our theoretical analysis further provides an under- standing for the new objective which can be equivalently seen an instantiation of the Information Bottleneck Principle under the self- supervueee setting. Despite its simplicity, our method performs competitively on seven public graph datasets. The code is available at: Attps://e /tte///ggttt.tt///e setthub.com/heegruilaaaae conn//engruizhang98// wwwp:hh/aangngaaaSS// CSC
Introduction
Self-supervised learning (SSL) has been promising paradigm for learning useful representations
without costly labels 7 46 15. general, it learns representations via proxy objective between
inputs and self-defined signals, among which contrastive methods [46 40
16.
5 12 have achieved
impressive performance on learning image representations by maximizing
mutual information of
two views (or augmentations) of the same input. Such methods can be interpreted as discrimination of a joint distribution (positive pairs) from the product of two marginal ones (negative pairs) [50].
similar methods Although these models have achieved impressive performance, they require complex designs and architectures. For example, DGI 48 and MVGRL [S]] rely on parameterized mutual information estimator to discriminate positive node-graph pairs from negative ones; GRACE S and GCA [58] harness an additional MI.P-projector to guarantee sufficient capacity. Moreover, negative pairs sampled or constructed from data often play an indispensable role in providing effective contrastive signals and have large impact on performance. Selecting proper negative samples is often nontrivial for graph-structured data, not to mention the extra storage cost for prohibitively large graphs. BGRL [39 is recent endeavor on
Inspired by the success of contrastive learning in vision 17
46
16
12 C
have been adapted to learning graph neural networks 48
15
S7
58
"This work done during the author's internship AWS Shanghai AT Lab. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table Technical comparison of self-supervised node representation learning methods. We provide a conceptual comparison with more self self-supervised methods in Appendix G Target denotes the comparison pair, N/G/F denotes node/graph/feature respectively. MI-Estimator: parameterized mutual information estimator. Proj/Pred: additional (MIP) projector or predictor. Asymmetric: asymmetric architectures such as EMA and Stop-Gradient, or two separate encoders for two branches. Neg examples: requiring negative examples to prevent trivial solutions. Space denotes space requirement for storing all the pairs. Our method is simple without any listed component and memory-efficient
Methods
Target
MI-Estimator
Proj/Pred Asymmetric
Neg examples
Space
Instance- level
DGI
N-G
O(N)
MVGRL 15
N-G
O(N)
GRACE 157
N-N
(
GCA 68
N-N
V
O
BGRL 39
N-N
V
O(N)
CCA-SSG (Ours)
F-F
O(D?)
targeting a Serrative-saple--aee free approach for GNN learning through asymmetric architectures 12 However, requires additional components, e.g., an exponential moving average (EMA) and Stop- Gradient, to empirically avoid degenerated solutions, leading to a more intricate architecture.
Deviating from the large body of previous works contrastive learning, in this paper we take new perspective to address SSI. on graphs. We introduce Canonical Correlation Analysis inspired Self-Supervised Learning on Graphs (CCA-SSG), simple yet effective approach that opens the way to a new SSL objective and frees the model from intricate designs. It follows the common practice of prior arts, generating two views of an input graph through random augmentation and acquiring node representations through a shared GNN encoder. Differently, we propose to harness a non-contrastive and non-discriminative feature-level objective, which is inspired by the well-studied Canonical Correlation Analysis (CCA) methods [8. JCC IIl [4, 2 g. More specifically, the new objective aims at maximizing the correlation between two augmented views of the same input and meanwhile decorrelating different (feature) dimensions of single view's representation. We show that the objective 1) essentially pursuits discarding augmentation-varian information and preserving augmentation-inyee invariant information, and 2) can prevent dimensional collapse [19 (i.e., different dimensions capture the same information) in nature. Furthermore, our theoretical analysis sheds more lights that under mild assumptions, model is instantiation of Information Bottleneck under SSL settings
Principle
45).
To sum up, as shown in Table I our new objective induces a simple and light model without
reliance on negative pairs 48
58 parameterized mutual information estimator 48 [S].
an additional projector or predictor [57 58
or asymmetric architectures 399 15]. We provide a
thorough evaluation for the model on seven node classification benchmarks. The empirical results demonstrate that despite its simplicity, CCA-SSG can achieve very competitive performance in general and even superior test accuracy in five datasets. It is worth noting that our approach agnostic to the input data format, which means that it can potentially be applied to other scenarios beyond graph-structured data (such as vision, language, etc.), We leave such a technical extension for future works.
Our contributions are as follows:
1) We introduce a non-contrastive and non non-discriminative objective for self-supervised learning, which is inspired by Canonical Correlation Analysis methods. It does not rely negative samples, and can naturally remove the complicated components. Based on we propose CCA-SSG, a simple yet effective framework for learning node representations without supervision (see Section H
2) We theoretically prove that the proposed objective aims keeping augmentattion-invarii formation while discarding augmentation-ve -variaa one, and possesses an inherent relationship to an embodiment of Information Bottleneck Principle under self-supervised settings (see Section +
in-
3) Experimental results show that without complex designs, our method outperforms state-of-the-art self-supervised methods MVGRL [SI] and GCA [58] on 5 out of 7 benchmarks. We also provide thorough ablation studies on the effectiveness of the key components of CCA-SSG (see Section B).

---

2 Related Works and Background
Contrastive Learning on Graphs. Contrastive methods 46 40
16 have been shown to
be effective for unsupervised learning in vision, which have also been adapted to graphs. Inspired by
the local-global mutual information maximization viewpoints 17. DGI 4 and InfoGraph [38] put
forward unsupervised schemes for node and graph representation learning, respectively. MVGRL [IS]
generalizes CMC [40] to graph-structured data by introducing graph diffusion [2] create another
view for graph. GCC B3 adopts InfoNCE loss 46 and MoCo-basee negative pool [T6] for large-
scale GNN pretraining. GRACE [57], GCA (58) and GraphCI. [52] follow the spirit SimCLR
and learn node/graph representations by directly treating other nodes/graphs negative samples. BGRL 39 targets negative-sample-free sample-free model, inspired by BYOL 12,. on node representation
learning. But it still requires complex asymmetric architectures.
Feature-level Self-supervised Objectives. The Above-mentioned methods all focus on instance- level contrastive learning. To address their drawbacks, some recent works have been turning to feature-level objectives. For example, Contrastive Clustering 25 regards different feature di- mensions as different clusters, thus combining the cluster-level discrimination with instance-level
discrimination. W-MSE [8] performs a differentiable whitening operation on learned embeddings, which implicitly scatters data points in embedding space. Barlow Twins [53] borrows the idea of redundancy reduction and adopts a soft decorrelation term that makes the cross-correlatio of two views' representations close to identity matrix. By contrast, our method is based on the classical Canonical Correlation Analysis, working by correlating the representations of two views from data augmentation and meanwhile decorrelating different feature dimensions of each view's representation.
matrix
Canonical Correlation Analysis. CCA is a classical multivariate analysis method, which is first introduced in [8]. For two random variables Σxy YY. CCA aims seeking two vectors E Ih" and E IR" such that the correlation Exyb
1TTh and E IR", their covariance matrix is
carr(a x,bTY)= = Vx Eyrb
is maximized. Formally, the objective is
max T Exyb, s.t. a Exxa= byyybb= o.b
(1)
For multi-dimensional cases, CCA seeks two sets of vectors maximizing their correlation and subjected the constraint that they are uncorrelated with each other [I0, Later studies apply CCA to multi-view learning with deep models [2] 14], by replacing the linear transformation with neural
networks. Concretely, assuming X1, X2 as two views of an input data, it optimizes
max Tr (Pa) (X1) (X2) Px (X1) Pø. (x (X2) I. 0,,02
where Po. and Poz are two feedforward neural networks and / is an identity matrix. Despite its preciseness, such computation is really expensive [.. Fortunately, soft CCA removes the hard decorrelation constraint by adopting the following Lagrangian relaxation:
min Laiet (Po, (X1), Pe. (X2)) (CSDL (X1)) (sDi(Po 01.02
where Ldist measures correlation between two views' representations and LSDL (called stochastic decorrelation loss) computes an L, distance between Po, (X;) and an identity matrix, for i = 1,2.
3 Approach
3.1
Model Framework
In this paper we focus on self node representation learning, where we consider a single graph (X,A) A). X E NNF and xN denote node features and adjacency matrix respectively. Here N is the number of nodes within the graph and F denotes feature dimension.
Our model simply consists of three parts: 1) a random graph augmentation generator T. GNN- based graph encoder fa where denotes its parameters. 3) a novel feature-level objective function based on Canonical Correlation Analysis. Fig. is an illustration of the proposed model.
3

---

(1) mandm augmentation
View A
(2) graph neural network
CCA-based objective
Z
G = (X.4)
ORCO
GNN,
14 m
DECO
0000
COO
share weights
CCA
D0.00
ZB 00t aoc
Ig
BCD8
DOOG
acce
GNN
8OOE
Figure 1: Illustration of the proposed model: given an input graph, first generate two views through random augmentations: edge dropping and node feature masking. The two views are subsequently put into shared GNN encoder to generate representations. The loss function is applied column-normalizer emhedding matrix of the two views. Note that this simple yet effective pipeline also be conceptually applied for other data like vision and texts, which we leave for future works.
the
Graph augmentations. We consider the standard pipeline for random graph augmen- tation that has been commonly used in pre- vious works (57 391
Algorithm 1: PyTorch-style code for CCA-SSG
I: encoder netsork
lanbda: trade-off
To be specific, harness two ways for augmentation: edge dropping and node feature masking, Edge dropping randomly drops fraction of edges from the original graph, while node feature masking randomly masks a fraction fea- tures for all the nodes. In this way, is composed of all the possible graph transfor- mation operations and each T denotes specific graph transformation for graph G. Note that we use commonly adopted aug- mentation methods to stay our focus on the design of objective function and conduct fair comparison with existing approaches. More complicated random augmentations [52 58 can also be readily plugged into our model. Details for the used augmentation functions in Appendix
D: anbedding dimension
# g: input graph
through augnentation g1. feati augnent(g, feat)
generate
82, feat2 augnent(g. feat)
r(gl, feat:) enbedding the 1at viee
x2 fg²,, feat2) enbadding of the 2st view
batch normalization
((z1 =0) /z)) (q)t(I))
22_norm ((x2 20.nean(0)) / #2.etd(0))/ sq)
covariance
c1 torch mml2_noor TO, zi_nora)
(tt.mmmmm2nn.oo000 22_nora)
iden torch.eye(D)
loss_iny (zl_norm 22_norn) (2) .aum()
loss_dec_1 (c1 iden) ppw ()
loss_dec_2 (c2 idan) pou .aun
loss_dec Less_dec__ 1oss_dec_2
loss loss_inv lambda loss_dec
Training. In each training iteration, we first randomly sample two graph transformations a and tg from T, and then generate two views G = (X.,, AA) and GB (X AB) according the transformations, The two views are subsequently fed into a shared GNN encoder to generate the node embeddings of the two views: Z fo(XA. AA). Z = fo (X B. AB), where Z Z E N DD and denotes embedding dimension. We further normalize the node embeddings along instance dimension so that each feature dimension has O-mean and 1/ /NN -tsandarrd deviation distribution:
Z - (Z) (ZZ) VN
Ż
The normalized A., Z will be used compute feature-level objective in Section 3.2 To help better understand the proposed framework, we provide the PyTorch-style pseudocode for training CCA-SSG in Algorithm
Inference. To generate node embeddings for downstream tasks, we put the original graph G (X, A) into the trained graph neural network fo and obtain node embeddings Z fo(fo(X A).

---

3.2 Learning Objective
Canonical Correlation Analysis has shown its great power in multi-view learning like instance recognition [ However, it still remains unexplored to leverage CCA for self- self-supervised learning. Note that in SSL, one generates two sets of data from the same input through transformation or random data augmentation, which could be regraded as two views of the input data. This inspires us to introduce the following objective for self-super representation learning:
L Z ZB
+A
ZTZZA 1 1
ZBZB
1 |1
(5)
invariance
decorrelation term
where À is a non-negative hyperparameter trading off two terms. Note that minimizing the invariance term is essentially maximizing the correlation between two views as their representations are already normalized. In SSL., as the two augmented views come randomly from the same distribution, we can adopt one encoder fe that is shared across two branches and seek for a regularization that encourages different feature dimensions to capture distinct semantics via the decorrelation term.
We next provide variance-covariance perspective the new objective, following similar lines of reasoning in 41 [ Assume that imput data come from distribution æ p(z) and is view of æ through random augmentation
P- ( Denote 2 as the representation of then minimizing the invariance term, by expectation, is to minimize the variance of the normalized representation že. conditioned on æ. Also, minimizing the decorrelation term push the off off-diagonal elements of the covariance matrix (given by two 2 's close to O. Formally, we have
N
D
Linv
LA
ZB
=
Σ k=1
V
s|n
* 22N,
(6)
1=1
Lacc
Zs - I
||oov
T1
=
Σ
(ptz)² for Zs (2 Zz).
$5
where p is the Pearson correlation coefficient.
3.3 Advantages over Contrastive Methods
In this subsection we provide systematic comparison with previous self-supervised methods for node representation learning, including DGI [48), MVGRL [15), GRACE 577. GCA 558 and BGRL 39
and highlight the merits of CCA-SSG. A quick overview is presented in Table
No reliance on negative samples. Most of previous works highly rely on negative pairs to avoid collapse interchangeable, trivial/degenerated solutions 48
57 58 E.g.. DGI and MVGRL generate negative examples by corrupting the graph structure severely, and GRACE/GCA treats all the other nodes within a graph negative examples. However, for self self-supervised learning on graphs, it is non-trit non-trivial to construct informative negative examples since nodes are structurally connected, and selecting negative examples in an arbitrary manner may lead to large variance for stochastic gradients and slow training convergence [51] The recently proposed BGRL model adopts asymmetric encoder architectures for SSL on graphs without the use of negative samples. However, though BGRL could avoid collapse empirically, it still remains as an open problem concerning its theoretical guarantee for preventing trivial solutions 41. Compared with these methods, our model does not rely negative pairs and asymmetric encoders. The feature decorrelation term can naturally prevent trivial solutions caused by the invariance term. We discuss the collapse issue detailedly in Appendix
15
No MI estimator, projector network nor asymmetric architectures. Most previous works rely on additional components besides the GNN encoder to estimate some score functions in final objectives. DGI and MVGRL require a parameterized estimator to approximate mutual information between two views, and GRACE leverages a MLP projector followed by an InfoNCE estimator. BGRL harnesses asymmetric encoder architecture which consists of EMA (Exponential Moving Average), Stop- Gradientt and additional projector. MVGRL also induces asymmetric architectures as it adopts two different GNNs for the input graph and the diffusion graph respectively. In contrast, our approach requires no additional components except single GNN encoder.
Better efficiency and scalability to large graphs. Consider a graph with nodes. DGI and MVGRL contrast node embeddings with graph embedding, which would require O(N) space cost.
5

---

GRACE treats two views of the same node as positive pairs and treat views of different nodes as negative pairs, which would take ((N2) space. BGRL focuses only on positive pairs, which will also take O(N) space. By contrast, our method works on feature dimension. If we embed each node into a D-dimensional vector, the computation of the loss function would require O(D2) space. This indicates that the memory cost does not grow consistently the size of graph increases. As result, our method is promising for handling large-scale graphs without prohibitively large space costs.
4 Theoretical Insights with Connection to Information Theory
In this section we provide some analysis of the proposed objective function: 1) Interpretation of the loss function with entropy and mutual information. 2) The connection between the proposed objective and the Information Bottleneck principle. 3) Why the learned representations would be informative to downstream tasks. The proof' of propositions, theorems and corollaries are in Appendix D
Notations. Denote the random variable of imput data as X and the downstream task as T (it could be the label Y if the downstream task is classification). Note that SSL, we have no access T in training and here we introduce the notation for our analysis. Define S as the self-supervised signal (i.e., an augmented view of X), and shares the same space as X. Our model learns a representation for the input, denoted by 7x and its views, denoted by Zs. Zx fo(X), Zs = fo(S). so(.) is a encoder shared by the original data and its views, which is parameterized by 0. The target of representation learning is to learn a optimal encoder parameter O. Furthermore, for random variable A, B, C, we use (A, B) to denote the mutual information between A and B, (AA B(C) to denote conditional mutual information of A and B on given C, H for the entropy, and H (AIB) for conditional entropy. The proofs of propositions, theorems and corollaries are Appendix D
4.1 An Entropy and Mutual Information Interpretation of the Objective
We first introduce an assumption about the distributions of P(Zs) and P(Zs|X). Assumption 1. (Gaussian assumption of XX)s(X) and P(Zs)):
P(Zs) X) = -Nxx Ex), P(Zs) = =N(
(8)
With Assumption | we can arrive at the following propositions: Proposition In expectation, minimizing Eq. is equivalent to minimizing the entropy of Zs conditioned on input X, i.e.,
min Lime min H[
6
Proposition 2. Minimizing Eq. is equivalent to maximizing the entropy of Zs, i.e.
min Lace = max H (Zss).
(10)
The two propositions unveil the effects of two terms in our objective. Combining two propositions,
we can further interpret Eq. (5) from an information-theoreii perspective.
Theorem 1. By optimizing Eq G). we maximize the mutual information between the augmented view's 's embedding Zg and the imput data X, and minimize the mutual information between Zg and the view itself S, conditioned on the input data X. Formally we have
min L + max I [Zs. X) and min I S(X).
(11)
9
6
The proof is based the facts (Zs.XX H(ss) (X) and I [Zzs = H(Zs H (Zs(S) = H(Ss (X). Theorem I indicates that our objective Eq. (5) learns representations that maximize the information of the input data, i.e., T(Zs, X), and meanwhile minimize the lost information during augmentation, i.e., I(Zs, SIX).
4.2 Connection with the Information Bottleneck Principle
The analysis in Section enables us to further build connection between our objective Eq. 5 and the well-studied Information Bottleneck Principle 4g 44 37
under SSL settings. Recall that the
supervised Information Bottleneck (IB) is defined as follows:

---

Definition 1. The supervised IB aims at maximizing an Information Bottleneck Lagrangian:
IB = T(Y, Zx) [T(X, Zx), where 0.
(12)
As see, sup attempts to maximize the information between the data representation Zx and its corresponding label Y, and concurrently minimize the information between Zx and the input data X (i.e., exploiting compression of Zx from X). The intuition of principle is that Zx is expected to contain only the information that is useful for predicting Y.
Several recent works 9
propose various forms of IB under self-supervise settings. The most relevant one names Self Self supervise Information Bottleneck:
45 3
Definition 2. (Self-supervised Information Bottleneck (33 The Self-supervised IB aims at maxi- mizing the following Lagrangian:
IB.st T(X, 3I 2s), where >> 0.
(13)
Intuitively, Bad posits that desirable representation expected be informative to augmentation invariant features, and to be maximally compressed representation of the input.
Our objective Eq. 55 is essentially embodiment of IB
Theorem 2. Assume 0 < 3 < 1, then by minimizing Eq. the self-supervised Information Bottleneck objective is maximized, formally:
min C max IBasl
(14)
Theorem 2 also shows that Eq. 5 implicitly follows the same spirit of IB principle under self- supervised settings. As further enlightenment, we can relate Eq. (5) with the multi-view Information Bottleneck and the minimal and sufficient representations, for self-supervision 45
Corollary 1. Let X1 S. X2 X and assume 0 < 3 S then minimizing Eq. minimizing the Multi-view Information Bottleneck Loss 191:
is equivalent to
LMIB = I 1Z,, X1|22) (T2,,, Z1, where < 11
(15)
Corollary 2. When the data augmentation process is reversible, minimizing Eq. is equivalent to learning the Minimal and Sufficient Representations for Self-supervision in 45
Z
arg max S), Z arg min (Zx s.t. 1(Zx, S) is maximized.
(16)
Zx
Zx
4.3 Influence on Downstream Tasks
We have provided principled understanding for our new objective. Next, we discuss its effect on downstream tasks T. The rationality of data augmentations in SSL is rooted in conjecture that an ideal data augmentation approach would not change the information related to its label. We formulate this hypothesis as building block for analysis on downstream tasks [36
Assumption 2. relevrtl information and data augmentation). All the task information
shared across the imput data X and its augmentations S, i.e., I(X,T) or equivalently, T(X,T|S,
(|)
This indicates that all the task-relevant information contained in augmentation invariant features. We proceed derive the following theorem which reveals the efficacy of the learned representations by our objective with respect to downstream tasks.
Theorem 3. "Taxk--elevan/irreeeevn information). By optimizing Eq. 5 the task-relevant informa-
tion T TZss, T) is maximized, and the task-irrelevan information H H T')
is minimized. Formally,
min C + max I(Zs, and min H (Zs)T).
(17)
0
Therefore, the learned representation Zs is expected to contain minimal and sufficient information about downstream tasks 45 which further illuminates the reason why the embeddings given by SSL approaches have superior performance various downstream tasks.
7

---

Table Test aceuracy on citation networks. The inpur column highlights the data used for training. (X for node features, A for adjacency matrix, S for diffusion matrix, and Y for node labels).
Methods
Input
Cora
Citeseer
Pubmed
MLP
55.1
46.5
71.4
LP 56
A,
68.0
45.3
63.0
Supervised
GCN 22
X. A
Y
81.5
70.3
79.0
GAT
3
X, A, Y
83.0 + 0.7
72.5 + 0.7
79.0 + 0.3
Raw Features
48
X
47,9 0,4
49.3 + 0.2
69.1 = 0.3
Linear CCA 18
X
58.9111 1.5
75.8 = 0.4
Deep Walk
32
70.7
51.4
74.3 1 0.9
GAB
X,
71.5
65.8 0.4
72.1 0.5
Unsupervised
DGI
48
X, A
+
7
76.8 0.6
MVGRL
X,S,A
73.3 0.5
80.1 0.7
GRACE²
S77
X,A
81.9 1 0.4
71.2 + 0.5
80.6 0.4
CCA-SSG (Ours)
X,A
84.2 1 0.4
73.1
13.1000 0.3
81.6 + 0.4
Results Cora with authors' code is inconsistent with [5
We adopt the results with authors' code.
Results are from reproducing with authors' code.
57
did not the public splits.
Experiments
We assess the quality of representations after self-supervised pretraining seven node classifica- tion benchmarks: Cora, Cileseer, Pubmed, Coauthor CS, Coauthor Physics and Amazon Computer, Amazon-Photo. We adopt the public splits for Cora, Citeseer, Pubmed, and 1:1:9 training/valida- tion/testing splits for the other 4 datasets. Details of the datasets are in Appendix
Evaluation protocol. We follow the linear evaluation scheme as introduced in 48 i) We first train the model on all the nodes in a graph without supervision, by optimizing the objective in Eq. ( ii) After that, we freeze the parameters of the encoder and obtain all the nodes' embeddings, which are subsequently fed into a linear classifier (i.e., a logistic regression model) to generate a predicted label for each node. In the second stage, only nodes in training set used for training the classifier, and we report the classification accuracy on testing nodes.
We implement the model with PyTorch. All experiments are conducted on NVIDIA V100 GPU with 16 GB memory. We use the Adam optimizer (20) for both stages. The graph encoder fo is specified as standard two-layer GCN model [22) for all the datasets except citeseer (where we empirically find that one-layer GCN better). We report the mean accuracy with a standard deviation through 20 random initialization (on Coauthor CS, Coauthor Physics and Amazon Computer, Amazon-Photo the split also randomly generated). Detailed hyperparameter settings are in Appendix
5.1 Comparison with Peer Methods
We compare CCA-SSG with classical unsupervised models, Deepwalk and GAE self-supervised models, DGI [8,, MVGRL (15), GRACE [57] and GCA 158
2
21 and
We also compare with supervised learning models, including MLP, Label Propagation (LP) [56] and supervised baselines
GCN 2 and GAT 47 The results baselines are quoted from 15 57
58
if not specified.
We report the node classification results of citation networks and other datasets in Table and Table respectively. As we can see, CCA-SSG outperforms both the unsupervised competitors and the fully supervised baselines on Cora and Pubmed, despite its simple architecture. On Citeseer, CCA-SSG achieves competitive results as of the most powerful baseline MVGRL. On four larger benchmarks, CCA-SSG also achieves the best performance in four datasets except Coauther-P Physicss It is worth mentioning that we empirically find that on Coauthor-CS pure 2-layer-MLP encoder is better than GNN models. This might because the graph-structure information is much less informative than the node features, presumably providing harmful signals for classification (in fact, on Coauthor- linear models using merely node features greatly outperform Deep Waalk//Deeaallk-featttee
TTe BGRL 39 is compared its source code has not been released.

---

Table Test accuracy on co-author and co-purchase networks. We report both mean accuracy and standard deviation. Results of baseline models are from [58]
Methods
Input
Computer
Photo
CS
Physics
Supervised GCN X, A, Y
86.51 0.54
92.42 0.22
93.03 0.31
95.65 = 0.16
Supervised GAT
47
X, A, Y
86.93 + 0.29
92.56 0.35
92.31 = 0.24
95.47 = 0.15
Raw Features
X
73.81 0.00
78.53 0.00
90.37 + 0.00
93.58 + 0.00
Linear CCA
18
X
79.84 + 0.53
86,92 + 0.72
93.13 0.18
95.04 + 0.17
Unsuperviscd
DeepWall Walk 62
A
85.68 + 0.06
89.44
84.61 0.22
91.77 = 0.15
DeepWar Walk features
X, A
86.28 + 0.07
90.05 0.08
87.70 0.04
94,90 = 0.09
GAE [21
X, A
85.27 0.19
91.62
90.01 - 0.71
94.92 = 0.07
DGI
X, A
83.95 + 0.47
91.61 0.22
92.15 0.63
94.51 = 0.52
MVGRL 15
X, S.S.A A
87.52 + 0.11
9.07
22.1100122
95.33 = 0.03
GRACE' 57
X, A
86.25 + 0.25
92.15
92.93 0.01
95.26 + 0.02
GCA
58
X,A
87.85 1 0.31
92.49 1 0.09
93.10 0.01
95.68 + 0.05
CCA-SSG (Ours)
X,A
88.74 1 0.28
93.14 0.14
93.31 + 0.22
95.38 + 0.06
GCA is essentially enhanced version of GRACE by adopting adaptive augmentations, Both GRACE and GCA would suffer from of memory Coauthor Physics using GPU wit 16GB memory. The reported results are from authors' papers using a 32GB GPU.
5.2 Ablation Study and Scalability Comparison
Effectiveness of invariance/decorrelation terms. We alter our loss by removing the invari- ance/decorrelation term respectively to study the effects of each component, with results reported in Table 4 We find that only using the invariance term will lead to merely performance drop instead of completely collapsed solutions. This is because node embeddings are normalized along the instance dimension to have a zero-mean and fixed-standar deviation, and the worst solution is no worse than dimensional collapse (i.e., all the embeddings lie in an line, and our decorrelation term can help to prevent it) instead of complete collapse (i.c., all the embeddings degenerate into a single point). As expected, only optimizing the decorrelation term will lead to poor result, the model learns nothing meaningful but disentangled representation. In Appendix B we discuss the relationship between complete/dimensionaal collapse, when the two cases happen and how to avoid them.
Effeet of decorrelation intensity. We study how the intensity of feature decorrelation improves/de- grades the performance by increasing the trade-off hyper-parametee X. Fig. 2 shows test accuracy w.r.t. different X's on Cora, Citeseer and Pubmed. The performance benefits from proper selection of X (from 0.0005 to 0.001 in our experiments). When À is too small, the decorrelation term does not work; if is too large, the invariance term would be neglected, leading to serious performance degrade. An interesting finding is that even when À is very small or even equals to (w/o Ldec in Table 4 the test accuracy on Citeseer does not degrade as much as that on Cora and Citeseer. The reason is that node embeddings of Citeseer is already highly uncorrelated even without the decorrelation term. Appendix F visualizes the correlation matrices without/with decorrelations.
Effeet of embedding dimension. Fig. shows the effect of the embedding dimension. Similar to contrastive methods [48 15 57
CCA-SSG benefits from large embedding dimension (compared with supervised learning), while the optimal embedding dimension of CCA-SSG (512 on most benchmarks) a bit larger than other methods (usually 128 or 256). Yet, we notice a performance drop as the embedding dimension increases. We conjecture that the CCA is essentially dimension-reduuttion method, the ideal embedding dimension ought to be smaller than the dimension of input. Hence we do not apply it on well-compressed datasets (e.g. ogbn-arXiv and producc).
58
Scalability Comparison. Table compares model size, training time (till the epoch that gives the highest evaluation accuracy) and memory cost of CCA-SSG with other methods, on Cora, Pubmed and Amuzon- Conitoorroomtrrs..... Overall, our method has fewer parameters, shorter training time, and fewer memory cost than MVGRL, GRACH and GCA in most cases. DGI is another simple and efficient model, but it yields much poorer performance. The results show that despite its simplicity and efficiency, method achieves even better (or competitive) performance.

---

Table 4: Ablation study of node classification accuracy (%) on the key components of CCA-SSG.
0.80
Crra
2.10
Nomred
Variants Cora Citeseer Pubmed
Guro
Baseline 84.2
73.1
81.6
Cheoser
Nutmed
w/o Ldec 79.1
72.2
75.3
Embeddiing dimension:
w/o Line 40.1
28.9
46.5
Figure 2: Effect of X.
Figure 3: Effect D.
Table 5: Comparison of the number of parameters, training time for achieving the best performance, and the memory cost of different methods on Cora, Pubmed and Amazon- Amazzon-Compenn Computer. MVGRL on Pubmed and Computer requires subgraph sampling with graph size 4000. Others are full- full-graph.
Cora (NV: 2,708)
Pubmed (NV: 19,717)
Computer (N: 13,752)
Methods
#Paras
Time
Mem
#Paras
Time
Mcm
#Paras
Time
Mem
DGI MVGRL
1260K
6.4s
1.4G
782K
5.9s
1.9G
919K
14.1s
1.9G
1731K
26.9s
4.6G
775K
29s
5.4G
1049K
31.5s
5.5G
GRACE/GCA
997K
8.3s
1.7G
520K
756s
12.6G
273K
314s
7.6G
CCA-SSG(Ours)
997K
3.8s
1.6G
519K
9.6s
2.7G
656K
14.8s
2.5G
Conclusion and Discussions
In this paper, we have introduced CCA-SSG, a conceptually simple, efficient yet effective method for self-supervised representation learning graphs, based on the idea of Canonical Correlation Analysis. Compared with contrastive methods, our model does not require additional components except random augmentations and a GNN encoder, whose effectiveness is justified in experiments.
Limitations of the work. Despite the theoretical grounds and the promising experimental justi- fications, our method would suffer from several limitations. 1) The objective Eq. (5) is essentially performing dimension reduction, while SSL approach usually requires a large embedding dimension. As a result, method might not work well on datasets where input data does not have large feature dimension. 2) Like other augmentation based methods, CCA-SSG highly relies on high-quality, informative and especially, label-invariant augmentations. However, the augmentations used in model might not perfectly meet these requirements, and remains an open problem how to generate informative graph augmentations that have non-negative impacts on the downstream tasks.
Potential negative societal impacts. This work explores a simple pipeline for representation learning without large amount of labeled data. However, in industry there many career workers whose responsibility is to label or annotate data. The proposed method might reduce the need for labeling data manually, and thus makes few individuals unemployed (especially for developing countries and remote areas). Furthermore, our model might be biased, as it tends to pay more attention the majority and dominant features (shared information across most of the data). The minority group whose features are scare are likely to be downplayed by the algorithm.
Acknowledgments and Disclosure of Funding
This work was supported in part by NSF under grants III-1763325, II-1999333, III-2106758, and Qitian Wu and Junchi Yan were partly supported by Shanghai Municipal Science and Technology Major Project SHZDZXX00))) We thank Amazon Web Services for sponsoring computation resources for this work.
References
[1] Rana Ali Amjad and Bernhard Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE Trans. Pattern Anal. Mach.
10

---

Intell., 42(9))2225-22) 2239, 2020.
[2] Galen Andrew, Raman Arora, Jeff A. Bilmes, and Karen Livescu. Deep canonical correlation analysis. In ICML, volume 28, pages 1247-1255, 2013.
[3] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pages 531-540, 2018.
[4] Xiaobin Chang, Tao Xiang, and Timothy M. Hospedales. Scalable and effective deep CCA via soft decorrelation. In CVPR, pages 1488-1497, 2018.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, Proceedings of Machine Learning Research, pages 1597-1607, 2020.
[6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011. 2020.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171-4186, 2019.
[8] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. arXiv preprint 2007:20077.03440 2020.
[9] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In ICLR, 2020.
[10] Gene H Golub and Hongyuan Zha. The canonical correlations of matrix pairs and their numerical computation. Linear algebra for signal processing, pages 27-29, 1995.
[11] Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. A multi-view embedding space for modeling internet images, tags, and their semantics. Int. J. Comput. Vis., 106(2):210-333, 2014.
[12] Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach self-supervised learning. In NeurIPS, 2020.
[13] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304, 2010.
[14] David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Comput., 16(12):2639-266) -2644, 2004.
[15] Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In /CML, volume 119 of Proceedings of Machine Learning Research, pages 41 16 4126. PMLR, 2020.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726-9735. 2020.
[17] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In /CLR, 2019.
[18] Harold Hotelling. Relations between two sets of variates. Biometrika, 28:322-377 17322-377, 1936.
[19] Tianyu Hua, Wenxiao Wang. Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao. On feature decorrelation self-supervised learning. arXiv preprint arXiv:2105.00477 2021.
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
[21] Thomas N. Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016.
[22] Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.

---

[23] Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. In NeurlPS, pages 13333-13345, 2019.
[24] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, volume 32, 2018.
[25] Yunfan Li, Peng Hu, Jerry Zitao Liu, Dezhong Peng. Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. arXiv preprint WrXiv:2009.09665 2020.
[26] Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations styles and substitutes. In SIGIR, pages 43-52, 2015.
[27] Jin Ming, Zheng Yizhen, Li Yuan-Fang: Gong Chen, Zhou Chuan, and Pan Shirui. Multi- scalecontrastive siamese networks for sulf--suprvised graph representation learning. In IJCAI, 2021.
[28] Xu Minghao, Wang Hang, Ni Bingbing. Guo Hongyu, and Tang Jian. Self-supervised graph- levelrepresentation learning with local and global structure. In ICML, 2021.
[29] Galileo Namata, Ben London, Lise Getoor, Bert Huang, and UMD EDU. Query-driven active surveying for collective classification. In 10th International Workshop on Mining and Learning with Gruphs, volume 8, 2012.
[30]
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In NIPS, pages 27 1-1-77 2016.
[31] Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In wWW. pages 259-270, 2020.
[32] Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: online learning of social repre- sentations. In KDD, pages 701-710. ACM, 2014.
[33] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre-training. In KDD, pages 1150-1160, 2020.
[34] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-
Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008
[35] Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. An overview of microsoft academic service (MAS) and applications. In WWW, pages 243-246, 2015.
[36] Karthik Sridharan and Sham M. Kakade, An information theoretic framework for multi-view learning. In COLR, pages 403-414. Omnipress, 2008
[37] DJ Strouse and David J. Schwab. The deterministic information bottleneck. In UAI, 2016.
[38] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR, 2020.
[39] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Velick- ovic, and Michal Valko. Bootstrapped representation learning graphs. arXiv preprint 2xXiv:2102.06514 2021.
[40] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, pages 776-794, 2020.
[41] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning
dynamics without contrastive pairs. arXiv preprint arXiv:2102.068100 2021.
[42] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.0057 2020.
[43] Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[44] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In
ITW, pages 1-5, 2015.
[45] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency, Self- supervised learning from a multi-vien perspective. In ICLR, 2021.
12

---

[46]
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
[47] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
[48] Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR, 2019.
[49] Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang. Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J. Smola, and Zheng 7hang. Deep graph library: Towards efficient and scalable deep learning on graphs. arXiv, 1909.01315, 2019.
[50] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, pages 9929-9939, 2020.
[51] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In ICLR, 2021.
[52] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurlPS, 2020.
[53] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML, 2021.
[54] Jiaqi Zeng and Pengtao Xie. Contrastive self- learning for graph classification. arXiv preprint arXiv:2009.0559
2020.
[55] Hanlin Zhang, Shuai Lin, Weiyang Liu, Pan Zhou, Jian Tang. Xiaodan Liang, and Eric P Xing. Iterative graph self aif-distillation. arXiv preprint arXiv:2010. /2609, 2020.
[56] Xiaojin Zhu. Zoubin Ghahramani, and John D. Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In ICML, pages 912-919, 2003.
[57] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04//3 2020.
[58] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section i
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Assump- tion and Assumption _
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix D
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g.. with respect to the random seed after running experi-
ments multiple times)? [Yes] See Section 5
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, cloud provider)? [Yes] See Section 5
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix B3
(b) Did you mention the license of the assets? [Yes] See Appendix B33
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent
obtained from people whose data you's 're using/curating? [N/A] The data used for experiments are all publicly available.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] The data we used contain no personally indentifiable information nor offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Q82YvK4-o4", "reviewer": "Reviewer_1tQC", "summary": "This work proposes an alternative self-supervised objective inspired by canonical correlation analysis. Theoretical proof shows its connection to mutual information (MI) estimator and information bottleneck principles. Empirical experimental results demonstrate the effectiveness on node classification. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed self-supervised objective shares the same idea on making the representations of augmented views from the same data be close as much as possible. While the difference lies at the usage of negative samples. The proposed loss function does not need negative sample, but a regularization term instead. The key idea of this work is very easy to follow, and the experimental results on node classification task demonstrates its superiority to baselines. \n\nConcerns:\nI have a question about the correctness of the theoretical proof to show its connection to previous approach needs more explanation. It seems to be very important to dissect the proposed loss function. Authors drop an assumption that S denotes the augmented view from original input data X, and they share the same space, referring to Section 4, line 185. Then two different distributions (i.e., P(Z_S) and P(Z_S | X)) for the variable Z_S are defined in Line 191. Theoretical analysis are conducted based on this assumption. However, I'm confused about the difference between P(Z_S) and P(Z_S | X). It's easy to understand the variable Z_S depends on X because the augmented view S is generated from X. But it's difficult to understand why we can define a marginal distribution P(Z_S). It's not reasonable because S still depends on X. It causes a big trouble to follow the conclusion in Proposition 2. Why we can discuss the entropy of variable Z_S without the condition on input data X.\n\nMinor Issue:\n1. In the appendix D, please double check the Remark 1. How can we get the equality H(Z_S|X) - H(Z_S | S, X) = H(Z_S | X) according to property 6?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I admit that the limitations and societal impact are given by authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BL1zdCIZPdH", "reviewer": "Reviewer_nj2q", "summary": "This paper focuses on self-supervised graph neural networks, aiming at training graph neural networks without labeled data, and an approach called the CCA-SSG is proposed. CCA-SSG constructs two views of the given graph through edge dropping and node feature masking, and further uses CCA for learning node representations from the two views. The idea of formalizing unsupervised node representation learning as a multi-view learning task looks novel to me. Also, CCA-SSG has intuitive connections with the information bottleneck principle, which is attractive. The authors do experiment on multiple datasets, and CCA-SSG outperforms many competitive baseline methods.\n\nStrengths:\n1. Important problem.\n2. Principled method with good theoretical connections.\n3. Strong results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Below are some detailed comments and questions:\n\n1. At line 553 of the appendix, it is said that s_1 and s_2 come from the same distribution p_{aug}(x). But from my understanding of the proposed method, s_1 and s_2 are graphs from two different views (i.e., s_1 from edge dropping and s_2 from node feature masking respectively). If this is the case, s_1 and s_2 should come from different distributions. Is it correct?\n\n2. CCA-SSG relies on edge dropping and node feature masking to augment the original graph, but the hyperparameters of edge dropping rate and feature masking rate are not studied in the paper. How will the results of CCA-SSG vary if the hyperparameters are changed?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "-SY2LcxbQoR", "reviewer": "Reviewer_N2Hb", "summary": "The authors propose CCA-SSG for self-supervised representation learning on graph. Tee method is simple and efficient. Experimental results show high performance of the method on the node classification task. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem setting is not clear.\nThe basic objective of CCA is to extract two different correlated features from two different observed data sets.\nThe link between CCA and node representation is not clear.\nThere is almost no explanation on the details of datasets in the experimental section. \nEach dataset has two different observations?\n\nThe authors made a comparison of the performance of many methods, but traditional CCA is missing. Traditional CCA can not be applied to the node representation learning problem? The title of this paper is \"From Canonical Correlation Analysis to Self-supervised Graph Neural Networks\". It would be better to show the performance of CCA as well.\n\nFor the label prediction, why did the authors use logistic regression?\n\nFrom Table 2, unsupervised methods tend to work better than supervised methods. It is not strange?\n\nThe perfromance is heavily dependent on lambda and the embedding dimension. How can appropriate parameters be determined in practice?\n\n\n\nThank you for the answers.\nI have raised my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The details of the proposed method in this paper are not clear.\n\nThe method is unable to extract features involving the minority.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new objective and framework for self-supervised representation learning on graphs. It formalizes unsupervised node representation learning as a multi-view learning task. Results show that the proposed approach outperforms many competitive baseline methods. The idea presented in the paper is interesting. The proposed approach depends on several hyperparameters. The effect of these hyperparameters are not studied and should be included. 
Decision: Accept (Poster)