SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To simplify the explanation, we focus on the vanilla graph convolution module proposed in GCN as an illustrative example. At the  $l$ -th layer, node representations are aggregated through two interleaving steps:
$$
(\text {G r a p h C o n v o l u t i o n}) \mathbf {H} ^ {\prime} = \hat {\mathbf {A}} \mathbf {H} ^ {(l)}, \tag {3}
$$
$$
(\text {F e a t u r e T r a n s f o r m a t i o n}) \mathbf {H} ^ {(l + 1)} = \sigma \left(\mathbf {H} ^ {\prime} \mathbf {W} ^ {(l)}\right), \tag {4}
$$
where  $\sigma$  is the activation function, and  $\mathbf{W}^{(l)}$  denotes the weight matrix.  $\hat{\mathbf{A}} = \bar{\mathbf{D}}^{-1 / 2}\bar{\mathbf{A}}\bar{\mathbf{D}}^{-1 / 2}$  is the symmetrically normalized version of the self-loop augmented adjacency matrix  $\bar{\mathbf{A}} = \mathbf{A} + \mathbf{I}$ , where  $\bar{\mathbf{D}}$  is the diagonal degree matrix of  $\bar{\mathbf{A}}$ . While various variants of GCN have been proposed, most include generalized graph convolution (GraphConv) operators that bring neighbor features closer through message passing [50]. For comparison, we also consider a vanilla MLP encoder, which extracts features from each node individually and can be seen as only applying the feature transformation step.
Importantly, we notice that the utilization of GraphConv within encoders is the key for most existing GCL methods to generalize well in the absence of positive samples. To demonstrate this, we compare two backbones: GCN (with GraphConv) and MLP (without GraphConv), on the node-classification task. The results are presented in Table 4. Remarkably, we observe that the distinctive property of GCL disappears under the MLP backbone. Compared to the performance achieved with the standard contrastive loss (e.g.,  $67.72\%$  on Cora), the MLP-based GCL exhibits significantly lower performance  $(56.10\%)$  under the no-positive setting. This highlights the importance of the feature propagation process in GraphConv, which underlies the unique positive-free behavior observed in GCL.
Here we provide theoretical insights into this phenomenon by uncovering the implicit regularization mechanism of GraphConv in graph contrastive learning. Through formal connections established between GraphConv and a neighbor-induced alignment objective, we demonstrate that GraphConv has the capability to replace the positive alignment loss in GCL. Consequently, GCL attains favorable performance even in the absence of explicit positive sample training.
Theorem 4.1. Suppose the positive node pairs  $(x, x^{+}) \sim P_{\mathcal{G}}$  is drawn from the following distribution defined via the normalized connection weight of the graph  $\mathcal{G}$
$$
\mathcal {P} _ {\mathcal {G}} \left(x, x ^ {+}\right) = \frac {\hat {\mathbf {A}} _ {x , x ^ {+}}}{\sum_ {u , v} \hat {\mathbf {A}} _ {u , v}}, \quad \forall x, x ^ {\prime} \in [ N ], \tag {5}
$$
then a step of GraphConv (Eq 3) will decrease the following feature alignment loss between positive samples (here  $\mathbf{h}_x$  refers to the  $x$ -th row of a feature matrix  $\mathbf{H}$ ):
$$
\tilde {\mathcal {L}} _ {\text {a l i g n}} \left(\mathbf {h} _ {x}\right) = - \mathbb {E} _ {x, x ^ {+} \sim \mathcal {P} _ {\mathcal {G}} (x, x ^ {+})} \left[ \mathbf {h} _ {x} ^ {\top} \mathbf {h} _ {x ^ {+}} \right]. \tag {6}
$$
Theorem 4.1 reveals that GraphConv implicitly achieves feature alignment among neighborhood samples. This alignment process is particularly effective in homophilic graphs, where neighbors predominantly belong to the same class. In this context, the neighbors essentially act as high-quality positive samples. As a result, the neighbor-induced alignment loss can effectively cluster intra-class samples together, providing a plausible explanation for the success of positive-free GCL. Interestingly, the connection between graph convolution and the alignment objective can also provide a natural explanation for why Yang et al. [63] works, which applies graph convolution to a trained MLP and observes improved performance. This strategy, from our perspective, is amount to further training the MLP features with a neighbor-induced alignment loss for a few steps (thus no severe feature collapse), thus helps improve MLP's performance.
5 How GCL Works without Negative Samples
In this section, we investigate intriguing phenomena of GCL in the absence of negative samples. There are works showing contrastive learning can get rid of negative samples by specific designs on architectures [46, 80] or objective functions [71, 2]. However, we observe that negative samples are dispensable without any specific designs for the graph classification task, whereas in the node classification task, simply removing them may not be sufficient. From the perspective of feature collapse, we emphasize the significant role played by the projection head in the graph classification task. Building upon this insight, we address the collapse issue in the node classification task by modifying the backbone encoder with a specialized normalization technique, and further give a theoretical explanation. The experimental settings are identical to those in Section 4, and the proof of theorems is attached to Appendix F.
5.1 Graph Classification: Both Negative Samples and Specific Designs Are Not Needed
In the context of VCL, it is widely recognized that the removal of negative samples alone leads to the failure of methods. This is primarily attributed to the fact that without negative samples, the alignment loss can be easily minimized by adopting a shortcut solution where the encoder generates a constant feature for all samples, i.e.,  $f(x) = c, \forall x$ . As a consequence, this collapsed feature lacks discriminative power for downstream tasks, resulting in a phenomenon referred to as feature collapse [21]. To empirically demonstrate this issue, we present the evident performance degradation observed in pure no-negative VCL in Table 1.
In contrast to the VCL scenario, our findings in GCL for the graph classification task reveal a notable difference. Surprisingly, we discover that GCL can perform well by utilizing the vanilla positive alignment loss alone, without the need for negative samples or any modifications to the network architecture. As demonstrated in Table 3, models trained exclusively with positive pairs achieve comparable or even superior performance compared to those trained with the default contrastive loss.
Further investigation into the architecture reveals the intriguing role of the projection head in the no-negative setting. Specifically, we estimate the average similarity of the representations  $\mathbf{H} = f(\mathbf{X})$  output by the encoder and  $\mathbf{Z} = g(\mathbf{H})$  output by the projection head (Figure 1(a)). The similarity of  $\mathbf{Z}$  is close to 1, indicating that the projection head indeed learns a collapsed solution. However, the similarity of  $\mathbf{H}$  is much lower. It is worth noting that in common GCL practice, the projection head is removed after training, and downstream tasks employ  $\mathbf{H}$  for evaluation. Therefore, thanks to the projection head, although the model learns a collapsed solution for optimizing the alignment loss, the downstream results remain unaffected.
To gain further insights into the role of the projection head, we explore the mechanism from a spectral perspective. We compare the singular value distributions of representations before and after the projection head on the MUTAG dataset (Figure 1(b)). The distribution of singular value becomes more concentrated after the projection head, indicating a decrease in rank. To validate this, we estimate the ranks of  $\mathbf{H}$  and  $\mathbf{Z}$ , revealing that the rank of  $\mathbf{Z}$  is noticeably lower than that of  $\mathbf{H}$  (Figure 1(c)). Inspired by Gupta et al. [13], we remove the projection head during training (where  $g(\cdot)$  is an identity function, making  $\mathbf{H}$  equal to  $\mathbf{Z}$ ). Comparing the rank of resulting representation  $\mathbf{H}^*$ , we find that it falls between the ranks of  $\mathbf{H}$  and  $\mathbf{Z}$ . These observations indicate that the projection head implicitly selects a low-rank subspace of features to satisfy the alignment loss.
(a) Average Similarity
Figure 1: Metrics before and after the projection head.  $\mathbf{H}$  and  $\mathbf{Z}$  denote the representations before and after the projection head, respectively. In Figure 1(c), we also report the rank of representations  $\mathbf{H}^*$  when training without the projection head. Experiments are conducted with GraphCL method.
(b) Singular Value Distribution
(c) Average Rank
Table 5: Test accuracy (\%) of node classification benchmarks using GRACE method. We compare the performances of models trained with InfoNCE loss (Contrast), alignment loss (NO Neg), and alignment loss with ContraNorm in encoder (GCN+CN). Mean accuracy with standard derivation is reported after 10 runs. Average accuracy across datasets is reported. We conduct significance testing using Wilcoxon Signed Rank Test [57], comparing the default setting (first line) with others. The p-value is averaged across datasets. A value below 0.05 denotes significant accuracy difference (red), while a value above 0.05 indicates insignificance (green).
Method	Loss	Encoder	Cora	CiteSeer	PubMed	Photo	Computers	Avg	Avg p-value
GRACE [77]	Contrast	GCN	84.67 ± 1.39	73.47 ± 2.32	85.80 ± 0.16	91.42 ± 1.27	89.01 ± 0.60	84.87	-
NO Neg	GCN	29.85 ± 1.45	20.42 ± 2.26	39.63 ± 0.81	25.10 ± 1.74	36.84 ± 1.30	30.37	0.0020
NO Neg	GCN + CN	82.35 ± 2.28	72.25 ± 1.86	83.30 ± 0.63	92.43 ± 0.82	84.48 ± 1.01	82.96	0.1621
5.2 Node Classification: Normalization in the Encoder Is Enough
When it comes to the node classification task, the absence of negative samples in GCL leads to a significant performance drop (Table 2), unlike the case in graph classification. Numerous factors may be responsible for the suboptimal performance. To confirm feature collapse is the underlying cause, we visualize the training process with alignment loss, where the average node similarities of  $\mathbf{H}$  and  $\mathbf{Z}$  both unite towards one at the end of training (see Appendix D).
Notably, different from graph classification, the representations learned by the encoder also collapse in the node classification task. One plausible conjecture is that learning a collapsed solution is relatively easier for the global graph representation, which can be achieved solely by the projection head. In this case, the encoder is preserved from collapse. However, for learning local node representations, the alignment loss requires each node to be collapsed, which often needs the encoder's involvement. We provide empirical insights in Appendix E, while a rigorous theoretical understanding remains a topic for future work.
Now, the question arises: how does GCL manage to work without negative samples for node classification? While previous solutions derived from VCL [11, 6, 69, 1], such as asymmetric architectures [46, 22] or feature decorrelation objectives [2, 71], exist, they are specific designs which cannot be easily generalized to other methods. Recalling that the feature collapse can be traced back to the encoder, a straightforward approach is directly changing the encoder to prevent collapse. Specifically, we find that just incorporating a normalization component called ContraNorm (CN) [12] into the encoder of GCL is enough.
ContraNorm is originally designed for alleviating the over-smoothing problem in GNNs and Transformers with the formulation:
$$
\mathrm {C N} (\mathbf {H}) = \mathbf {H} - \alpha \mathbf {D} \tilde {\mathbf {A}} \mathbf {H}. \tag {7}
$$
Here,  $\mathbf{H}$  is hidden representation, and  $\alpha$  is a hyper-parameter for scaling.  $\mathbf{D}$  is the diagonal degree matrix of  $\tilde{\mathbf{A}}$ , where  $\tilde{\mathbf{A}} = \mathrm{softmax}(\mathbf{D}\mathbf{H}\mathbf{H}^{\top})$  computes the row-wise normalized similarity matrix between node features. Here, we use a degree-normalized variant of ContraNorm and are the first to introduce it as a novel extension to GCL. As seen from Table 5, by simply incorporating the normalization layer into the encoder, the collapse issue can be rooted out for the GRACE method. Importantly, this normalization layer can be easily adapted by other GCL methods in a plug-and-play
manner. We validate the effectiveness of ContraNorm on multiple GCL methods and under different encoders. The results are provided in Appendix H due to space constraints. It is worth noting that our proposed approach maintains a symmetric model architecture with only the alignment loss, highlighting the ability of the normalization in the encoder for no-negative GCL to perform well.
5.3 ContraNorm Performs Negative Uniformity Implicitly
In this part, we explain how ContraNorm prevents feature collapse without other special designs in the no-negative setting. Similar to the GraphConv case, we define a uniformity loss among all nodes in the same graph to promote feature diversity, namely the neighbor-induced uniformity loss. The following theorem proves that the update of ContraNorm layer leads to a decrease in this uniformity loss.
Theorem 5.1. Suppose the sample  $x$  is drawn from the same distribution in Theorem 4.1, the neighbor-induced uniformity loss is defined as
$$
\tilde {\mathcal {L}} _ {\text {u n i f o r m}} = \mathbb {E} _ {x \sim P _ {\mathcal {G}} (x)} \left[ \log \mathbb {E} _ {x ^ {\prime} \sim P _ {\mathcal {G}} (x)} \exp \left(\mathbf {h} _ {x} ^ {\top} \mathbf {h} _ {x ^ {\prime}}\right) \right]. \tag {8}
$$
The gradient update of this uniformity loss with step size  $\alpha > 0$  gives the ContraNorm update (Eq. 7).
The derived update discussed in Guo et al. [12] suggests that the ContraNorm layer can implicitly promote the diversity among node features during the propagation process. This explains why combining GCL with ContraNorm can avoid feature collapse without explicitly relying on any negative samples.
By analyzing the roles of GraphConv and ContraNorm, we notice that in contrast to visual contrastive learning where the encoders primarily extract features from individual samples, the feature propagation layers in GNNs (GraphConv and ContraNorm) also capture the interaction between different samples. This property enables them to effectively replace the roles of inter-sample objectives like the alignment and uniformity losses. In other words, the feature propagation layers inherently encode the necessary information for learning meaningful representations without explicitly relying on inter-sample objectives. Specifically, by combining Theorem 4.1 and Theorem 5.1, we show that the joint update using graph convolution and ContraNorm implicitly optimizes a neighbor-induced contrastive learning loss:
Theorem 5.2. The joint update of GraphConv and ContraNorm, i.e.,
$$
\mathbf {H} _ {\text {n e w}} = (\mathbf {I} + \hat {\mathbf {A}}) \mathbf {H} - \alpha \mathbf {D} \tilde {\mathbf {A}} \mathbf {H} \tag {9}
$$
corresponds to a gradient descent update of the following contrastive learning loss:
$$
\begin{array}{l} \tilde {\mathcal {L}} _ {\mathrm {c o n t r a s t}} = \tilde {\mathcal {L}} _ {\mathrm {a l i g n}} + \tilde {\mathcal {L}} _ {\mathrm {u n i f o r m}} \\ = \mathbb {E} _ {x, x ^ {+} \sim \mathcal {P} _ {\mathcal {G}} (x, x ^ {+})} \left[ \mathbf {h} _ {x} ^ {\top} \mathbf {h} _ {x ^ {+}} \right] + \mathbb {E} _ {x \sim P _ {\mathcal {G}} (x)} \left[ \log \mathbb {E} _ {x ^ {\prime} \sim P _ {\mathcal {G}} (x)} \exp \left(\mathbf {h} _ {x} ^ {\top} \mathbf {h} _ {x ^ {\prime}}\right) \right]. \tag {10} \\ \end{array}
$$
6 Simple Augmentations Do Not Destroy GCL Performance
Data augmentation is the arguably most crucial component of VCL methods, since different kinds of data augmentations have a dramatic influence on its final per
formance. When examining Table 6, we observe a substantial degradation  $(83.51\% \rightarrow \approx 30\%)$  in VCL's performance when removing all augmentations or applying only random Gaussian noise. However, we find that data augmentations have a much smaller influence on GCL methods.
In GCL, basic augmentations are typically domain-specific, tailored to the node features and topology information, e.g., node feature masking, edge perturbation, and subgraph sampling. Here we choose the combination of node feature masking (FM) and edge perturbation (EP) as the baseline augmentation, which is widely adopted in GCL methods [77, 79, 46, 71].
Table 6: Linear probing accuracy  $(\%)$  of VCL with SimCLR on CIFAR-10 with different augmentation settings.
Augmentation	Accuracy(%)
NO Aug	28.29 ± 1.0
Default Aug	83.51 ± 0.3
Gaussian	36.56 ± 1.2
Taking the node classification as an example, following common practices when unknowning the data prior [33, 70], we further consider a simple augmentation: random Gaussian noise, where a random noise sample drawn from a Gaussian is directly added to node features. Formally, given a graph  $\mathcal{G} = (\mathbf{A},\mathbf{X})$ , the random noise augmentation is defined as  $\tau (\mathcal{G}) = (\mathbf{A},\mathbf{X} + \varepsilon),\varepsilon \sim \mathcal{N}(0,\sigma^2)$ . In practice, we select the standard deviation  $\sigma$  selected from  $[1e - 4,5e - 4,1e - 5]$ . For comparison, we also include the no augmentation setting. The results are shown in Table 7. Although independent of the graph structure and node attributes, the random noise augmentation still achieves a comparable performance computed to domain-specific augmentations, which is quite different from the observations in VCL (Table 6). For further verification, we also conduct experiments with MLP and under different loss settings, the details are shown in Appendix I.
The robustness of GCL to random noise augmentations highlights its flexibility and resilience in the absence of domain-specific augmentations. Recalling Section 4, graph contrastive learning is equipped with two kinds of alignment properties: alignment loss and graph convolution. Therefore, when the effect of alignment loss is weakened (corresponding to domain-agnostic augmentations) or even removed (corresponding to no augmentations), the performance of GCL is relatively slightly influenced with the graph convolution as backing.
Table 7: Test accuracy (\%) of node classification benchmarks using GRACE with different augmentations. We compare no augmentations (NO Aug), domain-agnostic augmentations (Gaussian), and default domain-specific augmentations (FM+EP). Average accuracy and p-value are reported.
Augmentation	Cora	CiteSeer	PubMed	Photo	Computers	Avg	Avg p-value
NO Aug	79.56 ± 2.18	71.83 ± 1.83	84.68 ± 0.58	90.99 ± 1.26	82.83 ± 0.86	81.98	0.1051
FM+EP	84.67 ± 1.39	73.47 ± 2.32	85.80 ± 0.16	91.42 ± 1.27	89.01 ± 0.60	84.87	-
Gaussian	82.72 ± 2.38	72.60 ± 1.21	85.24 ± 0.61	91.32 ± 1.37	82.87 ± 1.09	82.95	0.1778
In this paper, we have shown that GCL exhibits many intriguing phenomena that are rather contradictory to those in VCL. Specifically, we have found that GCL can work in the absence of positive samples. Second, GCL works well without negative pairs for the graph classification task. Third, GCL can achieve comparable performance with domain-agnostic data augmentations like random Gaussian noise. We have made these observations via extensive experiments with a wide range of representative GCL methods. However, the empirical experiments cannot cover all of the GCL methods. We indeed find some exceptions and give a concrete discussion in Appendix G, where exceptional observations can be attributed to the individual property of methods.
Notably, we highlight the implicit mechanisms of architectures to contrastive learning. Theoretically, we build the connection between graph convolution and a neighbor-induced alignment loss, as well as the connection between ContraNorm and a neighbor-induced uniformity loss, giving explanations for the above unique properties of GCL. Overall, our method suggests that graph contrastive learning may behave quite differently from its visual counterpart, and more efforts should be brought in for designing graph-specific self-supervised learning.
Since the main goal of this work is to examine the roles of each component of GCL objectives, one limitation is that it does not propose a new GCL method. Nevertheless, we believe that the new findings in this work would be valuable for future GCL designs. Also, the paper does not examine other SSL paradigms on graph, like masked modeling, which would be an interesting direction to explore in the future.
Yisen Wang was supported by National Key R&D Program of China (2022ZD0160304), National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), and Beijing Nova Program (20230484344).
[1] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022. 8
[2] Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V Chawla. Graph barlow twins: A self-supervised representation learning framework for graphs. Knowledge-Based Systems, 256: 109631, 2022. 3, 7, 8
[3] Karsten M Borgwardt, Cheng Soon Ong, Stefan Schonauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21 (suppl_1):i47-i56, 2005. 16
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020. 1
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 1, 2, 4
[6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021. 8
[7] Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. In ICML, 2023. 3
[8] Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):786–797, 1991. 16
[9] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020. 1
[10] Amur Ghose, Yingxue Zhang, Jianye Hao, and Mark Coates. Spectral augmentations for graph contrastive learning. In AISTATS, 2023. 1, 3
[11] Jean-Bastien Grill, Florian Strub, Florent Alché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020. 2, 8
[12] Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. In ICLR, 2023. 8, 9
[13] Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning. In NeurIPS, 2022. 7
[14] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In ICML, 2020. 15, 17, 21
[15] Kaveh Hassani and Amir Hosein Khasahmadi. Learning graph augmentations to learn graph representations. arXiv preprint arXiv:2201.09830, 2022. 3
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 17
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 1, 2, 4
[18] Christoph Helma, Ross D. King, Stefan Kramer, and Ashwin Srinivasan. The predictive toxicology challenge 2000–2001. Bioinformatics, 17(1):107–108, 2001. 16
A Details on GCL Methods, Benchmarks and Experiment Settings
A.1 Brief Introduction of GCL Methods
Methods for the node classification task.