SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaojun Guo $^{1*}$  Yifei Wang $^{2*}$  Zeming Wei $^{2}$  Yisen Wang $^{1,3\dagger}$
$^{1}$ National Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
$^{2}$ School of Mathematical Sciences, Peking University
$^{3}$ Institute for Artificial Intelligence, Peking University
With the prosperity of contrastive learning for visual representation learning (VCL), it is also adapted to the graph domain and yields promising performance. However, through a systematic study of various graph contrastive learning (GCL) methods, we observe that some common phenomena among existing GCL methods that are quite different from the original VCL methods, including 1) positive samples are not a must for GCL; 2) negative samples are not necessary for graph classification, neither for node classification when adopting specific normalization modules; 3) data augmentations have much less influence on GCL, as simple domain-agnostic augmentations (e.g., Gaussian noise) can also attain fairly good performance. By uncovering how the implicit inductive bias of GNNs works in contrastive learning, we theoretically provide insights into the above intriguing properties of GCL. Rather than directly porting existing VCL methods to GCL, we advocate for more attention toward the unique architecture of graph learning and consider its implicit influence when designing GCL methods. Code is available at https://github.com/PKU-ML/ArchitectureMattersGCL.
Over the past few years, Self-Supervised Learning (SSL) has emerged as a promising approach towards utilizing abundant real-world data without costly human-annotated information [34, 9, 54, 35]. Among various SSL techniques, contrastive learning (CL) has established itself as the avant-garde framework for self-supervised visual representation learning [5]. By pulling similar samples near and pushing dissimilar samples far apart, contrastive learning is able to learn semantically expressive representations and achieves huge success on multiple downstream tasks [5, 17, 4].
Inspired by the success of contrastive learning in the visual domain, contrastive learning is also extensively explored on graph data and achieves competitive performance to supervised ones [40, 26, 65]. Despite technical varieties, most graph contrastive learning (GCL) methods share the same high-level skeleton as visual contrastive learning (VCL). Generally speaking, GCL applies augmentations to generate different views of the original graph, and learn node or graph representations by contrasting positive and negative data pairs. Though some works argue that GCL requires domain-specific designs for graph [29, 59, 10], the design of GCL generally obeys the same paradigm as VCL with three key components: data augmentations, positive pairs for feature alignment, and negative pairs for feature uniformity.
†Corresponding Author: Yisen Wang (yisen.wang@pku.edu.cn)
In this paper, we challenge the commonly held beliefs regarding GCL by revealing its distinct characteristics in comparison to VCL. Specifically, we perform a systematic study with a wide range of representative GCL methods on well-known benchmarks and find three intriguing properties: 1) for the positive pair part, we demonstrate that GCL can achieve competitive performance without adopting any positive pairs. This finding stands in stark contrast to VCL, which dramatically fails in the absence of positive samples. 2) for the negative pair part, we observe that for the graph classification task, GCL performs well without any special designs in the no-negative setting, which is a notable departure from VCL. 3) for the data augmentation part, for VCL, elaborate data augmentations are indispensable in boosting performance [5, 55]. However, we find that GCL is relatively robust under vanilla domain-agnostic data augmentations (e.g., random Gaussian noise).
To explain these intriguing properties of GCL, we delve deep into the model architecture, and uncover the interesting interplay between model components and graph contrastive objectives. First, we shed light on the implicit regularization mechanism of graph convolution in GCL. As is known, graph convolution encourages neighbor nodes to have similar features during propagation. We rigorously show that graph convolution implicitly minimizes a neighbor-induced alignment loss, which reveals its complementary relationship with the learning objective and elucidates the positive-free GCL. Second, we highlight the role of the projection head in GCL without negative samples or any specific designs, where we find for graph classification, the projection head implicitly selects a low-rank feature subspace to satisfy the loss. Third, for the node classification, by incorporating a normalization layer capable of driving nearby node features apart, we show that, without uniformity loss, a GCN encoder alone can prevent features from collapsing to a single point. We theoretically characterize this by connecting this normalization layer with a neighbor-induced uniformity loss.
These intriguing distinctive properties of GCL reveal that the design of contrastive learning can be very domain-specific. Importantly, as also shown in previous works [48, 37], the contrastive algorithm has an implicit interaction with the architecture. Therefore, experiences obtained from one domain (e.g., images) should not be directly considered universal. Instead, when designing graph self-supervised learning methods, we should consider the unique properties of graphs and graph-based models. We hope that our findings could inspire a better understanding of graph contrastive learning and pave the way for new approaches to self-supervised learning on graphs.
We summarize our main contributions below:
- We perform comprehensive evaluations of popular GCL methods across various benchmarks, and find intriguing and general properties of GCL compared with VCL: 1) GCL works without positive samples; 2) GCL works without negative samples on graph classification task; 3) GCL shows less dependence on delicately designed augmentations, where random Gaussian noise even works.
- We reveal the reasons behind the intriguing properties of GCL by theoretically uncovering the interplay between contrastive learning objectives and model architectures: 1) We shed light on the implicit regularization mechanism of graph convolution by establishing its connection with a neighbor-induced alignment objective; 2) We show the collapse-preventing effect of ContraNorm in the no-negative setting by theoretically characterizing its connection with a neighbor-induced uniformity loss.
- Our findings appeal to a re-examination of the real effectiveness of each component in GCL, and provide new insights for designing graph-specific self-supervised learning.
Contrastive Learning. Contrastive learning has attracted intensive attention in self-supervised learning. Its primary objective is to learn a space where similar pairs are closely clustered while dissimilar pairs are far apart. The introduction of Information Noise Contrastive Estimation (InfoNCE) [32] propels contrastive learning to a new climax in the vision domain. Methods such as SimCLR [5] have achieved performance comparable to supervised methods on vision tasks by leveraging stronger augmentations, larger batch size, and a non-linear projection head. MoCo [17] employs a dynamic queue and a moving-averaged encoder to generate more negative samples. The heavy computational burden imposed by a large number of negative pairs prompts the search for alternatives in contrastive learning. BYOL [11] addresses this by enforcing diversity in representations through an asymmetric
architecture and stop-gradient techniques. Barlow-Twins [69] proposes a feature regularization method that maximizes agreements between different views of a sample while eliminating redundancy. On the understanding of contrastive learning, Wang and Isola [53] highlight alignment and uniformity as key properties of contrastive loss, where alignment refers to the closeness of positive pairs and uniformity pertains to the uniform distribution of normalized features on the hypersphere. Wang et al. [56] establish the relationship between contrastive learning and graph neural networks. Zhuo et al. [80] analyze one kind of contrastive learning without negative samples like BYOL from the perspective of rank difference. Zhang et al. [72] reveal the connection between contrastive learning and masked image modeling. There are also other works studying how extra information helps contrastive learning [7, 73].
Graph Contrastive Learning. Stimulated by the success of contrastive learning on images, assorted contrastive attempts are applied to graphs. Motivated by Deep InfoMax (DIM) [19], DGI [51] learns by maximizing mutual information between node representations and corresponding high-level summaries of graphs. InfoGraph [42] adopts the DIM principle on the graph classification task. Inspired by SimCLR, GRACE [77] maximizes the agreement of corresponding node representations in two augmented views for a graph. Similarly, GraphCL [66] learns graph-level representations by maximizing the global representations of two views for a graph. Building upon these pioneer efforts, a plethora of GCL methods have been proposed recently [59, 30, 10, 45, 79]. Inspired by the sparsest cut problem, SCE [74] introduces a Laplacian smoothing trick and then proposes a model without positive samples but only using negative samples for training. It is noted that SCE adopts a specific design of the backbone network and learning objectives (details in Appendix A.2). Different from the above methods, our work does not focus on designing a specific graph contrastive method but generally discusses whether components like positive samples and negative samples are needed in the context of graph contrastive learning.
To relieve the learning from negative samples, some methods are also been proposed. Borrowing the idea of BYOL, BGRL [46] scales up graph contrastive learning to large-scale datasets. Along another line, Bielak et al. [2], Zhang et al. [71] add feature regularization objectives to make the cross-correlation matrix between two views close to an identity matrix, ensuring the two representations are distinguishable. However, these works are all proposed as a specific design that can not be straightforwardly transferred to other GCL methods.
In GCL, augmentations have also emerged as a subject of intensive research. Basic augmentations depend on the node features and topology information, e.g., node feature masking, edge perturbation, and subgraph sampling. Existing methods mostly adopt an empirical combination of basic graph augmentations [77, 66], which is found to benefit more than augmentation of a single type [66]. Adaptive selections of augmentations with learnable strategies have also been proposed [43, 15, 30, 68]. Moreover, some works infuse domain knowledge in finding proper augmentations [44, 41], or design advanced augmentations from the spectral perspective [10, 75, 28, 29]. There are also works identifying limitations in existing task-irrelevant graph augmentations, and expect better practices in augmentations considering the graph-domain knowledge [47, 48, 52]. However, our works are not intended to design stronger data augmentations, but we go another way that finding simple augmentations like random Gaussian noise also work on real-world graph datasets in GCL, while it causes a steep performance drop in VCL.
Inductive Bias of Architecture to Contrastive Learning. For the inductive bias of architecture, Saunshi et al. [37] proposes a general theoretical framework showing the importance of architecture inductive bias to standard contrastive learning. Trivedi et al. [48] presents comparable performances between GCL and untrained GCNs on some relatively simple benchmarks, showing the existence of inductive bias of GCL. In contrast, our work firstly uncovers what exactly the inductive bias of GNNs is by exploring the dynamic interplay between the GNN architecture and the contrastive optimization objective during training.
Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  be a graph with  $n$  nodes, where  $\mathcal{V}$  and  $\mathcal{E}$  are the node set and edge set. We denote  $\mathbf{X}\in \mathbb{R}^{n\times h}$  as the node attribute matrix with input dimension  $h$  and  $\mathbf{A}\in \mathbb{R}^{n\times n}$  as the adjacency matrix. Given augmentation functions  $\tau_{1},\tau_{2}\in \Gamma$ , where  $\Gamma$  is the set of all possible augmentations defined on graphs, GCL generates two augmented views  $\tilde{\mathcal{G}}_1 = \tau_1(\mathcal{G})$  and  $\tilde{\mathcal{G}}_2 = \tau_2(\mathcal{G})$ . These
augmented views are then embedded into representations via an encoder  $f$ , which is often followed by a projection head  $g$ . We denote the backbone features output by the encoder as  $\mathbf{H} = f(\mathbf{X}) \in \mathbb{R}^{n \times m}$  and the projection output as  $\mathbf{Z} = g(\mathbf{H}) \in \mathbb{R}^{n \times d}$ , where the dimension  $d$  is often smaller than  $m$ . During the evaluation, the projection head is removed and only  $\mathbf{H}$  is used for downstream tasks. For graph-level tasks, a global representation can be further obtained by applying a pooling operation  $r$ . The aim of GCL is to learn neural networks that embed a graph into representations by maximizing the representation consistency between different views of the input graph.
For the training objective, the InfoNCE loss [32] is widely used in GCL. Given an anchor view  $\mathbf{u}$ , which can be a node representation  $\mathbf{u} = g(f(\tilde{\mathcal{G}}))_i$  or the global representation of a graph  $\mathbf{u} = r(g(f(\tilde{\mathcal{G}})))$ , we denote its corresponding positive pair as  $\mathbf{v}$ . Then, the InfoNCE loss of  $\mathbf{u}$  is defined as
$$
\mathcal {L} _ {\text {I n f o N C E}} (\mathbf {u}) = - \log \frac {\exp (s (\mathbf {u} , \mathbf {v}) / t)}{\sum_ {\mathbf {q} \in \mathcal {N} _ {\text {n e g}}} \exp (s (\mathbf {u} , \mathbf {q}) / t)}, \tag {1}
$$
where  $\mathcal{N}_{\mathrm{neg}}$  is the set comprising negative samples of  $\mathbf{u}$ ,  $s(\cdot, \cdot)$  denotes the cosine similarity, and  $t$  is the temperature scale. The InfoNCE loss can be decoupled into two non-overlapping parts, each of which includes only positives or negatives, named as alignment loss and uniformity loss [53]:
$$
\mathcal {L} _ {\text {a l i g n}} (\mathbf {u}) = - s (\mathbf {u}, \mathbf {v}) / t, \quad \mathcal {L} _ {\text {u n i f o r m}} (\mathbf {u}) = \log \sum_ {\mathbf {q} \in \mathcal {N} _ {\text {n e g}}} \exp (s (\mathbf {u}, \mathbf {q}) / t). \tag {2}
$$
4 How GCL Works without Positive Samples
In this section, we investigate intriguing phenomena of GCL in the absence of positive samples, which are greatly different from the common understanding and practice in VCL. Specifically, we find that positive samples are not necessary for GCL. We highlight the importance of graph convolution in such discrepancy and provide theoretical insights for explaining the success of positive-free GCL. To ensure the validity of our conclusion, we choose seven highly cited GCL methods and evaluate on well-known benchmarks for both node classification and graph classification tasks. Following the convention of GCL, we use the linear-probing protocol for evaluation. We also report the results of the fine-tuning protocol in Appendix K. More details about the adopted methods, experimental settings, and benchmarks can be found in Appendix A. The proof of theorems is attached to Appendix F.
4.1 Positive Samples Are NOT a Must in GCL
Upon examining existing approaches like SimCLR [5] and MoCo [17], it becomes evident that positive samples play a vital role in VCL. By maximizing the agreement between positive samples, the neural networks can effectively learn semantic information relevant to downstream tasks [55]. It is widely recognized that without this alignment effect, learned rep
resentations may lose meaning and incur poor performance [53, 55]. To illustrate this, we conduct experiments on the CIFAR-10 dataset [25], comparing the InfoNCE loss (including positive samples) and the uniformity loss (excluding positive samples). As shown in Table 1, optimizing only the uniformity loss significantly degrades performance.
Considering the practice in VCL, one would naturally assume that positive samples are equally important and necessary in GCL. However, we unexpectedly find that many existing GCL methods achieve decent performance even without using any positive pairs. To demonstrate this, we conduct comprehensive experiments on both node classification and graph classification tasks. As shown in Table
Table 1: Linear probing accuracy  $(\%)$  of VCL with SimCLR on CIFAR-10: InfoNCE loss, uniformity loss (no positives), alignment loss (no negatives) and no training (random initialization).
Loss	Accuracy (%)
NO Training	27.20 ± 0.9
InfoNCE	83.51 ± 0.3
Uniformity	27.51 ± 0.5
Alignment	29.67 ± 0.8
2, the accuracy gap between the contrastive loss (Contrast) and the loss without positives (NO Pos) is relatively narrow across most node classification datasets. Similarly, in Table 3, we observe similar phenomena in graph classification, where using loss without positive samples sometimes even outperforms the contrastive loss. Additionally, we perform experiments on randomly initialized
Table 2: Test accuracy (\%) of node classification benchmarks using GCL methods. We compare the performances of models trained with InfoNCE loss (Contrast), uniformity loss (NO Pos), alignment loss (NO Neg), and no optimization objective (NO Training). Mean accuracy with standard derivation is reported after 10 runs. Average accuracy across datasets is reported. We conduct significance testing using Wilcoxon Signed Rank Test [57], comparing the contrastive loss and other loss types. The p-value is averaged across datasets. A value below 0.05 denotes significant accuracy difference (red), while a value above 0.05 denotes insignificance (green). OOM denotes out of memory.
Method	Loss	Cora	CiteSeer	PubMed	Photo	Computers	Avg	Avg p-value
GRACE [77]	Contrast	84.67 ± 1.39	73.47 ± 2.32	85.80 ± 0.16	91.42 ± 1.27	89.01 ± 0.60	84.87	-
NO Training	69.12 ± 4.18	60.60 ± 2.59	80.65 ± 0.80	68.37 ± 3.76	57.02 ± 1.93	67.15	0.0020
NO Pos	82.65 ± 1.18	73.50 ± 2.41	85.28 ± 0.79	91.32 ± 1.10	84.40 ± 0.43	83.43	0.2270
NO Neg	29.85 ± 1.45	20.42 ± 2.26	39.63 ± 0.81	25.10 ± 1.74	36.84 ± 1.30	30.37	0.0020
GCA [79]	Contrast	84.04 ± 1.55	72.63 ± 2.68	85.92 ± 0.69	93.07 ± 0.66	86.58 ± 0.75	84.45	-
NO Training	71.25 ± 2.32	58.50 ± 1.32	80.07 ± 0.47	84.92 ± 1.60	68.33 ± 1.23	72.61	0.0020
NO Pos	83.09 ± 2.03	70.42 ± 3.07	84.68 ± 0.63	91.50 ± 0.26	85.19 ± 0.93	82.98	0.1465
NO Neg	31.40 ± 3.61	22.16 ± 3.01	39.58 ± 0.83	28.13 ± 1.14	37.34 ± 0.95	31.72	0.0020
ProGCL [59]	Contrast	85.42 ± 3.41	72.85 ± 2.99	OOM	93.81 ± 0.48	86.35 ± 1.28	84.61	-
NO Training	79.41 ± 0.90	58.08 ± 1.27	83.54 ± 0.83	84.84 ± 1.98	68.39 ± 1.49	74.85	0.0023
NO Pos	86.76 ± 0.52	70.76 ± 1.63	OOM	92.59 ± 0.16	85.71 ± 1.32	83.96	0.2523
NO Neg	30.15 ± 2.70	21.08 ± 1.45	21.13 ± 1.20	4.88 ± 0.33	3.11 ± 0.65	16.07	0.0020
Table 3: Test accuracy (\%) of graph classification benchmarks using GCL methods. We compare the performances of models trained with InfoNCE loss (Contrast), uniformity loss (NO Pos), alignment loss (NO Neg), and no optimization objective (NO Training). Mean accuracy with standard derivation is reported after 10 runs. Average accuracy across datasets is reported. We conduct significance testing using Wilcoxon Signed Rank Test [57], comparing the contrastive loss with other loss types. The p-value is averaged across datasets. A value below 0.05 denotes significant accuracy difference (red), while a value above 0.05 indicates insignificance (green).
Method	Loss	MUTAG	PTC-MR	PROTEINS	IMDB-B	IMDB-M	REDDIT-B	Avg	Avg p-value
GraphCL [66]	Contrast	86.36 ± 1.74	61.73 ± 1.40	72.98 ± 0.52	71.96 ± 0.29	49.80 ± 0.23	84.92 ± 0.40	71.29	-
NO Training	80.85 ± 2.99	57.60 ± 0.66	56.97 ± 4.08	59.24 ± 1.64	34.65 ± 0.67	80.05 ± 0.35	61.56	0.0039
NO Pos	87.97 ± 1.85	62.27 ± 1.29	73.44 ± 0.97	72.38 ± 0.83	48.72 ± 0.68	82.05 ± 0.89	71.14	0.3281
NO Neg	88.73 ± 0.52	58.03 ± 2.24	73.60 ± 0.79	72.10 ± 0.32	49.61 ± 0.33	82.56 ± 0.76	70.77	0.1650
ADGCL [45]	Contrast	90.43 ± 1.18	57.05 ± 2.58	74.29 ± 1.02	71.96 ± 0.15	50.16 ± 0.17	84.84 ± 0.45	71.46	-
NO Training	59.99 ± 0.43	53.80 ± 0.82	55.26 ± 4.30	50.98 ± 0.91	33.55 ± 0.46	59.08 ± 3.49	52.11	0.0026
NO Pos	89.47 ± 0.77	57.54 ± 1.93	72.81 ± 0.63	71.72 ± 0.27	49.89 ± 0.46	84.35 ± 0.49	70.96	0.2419
NO Neg	88.51 ± 0.85	56.14 ± 2.49	73.98 ± 0.32	71.74 ± 0.12	48.93 ± 0.33	74.81 ± 0.35	69.02	0.1631
JOAO [67]	Contrast	86.17 ± 1.55	61.47 ± 1.53	73.15 ± 0.92	71.86 ± 0.32	48.80 ± 0.52	82.51 ± 0.87	70.66	-
NO Training	78.54 ± 2.76	55.48 ± 1.85	56.78 ± 2.91	55.08 ± 1.55	34.93 ± 0.67	52.35 ± 1.19	55.53	0.0020
NO Pos	85.42 ± 1.33	60.36 ± 2.19	73.98 ± 0.53	71.06 ± 0.52	48.11 ± 0.61	83.14 ± 0.34	70.35	0.3057
NO Neg	85.19 ± 0.84	58.13 ± 0.92	73.64 ± 0.73	69.46 ± 0.52	47.93 ± 0.56	82.31 ± 1.47	69.44	0.1211
InfoGraph [42]	Contrast	89.75 ± 1.35	64.26 ± 0.30	72.22 ± 0.51	72.04 ± 0.54	49.49 ± 0.31	82.46 ± 0.52	71.70	-
NO Training	85.63 ± 0.32	54.84 ± 1.56	56.93 ± 0.30	58.44 ± 1.66	35.79 ± 0.81	56.61 ± 2.07	58.04	0.0020
NO Pos	88.58 ± 1.49	62.28 ± 1.34	70.58 ± 0.54	73.98 ± 0.60	49.37 ± 0.54	81.55 ± 0.88	71.06	0.2975
NO Neg	88.19 ± 0.90	62.23 ± 1.62	68.88 ± 0.98	72.34 ± 0.66	49.19 ± 0.63	80.46 ± 0.87	70.22	0.1680
AutoGCL [64]	Contrast	86.26 ± 1.14	61.67 ± 0.60	67.44 ± 1.16	72.50 ± 0.68	49.89 ± 0.45	81.43 ± 1.89	69.87	-
NO Training	85.80 ± 4.25	56.27 ± 3.86	53.98 ± 4.40	57.96 ± 0.76	34.61 ± 0.82	66.30 ± 0.05	59.15	0.0052
NO Pos	87.23 ± 1.13	59.15 ± 0.76	66.57 ± 1.53	70.60 ± 1.13	49.31 ± 0.52	79.61 ± 1.05	68.75	0.1566
NO Neg	87.12 ± 1.71	60.35 ± 1.14	70.51 ± 1.18	73.20 ± 0.32	50.08 ± 0.54	82.33 ± 0.90	70.60	0.1403
models without training (NO Training) for comparison, which result in poor representations. These findings suggest that the removal of positive samples in GCL has minimal impact on the performance of downstream benchmarks, in stark contrast to the results in VCL (Table 1). For further illustration, we visualize the representations learned with contrastive loss and uniformity loss using T-SNE [49] in Appendix B. We also conduct extensive experiments on heterophily datasets and large benchmark OGB-arxiv [20] (See Appendix C).
Table 4: Test accuracy (%) of node classification benchmarks using GRACE method with MLP encoder. We compare the performances of models trained with InfoNCE loss (Contrast), uniformity loss (NO Pos), alignment loss (NO Neg), and no training objective (NO Training). Mean accuracy with standard derivation is reported after 10 runs. Average accuracy across datasets is reported. We conduct significance testing using Wilcoxon Signed Rank Test [57], comparing the contrastive loss with other loss types. The p-value is averaged across datasets. A value below 0.05 denotes a significant accuracy difference (red), while a value above 0.05 indicates insignificance (green).
Method	Loss	Encoder	Cora	CiteSeer	PubMed	Photo	Computers	Avg	Avg p-value
GRACE [77]	Contrast	MLP	67.72 ± 0.88	65.51 ± 2.63	83.29 ± 0.49	87.92 ± 0.59	80.89 ± 1.21	77.07	-
NO Training	MLP	40.66 ± 2.49	42.81 ± 4.82	78.53 ± 0.90	62.12 ± 0.97	57.97 ± 1.13	56.42	0.0020
NO Pos	MLP	56.10 ± 1.08	49.82 ± 3.76	81.32 ± 0.77	65.25 ± 1.13	61.37 ± 0.74	62.77	0.0020
NO Neg	MLP	51.69 ± 3.04	50.36 ± 1.14	79.00 ± 0.63	61.33 ± 0.75	55.07 ± 0.99	59.49	0.0020
4.2 The Implicit Regularization of Graph Convolution in GCL
The intriguing property of positive samples in GCL encourages us to explore the underlying reasons behind this phenomenon. It is worth noting that all the GCL methods analyzed above adopt message-passing graph neural networks (GNNs) like GCN [24] as backbone encoders. We aim to demonstrate that these GNNs inherently possess an implicit regularization effect that facilitates the aggregation of positive samples. This finding helps elucidate why GCL can achieve satisfactory performance without explicitly incorporating an alignment objective.