SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- GRACE [77]. GRACE generates two graph views by corruption and learns node representations by maximizing the agreement of node representations in these two views. To provide diverse node contexts for the contrastive objective, GRACE proposes a hybrid scheme for generating graph views on both structure and attribute levels.
- GCA [79]. GCA proposes adaptive augmentation that incorporates various priors for topological and semantic aspects of the graph. On the topology level, GCA designs augmentation schemes based on node centrality measures, while on the node attribute level, GCA corrupts node features by adding more noise to unimportant node features.
- ProGCL [59]. ProGCL observes limited benefits when adopting existing hard negative mining techniques of other domains in graph contrastive learning. ProGCL proposes an effective method to estimate the probability of a negative being true one, and devises two schemes to boost the performance of GCL.
- DGI [51]. DGI relies on maximizing mutual information between patch representations and corresponding high-level summaries of graphs—both derived using established graph convolutional network architectures. The learnt patch representations summarize subgraphs centered around nodes of interest, and can thus be reused for downstream node-wise learning tasks.
- MVGRL [14]. MVGRL introduces a self-supervised approach for learning node and graph level representations by contrasting structural views of graphs. MVGRL shows that unlike visual representation learning, increasing the number of views to more than two or contrasting multi-scale encodings does not improve performance, and the best performance is achieved by contrasting encodings from first-order neighbors and graph diffusion.
Methods for the graph classification task.
- GraphCL [66]. GraphCL designs four types of graph augmentations to incorporate various priors, and learns graph-level representations by maximizing the global representations of two views for a graph.
- ADGCL [45]. ADGCL proposes a novel principle, adversarial GCL, which enables GNNs to avoid capturing redundant information during training by optimizing adversarial graph augmentation strategies used in GCL.
- JOAO [67]. JOAO proposes a unified bi-level optimization framework to automatically, adaptively and dynamically select data augmentations when performing GraphCL on specific graph data. JOAO is instantiated as min-max optimization.
- InfoGraph [42]. InfoGraph maximizes the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles). By doing so, the graph-level representations encode aspects of the data that are shared across different scales of substructures.
- AutoGCL [64]. AutoGCL employs a set of learnable graph view generators orchestrated by an auto augmentation strategy. The learnable view generators, the graph encoder, and the classifier are trained jointly in an end-to-end manner.
A.2 Relation between Our Work and SCE [74]
Inspired by the sparsest cut problem, SCE [74] introduces a Laplacian smoothing trick and then proposes a model without positive samples but only using negative samples for training. It is noted that SCE is a specific node-level GCL method, while we provide a comprehensive comparison for representative GCL methods on a range of datasets. Specifically, the two differ in: 1) Tasks: SCE only considers node classification tasks while we consider both graph and node classification tasks on various datasets. 2) Backbone networks: The backbone of SCE is a special multi-scale GCN variant, which adopts linear graph convolution (like SGC [58]) and aggregates multi-scale features
at last (like JK-Net [60]). In comparison, we adopt GCN for node classification tasks and GIN for graph classification tasks following the common practice. 3) Learning objectives: For training, SCE designs a new formulation of uniformity loss (the inverse of total pairwise distance)  $(L_{\mathrm{unsup}})$  and an L2 regularization on model weights  $(L_2)$ :
$$
\mathcal {L} = \alpha \mathcal {L} _ {\text {u n s u p}} + \beta \mathcal {L} _ {2} = \frac {\alpha}{\sum_ {(v _ {i} , v _ {j}) \in \mathcal {N}} \| z _ {i} - z _ {j} \| ^ {2}} + \beta \| \theta \| ^ {2}.
$$
Therefore, SCE's unique backbone and objectives raise questions about the general applicability of their findings. Instead, theoretically and empirically, we demonstrate the general validity of the non-necessity of positive samples across various tasks, backbones, and GCL objectives.
A.3 Introduction of Graph Benchmarks
Node classification benchmarks. 1) Citation Networks [38, 31]. Cora, CiteSeer and PubMed are three popular citation graph datasets. In these graphs, nodes represent papers and edges correspond to the citation relationship between two papers. Nodes are classified according to academic topics. 2) Amazon Co-purchase Networks [39]. Photo and Computers are collected by crawling Amazon websites. Goods are represented as nodes and the co-purchase relationships are denoted as edges. Node features are the bag-of-words representation of product reviews. Each node is labeled with the category of goods. 3) Wikipedia Networks [36]. Squirrel and Chameleon was collected from the English Wikipedia, representing page-page networks on specific topics. Nodes represent articles and edges are mutual links between them.
Graph Classification benchmarks. 1) Molecules. MUTAG [8] is a dataset of nitroaromatic compounds and the goal is to predict their mutagenicity on Salmonella typhimurium. PTC-MR [18] is a collection of 344 chemical compounds represented as graphs that report carcinogenicity for male or female rats. 2) Bioinformatics. PROTEINS [3] is a dataset of proteins that are classified as enzymes or non-enzymes. Nodes represent the amino acids and two nodes are connected by an edge if they are less than 6 Angstroms apart. 3) Social Networks. IMDB-BINARY and IMDB-MULTI [62] are movie collaboration datasets consisting of a network of 1,000 actors/actresses who played roles in movies in IMDB. In each graph, nodes represent actors/actresses, and corresponding nodes are connected if they appear in the same movie. REDDIT-BINARY [62] consists of graphs corresponding to online discussions on Reddit. In each graph, nodes represent users, and there is an edge between them if at least one of them responds to the other's comment.
Statistics of datasets are shown in Table 8.
Table 8: Statistics of classification benchmarks. We report average numbers of nodes, edges, and features across graphs in graph classification datasets. For datasets lacking feature attributes, we use all-one vectors as pseudo attributes in practice.
Task	Category	Dataset	#Graphs	# Nodes	# Edges	# Features	# Classes
Node	Citation	Cora	1	2,708	5,278	1,433	7
CiteSeer	1	3,327	4,552	3,703	6
PubMed	1	19,717	44,338	500	3
Co-purchase	Photo	1	7,650	119,081	745	8
Computers	1	13,752	245,861	767	10
Wikipedia	Chameleon	1	2,277	36,101	500	6
Squirrel	1	5,201	217,073	2,089	4
Graph	Protein	MUTAG	188	17.9	39.6	7	2
PTC-MR	344	14.3	29.4	18	2
Bioinformatics	PROTEINS	1113	39.1	145.6	0	2
Social Networks	IMDB-BINARY	1000	19.8	193.1	0	2
IMDB-MULTI	1500	13.0	131.9	0	3
REDMIT-BINARY	2000	429.6	995.5	0	2
A.4 Experimental Details
For the node classification task, following Zhu et al. [77], Velickovic et al. [51], Hassani and Khasahmadi [14], we use linear evaluation protocol, where the model is trained in an unsupervised manner and feeds the learned representation into a linear logistic regression classifier. In the training procedure, a 2-layer Graph Convolutional Network (GCN) [24] is adopted as the encoder. We adopt the default settings of Zhu et al. [77]. Specifically, we use removing edges and masking node features as data augmentations. We grid search augmentation ratios in  $\{0.0, 0.1, 0.2, 0.3, 0.4\}$ . All experiments are trained with Adam SGD optimizer [23] with the learning rate selected from  $\{0.01, 0.001, 0.0005\}$ . The epoch number is selected from  $\{200, 1000, 2000\}$ . The other parameters are fixed for all datasets. In the evaluation procedure, we randomly split each dataset with a training ratio of 0.8 and a test ratio of 0.1, and hyperparameters are fixed as the same for all the experiments. Each experiment is repeated ten times with mean and standard derivation of accuracy score.
For the graph classification task, in the training procedure, a Graph Isomorphism Network (GIN) [61] is adopted as the encoder whose layer number is chosen from  $\{4,8,12\}$  and hidden dimension chosen from  $\{32,512\}$ . We use Adam SGD optimizer with the learning rate selected in  $\{10^{-3},10^{-4},10^{-5}\}$  and the number of epochs in  $\{20,100\}$ . Following Sun et al. [42], You et al. [66], we feed the generated graph embeddings into a linear Support Vector Machine (SVM) classifier, and the parameters of the downstream classifier are independently tuned by cross-validation. The C parameter is tuned in  $\{10^{-3},10^{-2},\dots ,10^{2},10^{3}\}$ . We report the mean 10-fold cross-validation accuracy with standard deviation. All experiments are conducted on a single 24GB NVIDIA GeForce RTX 3090.
For the image classification task, we pretrain ResNet-18 [16] on the CIFAR-10 dataset for 200 epochs, with a projection dimension of 128 and a batch size of 512. We use the SGD optimizer and cosine annealing schedule to set the learning rate, which is initialized as 0.6. During the fine-tuning phase, we only optimize the linear layer for 100 epochs, using the same learning rate schedule as in the pretrain phase. To evaluate the performance, we report the mean accuracy and standard deviation over 5 independent experiments.
Across our experiments, we follow the standard data splits in each domain. To further resolve concerns on the consistency of evaluation setups, we unify the evaluation settings for GCL and VCL. The detailed results are shown in Appendix L.
B Visualization of VCL and GCL via T-SNE
To further illustrate the difference between VCL and GCL, we visualize the representations learned with contrastive loss and uniformity loss using T-SNE [49]. The results are shown in Figure 2. For VCL, the representations learned by uniformity loss distribute more randomly without clear decision boundaries, compared to those learned by InfoNCE loss. However, for GCL, the representations learned by the two losses both achieve good clustering effects.
(a) VCL-InfoNCE
(b) VCL-Uniformity
(c) GCL-InfoNCE
Figure 2: T-SNE visualization of representations learned by VCL and GCL, with InfoNCE loss and uniformity loss. Figure 2(a) and 2(b) are conducted with SimCLR on CIFAR10. Figure 2(c) and 2(d) are conducted with GRACE on Amazon-Photo dataset.
(d) GCL-Uniformity
C Results of Extensive benchmarks
In our paper, we have chosen commonly estimated benchmarks (Cora, CiteSeer, PubMed, AmazonComputers, and Amazon Photo) following the original papers (GRACE [77], GCA [79], and so on). Here, we also provide results and discussions about extensive benchmarks including heterophily benchmarks and large benchmarks.
Heteophily benchmarks. We conduct experiments on two heterophilic datasets Wikipedia-Chameleon and Wikipedia-Squirrel [36] with the GRACE method. As observed in Table 9, training with only negative samples (NO Pos) also gains benefits compared with randomly initialized models (NO Training). However, the gap between using uniformity loss (NO Pos) and using contrastive loss (Contrast) is larger than that of homophilic datasets. In conclusion, the positive-free property of GCL is more applicable to homophilic graphs. It agrees with our theoretical analysis in Section 4.2 which assumes neighbors as positive samples.
Large benchmarks. Here, we further consider a larger node classification benchmark OGB-arfiv [20] with 169,343 nodes and 1,166,243 edges, using the GRACE method. A node-wise similarity matrix is needed when computing the contrastive loss, but its time complexity and space usage are intolerable for large datasets. The scalability problem is one of the reasons why larger datasets are not reported in many original papers. To solve this problem, we randomly sample  $N = 5000$  nodes when computing the similarity matrix, and send the resulting matrix to the objective function. For each iteration, we repeat such sampling 5 times and use the mean loss. The random sampling strategy is simple and straightforward, and more complicated strategies will be considered in the future.
As shown in Table 10, the performance only using negative samples is on par with that using contrastive objectives. And only using positive samples on the node classification task also results in collapse. These observations are consistent with our findings.
Table 9: Test accuracy (\%) on the homophily and heterophily datasets with the GRACE methods. We compare the performances of models trained the InfoNCE loss (Contrast), uniformity loss (NO Pos), alignment loss (NO Neg), and no optimization objective (NO Training). Mean accuracy with standard derivation is reported after 10 runs. Average accuracy across datasets is reported. We conduct significance testing using Wilcoxon Signed Rank Test [57], comparing the contrastive loss with other loss types. The p-value is averaged across datasets. A value below 0.05 denotes significant accuracy difference (red), while a value above 0.05 indicates insignificance (green).
Homophily	Heteophily
Cora	CiteSeer	PubMed	Avg	Avg p-value	Chameleon	Squirrel	Avg	Avg p-value
GRACE	Contrast	84.67 ± 1.39	73.47 ± 2.32	85.80 ± 0.16	81.31	-	48.12 ± 2.35	33.63 ± 1.86	40.88	-
NO Training	69.12 ± 4.18	60.60 ± 2.59	80.65 ± 0.80	70.12	0.0020	32.23 ± 1.82	25.34 ± 1.22	28.79	0.0020
NO Pos	82.65 ± 1.18	73.50 ± 2.41	85.28 ± 0.79	80.48	0.1934	42.97 ± 2.11	30.48 ± 2.25	36.73	0.0254
NO Neg	29.85 ± 1.45	20.42 ± 2.26	39.63 ± 0.81	29.97	0.0020	20.61 ± 2.38	19.58 ± 1.36	20.10	0.0020
GCA	Contrast	84.04 ± 1.55	72.63 ± 2.68	85.92 ± 0.69	80.86	-	46.64 ± 2.85	35.24 ± 1.57	40.94	-
NO Training	71.25 ± 2.32	58.50 ± 1.32	80.07 ± 0.47	69.94	0.0020	33.36 ± 2.04	25.76 ± 2.39	29.56	0.0020
NO Pos	83.09 ± 2.03	70.42 ± 3.07	84.68 ± 0.63	79.40	0.1322	40.17 ± 3.93	28.60 ± 1.05	34.39	0.0107
NO Neg	31.40 ± 3.61	22.16 ± 3.01	39.58 ± 0.83	31.05	0.0020	21.92 ± 4.15	20.19 ± 0.55	21.10	0.0020
ProGCL	Contrast	85.42 ± 3.41	72.85 ± 2.99	OOM	79.14	-	48.38 ± 3.65	33.47 ± 1.93	40.93	-
NO Training	79.41 ± 0.90	58.08 ± 1.27	83.54 ± 0.83	73.68	0.0026	34.21 ± 1.15	25.26 ± 2.24	29.74	0.0020
NO Pos	86.76 ± 0.52	70.76 ± 1.63	OOM	78.76	0.2266	46.44 ± 4.14	30.98 ± 4.32	38.71	0.1064
NO Neg	30.15 ± 2.70	21.08 ± 1.45	21.13 ± 1.20	24.12	0.0020	20.09 ± 1.63	20.46 ± 1.57	20.28	0.0020
Table 10: Test accuracy  $(\%)$  on the OGB-arxiv benchmark using GRACE method with the sampled InfoNCE loss (Contrast), uniformity loss (NO Pos), and alignment loss (NO Neg).
Contrast	NO Pos	NO Neg
OGB-arxiv	65.97 ± 0.23	65.49 ± 0.32	23.88 ± 0.46
D Feature Collapse in Negative-free GCL for Node Classification
In Table 2, we find that the absence of negative samples in GCL leads to a significant performance drop for the node classification task. Numerous factors may be responsible for the suboptimal performance. Here we visualize the training process with alignment loss and InfoNCE loss to show that feature collapse is the underlying cause.
Specifically, we show the tendency of loss, average similarities of node representations  $\mathbf{H} = f(\mathbf{X})$  and  $\mathbf{Z} = g(\mathbf{H})$ , and  $L_{2}$  norms of weight matrices in Figure 3. From Figure 3(a), we can find that when trained with the alignment loss, the training loss steeply converges to  $-1$  (optimal for the alignment loss) after the start of training. However, the similarities among node representations  $\mathbf{H}$
and  $\mathbf{Z}$  both unite towards one. It indicates that once the training starts, the model quickly learns the short-cut where most node representations are identical to meet the alignment loss. We also delineate  $L_{2}$  norms of the weight matrices, which consistently converge to zero during training. As a comparison, we show the training process with InfoNCE loss in Figure 3(b). When trained with InfoNCE loss, the average similarities of node representations are relatively low and norms of weights are non-zero, showing that the collapse issue does not occur in the training process.
(a) Alignment Loss
Figure 3: Tendency of loss, average similarities of node representations  $\mathbf{H}$  and  $\mathbf{Z}$ , and  $L_{2}$  norms of weight matrices. We choose weight matrices of the first and the second convolutional layer (Convs-W1 and Convs-W2), and the first linear layer of the projection head (Linear-W3). Experiments are conducted on Cora with GRACE.
(b) InfoNCE Loss
E Why No-negative GCL Not Collapse in the Graph Classification
In Section 5, we observe different phenomena in the graph classification and node classification. Specifically, in the graph classification task, GCL methods achieve decent performance in the no-negative setting, while the representations collapse in the node classification task. From the architecture perspective, we find in the graph classification task, the representations learned by the projector tend to be identity, while the representations learned by the encoder escape from collapse. We suspect that learning a collapsed solution is relatively easier for the global graph representation, which can be achieved solely by the projection head.
Here, we provide some empirical insights into these conjectures. Instead of researching how to make representations not collapse in the node classification, we choose to explore when no-negative GCL collapses in the graph classification.
The well-known over-smoothing phenomenon states that when repeatedly applying the graph convolution, node features become indistinguishable [27]. The feature collapse is observed in contrastive learning with the alignment loss alone, where all sample features collapse to a single point. The formal equivalence established between graph convolution and the alignment loss (Theorem 4.1) reveals that the two phenomena inherently describe the same thing. To make no-negative GCL fail in graph classification, a straightforward method is stacking more layers within the encoder. Taking the MUTAG dataset as an illustrated example, we indeed find an increase in the similarities of representation  $\mathbf{H}$  and  $\mathbf{Z}$ , and a drop in the performance (Figure 4(a)) when the layer number increases. Another choice is removing the projection head and exposing the encoder. Additionally, we increase the learning rate, whose motivation is enforcing the encoder to iterate to the collapsed solution more quickly. In Figure 4(b), we find that after removing the projection head, the encoder also collapses when the learning rate is raised to 0.01.
Besides the above two extreme cases, here we propose a more convincing method. Imitating the node-wise loss in the node classification, we transform the loss in GraphCL to an L-L version. Formally, the L-L align loss for the graph classification is:
$$
\hat {\mathcal {L}} _ {\text {a l i g n}} = - \frac {1}{M} \sum_ {i = 1} ^ {M} \frac {1}{N _ {i}} \sum_ {\mathbf {u} \in \mathcal {G} _ {i}} s (\mathbf {u}, \mathbf {v}), \tag {11}
$$
where  $M$  denotes the number of graphs,  $N_{i}$  denotes the number of nodes in the graph  $\mathcal{G}_i$ , and the positive sample  $\mathbf{v}$  is the corresponding node of  $\mathbf{u}$  in the augmented graph. Using this alignment loss, we train the modified GraphCL method and get a terrible test accuracy of  $68.18\%$  compared to the original performance of  $86.36\%$ . Figure 4(c) shows that the similarities of  $\mathbf{H}$  and  $\mathbf{Z}$  both converge close to one during training under this loss. These observations further validate our conjecture.
(a) Increase Layer Number
(b) Remove Projection Head
(c) Use L-L Loss
Figure 4: Experiments for the collapse of no-negative GCL in the graph classification. As the layer number of encoder increases, the similarity of representations  $\mathbf{H}$  converges close to one and the performance degrades greatly (Figure 4(a)). A similar phenomenon is observed when removing the projection head and training the encoder with a relatively high learning rate (Figure 4(b)). Additionally, by modifying the graph-level alignment loss to a local node-wise version, we also observe a collapse in the encoder (Figure 4(c)). Experiments are conducted on MUTAG with GraphCL.
F Proof of Theorems
F.1 Reality of Assumption in Theorem 4.1
The augmentations in GCN's implicit alignment loss (using neighbor nodes as positive pairs) differ from those in GCL methods (like node dropping and edge perturbation). However, Table 11 demonstrates their comparable performances, suggesting the complementarity. Therefore, GCN's implicit alignment can replace positive samples under the no-positive setting, achieving good performance. This phenomenon could arise from shared domain priors: neighboring nodes have similar labels, making slight perturbations of edges/nodes inconsequential for their class membership.
Table 11: Node classification accuracy (\%) under GRACE backbone with different augmentations. Mean accuracy with standard derivation is reported after 10 runs. Average accuracy across datasets is reported. We conduct significance testing using Wilcoxon Signed Rank Test [57], comparing the contrastive loss with other loss types. The p-value is averaged across datasets. A value below 0.05 denotes significant accuracy difference (red), while a value above 0.05 indicates insignificance (green).
Augmentation	Cora	CiteSeer	PubMed	Photo	Computers	Avg.	Avg p-value
NO Training	69.12 ± 4.18	60.60 ± 2.59	80.65 ± 0.80	68.37 ± 3.76	57.02 ± 1.93	67.15	0.0020
GRACE augmentations	84.67 ± 1.39	73.47 ± 2.32	85.80 ± 0.16	91.42 ± 1.27	89.01 ± 0.60	84.87	-
Neighbor nodes (ours)	84.93 ± 2.63	71.92 ± 1.51	84.72 ± 0.32	90.54 ± 0.64	86.21 ± 0.58	83.66	0.1668
F.2 Derivation of Theorem 4.1
Proof. It is easy to see that under the definition of the positive samples, the alignment loss can be written equivalently as
$$
\begin{array}{l} \tilde {\mathcal {L}} _ {\text {a l i g n}} (\mathbf {H}) = - \mathbb {E} _ {x, x ^ {+} \sim \mathcal {P} _ {\mathcal {G}} (x, x ^ {+})} \left[ \mathbf {h} _ {x} ^ {\top} \mathbf {h} _ {x ^ {+}} \right] (12) \\ = - \sum_ {x, x ^ {+}} \mathcal {P} _ {\mathcal {G}} \left(x, x ^ {+}\right) \left[ \mathbf {h} _ {x} ^ {\top} \mathbf {h} _ {x ^ {+}} \right] (13) \\ = - \sum_ {x, x ^ {+}} \left[ \hat {\mathbf {A}} _ {x, x ^ {+}} \mathbf {h} _ {x} ^ {\top} \mathbf {h} _ {x ^ {+}} \right] / \sum_ {x, x ^ {+}} \left[ \hat {\mathbf {A}} _ {x, x ^ {+}} \right] (14) \\ = - \operatorname {t r} \left(\mathbf {H} \hat {\mathbf {A}} \mathbf {H} ^ {\top}\right) / c, (15) \\ \end{array}
$$
where  $c = \sum_{x,x^{+}}[\hat{\mathbf{A}}_{x,x^{+}}]$  is a constant.
Here, to maintain the feature scale, we further consider a regularization term on the norm of node features:
$$
\hat {\tilde {\mathcal {L}}} _ {\text {a l i g n}} (\mathbf {H}) = \tilde {\mathcal {L}} _ {\text {a l i g n}} (\mathbf {H}) + \| \mathbf {H} \| ^ {2} / c. \tag {16}
$$
Therefore, the gradient update of the alignment objective (Eq 6) gives the following update rule of node features  $\mathbf{H}$ :
$$
\mathbf {H} _ {\text {n e w}} = \mathbf {H} - \alpha \nabla_ {\mathbf {H}} \hat {\tilde {\mathcal {L}}} _ {\text {a l i g n}} (\mathbf {H}) \tag {17}
$$
$$
= \mathbf {H} - \alpha / c (- 2 \mathbf {A H} + 2 \mathbf {H}) \tag {18}
$$
$$
= (1 - 2 \alpha / c) \mathbf {H} + 2 \alpha / c \cdot \mathbf {A H}, \tag {19}
$$
where  $\alpha$  is the step size. When we choose a specific learning rate  $\alpha = c / 2$ , we recover the graph convolution operation in GCN [24]:
$$
\mathbf {H} _ {\text {n e w}} = \mathbf {A H}, \tag {20}
$$
F.3 Derivation of Theorem 5.1
Proof. Denote  $c = \sum_{x,x^+} [\hat{\mathbf{A}}_{x,x^+}]$  as a constant. Calculating the gradient of the uniformity loss w.r.t. each node feature  $\mathbf{h}_x$  gives the following rule
$$
\nabla_ {\mathbf {h} _ {x}} \tilde {\mathcal {L}} _ {\text {u n i f o r m}} = 2 / c P _ {\mathcal {G}} (x) \sum_ {x ^ {\prime}} \mathbf {A} _ {x, x ^ {\prime}} \mathbf {h} _ {x ^ {\prime}}. \tag {21}
$$
In a matrix form, we have