SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From the contrastive learning perspective, the above process is not inconsistent with its goals. But there is a fundamental difference between graph transport alignment and graph contrastive learning: GALOPA directly utilizes calibration signal from the graph space. Precisely because of this signal, we do not have to distinguish between positive and negative samples, like Maxwell's demon.
4.4 Complexity
The time complexity of the model GALOPA is mainly influenced by the optimization process of Equations (7) and (8). To optimize Equation (7), which contains the fused Gromov-Wasserstein term, we utilize a conditional gradient (CG) solver [21]. This solver necessitates the computation of a gradient with a nearly cubic time complexity at each iteration concerning the size of the graph, i.e., the number of nodes. On the other hand, Equation (8) with the Wasserstein term can be optimized using the Sinkhorn-Knopp algorithm [12], which is highly time-efficient with a nearly square complexity.
5 Plan or Distance?
Thanks to the alignment of the plan and cost for the representation, as a byproduct, we find that the OT distance between the optimal node representations  $Z^{*}$  in Equation (11) is equal to the distance between its corresponding graphs. This is due to the two losses (10) and (9) constrain the optimal node representation to satisfy  $J(Z_{1}^{*},Z_{2}^{*}) = \sigma K(X_{1},X_{2}) + (1 - \sigma)L(A_{1},A_{2})\otimes \pi_{\mathcal{G}}^{*}$  and  $\pi_{\mathcal{Z}}^{*}(Z_{1}^{*},Z_{2}^{*}) = \pi_{\mathcal{G}}^{*}$ , respectively. This means that our losses can prompt the encoder to learn an isometric embedding that preserves the distance between graphs, which is one of the pursuits of the general representation model. Hence, we became interested in the question of who is more important, distance or plan? How would the model perform if we
Figure 4: Plan versus distance. Comparing mean graph/node classification accuracy between transport alignment loss and distance loss on 4 datasets.
drop the alignment of the plan and cost but instead optimize the distance directly? In this section, we assess and rationalize the role of the plan for graph structure data in our self-supervised framework. To compare the possible performance gap between distance and plan, instead of directly optimizing the plan, we construct a new loss with distance as following
$$
\mathcal {L} _ {\text {d i s t}} = \left| \mathcal {W} _ {\mathcal {G}} \left(\mathcal {G} _ {1}, \mathcal {G} _ {2}\right) - \mathcal {W} \left(\mathbf {Z} _ {1}, \mathbf {Z} _ {2}\right) \right| \tag {12}
$$
where  $|\cdot|$  denotes absolute value, and  $\mathcal{W}_{\mathcal{G}}(\mathcal{G}_1,\mathcal{G}_2) = \min_{\pi \in \Pi (\pmb {h}_1,\pmb {h}_2)}\sigma \sum_{ij}c_{\chi}(\pmb {X}_1^i,\pmb {X}_2^j)\cdot \pi_{ij}^{\mathcal{G}} + (1 - \sigma)\sum_{ijkl}c_{\mathcal{A}}(A_1^{ik},A_2^{jl})\cdot \pi_{ik}^{\mathcal{G}}\pi_{jl}^{\mathcal{G}}$
We evaluate the performance of using the pretraining representations on 2 social network datasets, CORA and CITESEER [25] for node classification, and 2 graph classification data PROTEINS and
MUTAG from TUDataset [36] for graph classification. See Section 8 for detailed experimental configurations. Figure 4 reports the averaged node/graph classification accuracy results over the node/graph-level datasets. The results suggest that the model using the plan as an objective significantly outperforms the counterpart models using the distance. Although the experimental result may lead to 'surprise', it demonstrates that the plan is closer to the essence than the distance, for discrete structured data. The optimal transport formulation Equation (5) contains both matching and implicit structural information. If only the final distance is retained instead of capturing the two types of information separately, the learned representation may fail to align properly with the input element. Because the optimal transport plan for the discrete OT problem is not unique in general and the optimal distance may correspond to several plans.
6 Node Attributes or Edges?
Among the information encoded in a graph, the structure and node attributes are two crucial elements for representation learning. The basic requirement of the encoder is to preserve the topology structures and capture the vertex feature of graphs. Thus a problem is encountered in our self-supervised learning paradigm: if we try to calibrate the representations learned by the backbone encoder, which is more important, the edge structure or the node attributes? In other words, if the calibration signal from the input space contains only node attribute information and completely ignores the explicit edge connectivity, will the performance of the proposed model deteriorate
significantly? The answer seems obvious—it should be. But the case seems to be different. Let's take a look at the experiment below.
Figure 5: The mean graph/node classification accuracy on 4 datasets under different values of parameter  $\sigma$ .
As in the previous section, we performed the comparative experiments on four datasets CORA, CITESEER, PROTEINS, and MUTAG. Here we first consider the normal case of our loss function (i.e.,  $\rho \neq 0$ ). We set the value of the parameter  $\sigma$  to adjust the bias of node attributes or edge connections for the plan in the graph space. If  $\sigma = 1$ , the model takes into account only node attributes in the transportation plans. When  $\sigma = 0$ , it integrates explicit edge information while completely ignoring the node feature. Figure 5 reports the results with different  $\sigma$  and shows a surprising outcome: With only node attributes for the calculation of the plan, the model achieves outstanding performance on all the datasets, even optimal on some data. However, if we set  $\rho = 0$  to remove the implicit structure constraint term  $\mathcal{L}_{(\mathrm{im})\mathrm{strc}}$ , the performance of the model suddenly deteriorates dramatically.
This verifies that the constraint  $\mathcal{L}_{\mathrm{(im)strc}}$  is necessary and the implicit structural information it captures does calibrate the encoder even in the case of missing explicit edge connection. It therefore inspires self-supervised graph learning: it may not be able to tell that edge information does not contribute to correction for significant performance gains, but it is perfectly feasible to use only node attributes as a calibration supervisory signal for backbone model. This is a practical and valuable finding since the number of edges in a real-world graph dataset is much more than the number of nodes. The time and space complexity of the model can be reduced greatly if only the node information is used in the calculation of the correction signal while ignoring the edge connections.
7 Are the Transport Alignment Free from Positive/Negative Samples?
As shown in Equation (11), the proposed objective function does not distinguish whether the two different graphs/nodes are positive (negative) samples or not. Here, we want to show that the graph transport alignment strategy can be independent of the label invariance assumption. We conduct experiments below to see how different levels of perturbation affect the performance of GALOPA. When augmenting the graph, we fixed one of the augmentations as NoAug and the other augmentation
Figure 6: The mean graph/node classification accuracy when contrasting with different perturbation rates under 4 datasets. Fix one of the augmentations as NoAug and the other augmentation be the combination of edge perturbation and feature masking. Darker colors indicate better performance.
requires a hyper-parameter "aug ratio" that controls the portion of node attributes/edge that are selected for perturbing. Note that different augmentation strategies can be combined. Since the computation of the optimal transport plan in this paper involves node attributes and edge, we perform two augmentation policies, edge perturbation and feature masking, with different augmentation rates on four datasets (i.e., CORA, CITESEER, MUTAG, and PROTEINS) as shown in Figure 6.
From Figure 6 we find that the performance of our model does not change much even when the original graph is perturbed heavily. For example, even if eighty percent of the edges are removed while eighty percent of the node attributes are destroyed, the model's performance on the dataset CITESEER (0.742) is almost equal to the optimal result (0.743). Fluctuations of only zero point five to two percent are also observed on other data sets, such as one point three in PROTEINS data. Hence, it validates that the alignment of optimal transport between the source space and target space is indeed free from the label-invariant assumption.
8 Comparison with the State-of-the-art Methods
In this section, we compare our proposed self-supervised pre-training framework, GALOPA, with state-of-the-art methods in the settings of unsupervised learning on graph/node classification. More results can be found in the Appendix.
Table 1: Mean node classification accuracy (%) for supervised and unsupervised models. The highest performance of unsupervised models is highlighted in boldface. OOM indicates Out-Of-Memory.
Model	CORA	CITESEER	PUBMED	WikiCS	Amz-Comp.	Amz-Photo	Coauthor-CS	Average
MLP	47.92 ± 0.41	49.31 ± 0.26	69.14 ± 0.34	71.98 ± 0.42	73.81 ± 0.21	78.53 ± 0.32	90.37 ± 0.19	68.72 ± 0.31
GCN	81.54 ± 0.68	70.73 ± 0.65	79.16 ± 0.25	93.02 ± 0.11	86.51 ± 0.54	92.42 ± 0.22	93.03 ± 0.31	85.20 ± 0.39
DEEPWALK	70.72 ± 0.63	51.39 ± 0.41	73.27 ± 0.86	74.42 ± 0.13	85.68 ± 0.07	89.40 ± 0.11	84.61 ± 0.22	75.64 ± 0.35
NODE2VEC	71.08 ± 0.91	47.34 ± 0.84	66.23 ± 0.95	71.76 ± 0.14	84.41 ± 0.14	89.68 ± 0.19	85.16 ± 0.04	73.67 ± 0.46
GAE	71.49 ± 0.41	65.83 ± 0.40	72.23 ± 0.71	73.97 ± 0.16	85.27 ± 0.19	91.62 ± 0.13	90.01 ± 0.71	78.63 ± 0.39
VGAE	77.31 ± 1.02	67.41 ± 0.24	75.85 ± 0.62	75.56 ± 0.28	86.40 ± 0.22	92.16 ± 0.12	92.13 ± 0.16	80.97 ± 0.38
DGI	82.34 ± 0.71	71.83 ± 0.54	76.78 ± 0.31	75.37 ± 0.13	84.01 ± 0.52	91.62 ± 0.42	92.16 ± 0.62	82.02 ± 0.46
GMI	82.39 ± 0.65	71.72 ± 0.15	79.34 ± 1.04	74.87 ± 0.13	82.18 ± 0.27	90.68 ± 0.18	OOM	-
MVGRL	83.45 ± 0.68	73.28 ± 0.48	80.09 ± 0.62	77.51 ± 0.06	87.53 ± 0.12	91.74 ± 0.08	92.11 ± 0.14	83.67 ± 0.31
GRACE	81.92 ± 0.89	71.21 ± 0.64	80.54 ± 0.36	78.19 ± 0.10	86.35 ± 0.44	92.15 ± 0.25	92.91 ± 0.20	83.32 ± 0.41
GCA	82.38 ± 0.47	71.51 ± 0.32	80.89 ± 0.28	78.29 ± 0.36	87.88 ± 0.26	92.33 ± 0.68	92.64 ± 0.34	83.70 ± 0.39
BGRL	81.30 ± 0.54	72.06 ± 0.63	80.52 ± 0.30	76.13 ± 0.18	89.09 ± 0.51	92.15 ± 0.32	92.33 ± 0.39	83.37 ± 0.41
GALOPA	84.21 ± 0.30	74.34 ± 0.18	84.57 ± 0.34	81.23 ± 0.19	88.65 ± 0.11	92.77 ± 0.40	93.04 ± 0.25	85.54 ± 0.25
8.1 Experimental Setup
Datasets. We analyze the quality of representations learned by GALOPA on node and graph classification benchmarks. For node classification, we evaluate the performance of using the pretraining representations on 7 benchmark graph datasets, namely, CORA, CITESEER, PUBMED [25] and Wiki-CS, Amazon-Computers, Amazon-Photo, and Coauthor-CS [47]. For graph classification, we follow GRAPHCL [72] to perform evaluations on 6 graph classification data NCI1, PROTEINS, DD, MUTAG, COLLAB, and IMDB-B from TUDataset [36].
Baselines. For node-level tasks, we adopt three types of baselines: 1) Supervised learning methods, including MLP and GCN [25]; 2) Graph embedding methods, including DEePWALK [42] and
NODE2VEC [17]; 3) Graph contrastive learning methods, including GAE, VGAE [24], DGI [58], GMI [41], MVGRL [19], GRACE [77], GCA [78], and BGRL [55]. For graph-level task, we evaluate the performance of GALOPA in terms of the linear classification accuracy and compare it with 1) two supervised learning methods, including GCN [25] and GIN [69]; 2) seven kernel-based methods, including SP [5], GK [49], WL [50], WLPM [39], FGW [56], DGK [70], and MLG [26]; 3) three unsupervised methods, including NODE2VEC [17], SUB2VEC [1], GRAPH2VEC [37]; 4) five recent SOTA self-supervised learning methods based on contrastive learning, including INFOGRAPH [52], GRAPHCL [72], AD-GCL [53], JOAOv2 [73], RGCL [28] and SIMGRACE [66].
Protocol. We follow the standard evaluation protocol of previous state-of-the-art graph self-supervised learning approaches at the graph and node levels, respectively. Specifically, for graph classification, we report the mean 10-fold cross-validation accuracy after 5 runs followed by a linear SVM. The linear SVM is trained by applying cross-validation on training data folds and the best mean accuracy is reported. For node classification, we report the mean accuracy on the test set after 50 runs of training followed by a linear neural network model. For the graphs (nodes) datasets, we randomly split the data, where  $80\% / 10\% / 10\%$  ( $10\% / 10\% / 80\%$ ) of graphs (nodes) are selected for the training, validation, and test set, respectively.
Implementation Details. In the experiments, we use the Adam optimizer [23] with learning rate is tuned in  $\{0.0001, 0.001, 0.01\}$ . The optimization routine and the convergence analysis are summarized in Appendix B. We conduct the experiment with the trade-off parameter  $\rho$  and  $\sigma$ , the parameter  $C$  of SVM, batch size in the sets  $\{10^{-3}, 10^{-2}, \ldots, 10^{2}, 10^{3}\}$ ,  $\{0, 0.1, \ldots, 0.9, 1\}$ ,  $\{10^{-3}, \ldots, 10^{3}\}$ ,  $\{16, 64, 128, 256, 512\}$ , respectively. To perform graph augmentation, we use 4 types of operations: Edge Perturbation, Feature Masking, Node Dropping, and Graph Sampling. Our model is implemented with Pytorch Geometric [13] and Deep Graph Library [63].
Table 2: Supervised and unsupervised representation learning classification accuracy (%) along with average accuracy of the algorithms on TU datasets. **Bold** indicates the best performance for unsupervised methods on each dataset. ‘-’ means that the results are unavailable.
Model	PROTEINS	DD	MUTAG	NCI1	COLLAB	IMDB-B	Average
GCN	74.92 ± 0.33	76.24 ± 0.14	85.63 ± 0.24	80.20 ± 0.14	79.01 ± 0.18	70.45 ± 0.37	77.74 ± 0.23
GIN	76.28 ± 0.28	78.91 ± 0.13	89.47 ± 0.16	82.75 ± 0.19	80.23 ± 0.19	73.70 ± 0.60	80.22 ± 0.25
SP	75.07 ± 0.54	>1d	85.25 ± 0.24	73.53 ± 0.16	-	55.62 ± 0.02	-
GK	71.67 ± 0.55	78.53 ± 0.03	81.71 ± 0.21	66.06 ± 0.12	71.81 ± 0.31	65.93 ± 0.10	72.61 ± 0.22
WL	72.92 ± 0.56	79.78 ± 0.36	80.76 ± 0.30	80.01 ± 0.50	69.30 ± 0.42	72.30 ± 0.44	75.84 ± 0.43
WLPM	-	78.79 ± 0.38	87.13 ± 0.42	86.32 ± 0.19	-	-	-
FGW	74.50 ± 0.23	-	88.34 ± 0.12	86.24 ± 0.31	-	62.97 ± 0.24	-
DGK	73.21 ± 0.61	74.79 ± 0.32	87.51 ± 0.65	79.98 ± 0.36	64.43 ± 0.48	67.09 ± 0.37	74.50 ± 0.46
MLG	41.23 ± 0.27	>1d	87.94 ± 0.16	>1d	>1d	66.67 ± 0.30	-
NODE2VEC	57.58 ± 0.36	-	72.62 ± 1.02	54.93 ± 0.16	56.12 ± 0.02	50.25 ± 0.09	-
SUB2VEC	53.06 ± 0.56	54.33 ± 0.24	61.17 ± 1.59	52.82 ± 0.15	55.26 ± 0.15	55.34 ± 0.15	55.33 ± 0.47
GRAPH2VEC	73.33 ± 0.21	79.32 ± 0.29	83.28 ± 0.93	73.21 ± 0.18	71.10 ± 0.54	71.16 ± 0.05	75.23 ± 0.36
INFOGRAPH	74.44 ± 0.31	72.85 ± 1.78	89.01 ± 1.13	76.20 ± 1.06	70.05 ± 1.13	73.03 ± 0.87	75.93 ± 1.04
GRAPHCL	74.39 ± 0.45	78.62 ± 0.40	86.80 ± 1.34	77.87 ± 0.41	71.36 ± 1.15	71.14 ± 0.44	76.69 ± 0.69
AD-GCL	73.28 ± 0.46	75.79 ± 0.87	88.74 ± 1.85	73.91 ± 0.77	72.02 ± 0.56	70.21 ± 0.68	75.65 ± 0.86
JOAOV2	74.13 ± 0.51	77.32 ± 0.29	87.17 ± 1.09	78.40 ± 0.17	69.19 ± 0.16	70.37 ± 0.37	76.09 ± 0.43
RGCL	75.03 ± 0.43	78.86 ± 0.48	87.66 ± 1.01	78.14 ± 1.08	70.92 ± 0.65	71.85 ± 0.84	77.07 ± 0.74
SIMGRACE	75.23 ± 0.19	77.45 ± 1.03	89.27 ± 1.39	79.10 ± 0.25	71.37 ± 0.44	71.45 ± 0.29	77.31 ± 0.59
GALOPA	76.93 ± 0.18	83.87 ± 0.42	91.11 ± 1.27	77.86 ± 0.36	73.20 ± 0.37	70.72 ± 0.48	78.94 ± 0.51
8.2 Performance Comparison
Performance under Node-level. Table 1 reports the averaged results over the node-level datasets. Comparing the results in Table 1, we have the following major observations. The proposed method outperforms the state-of-the-art self-supervised models significantly and even exceeds the supervised models on several datasets. For example, on PUBMED, GALOPA achieves  $84.57\%$  accuracy, which is a  $3.68\%$  relative improvement over previous state-of-the-art unsupervised algorithms. When compared to supervised baselines, it outperforms strong supervised baselines: On CORA, CITESEER and PUBMED benchmarks we observe  $2.67\%$ ,  $3.61\%$  and  $5.41\%$  relative improvement over GCN, respectively. On Coauthor-CS, the proposed unsupervised method shows competitive performance compared to the supervised models. Tabel 1 lists the average accuracy of 7 benchmark datasets, from which GALOPA achieves the best performance as well. For example, our proposed GALOPA
outperforms the unsupervised SOTA baseline SIMGRACE by  $1.84\%$  on average, and even outperforms supervised GCN by  $0.34\%$ . These results further validate that calibrating the backbone model by optimal transport alignment can produce expressive and generalizable representations.
Performance under Graph-level. In this section, we examine whether the proposed GALOPA performs better than state-of-the-art methods at graph-level datasets. The results of supervised learning baselines and unsupervised methods are reported in Table 2. The results shown in Table 2 suggest that GALOPA achieves state-of-the-art results with respect to unsupervised models. For example, on DD it achieves  $83.87\%$  accuracy, a  $4.09\%$  relative improvement over the previous state-of-the-art baselines. For kernel methods, our approach achieves better performance on most datasets. When compared to supervised baselines individually, our model outperforms GCN in 4 out of 6 datasets and outperforms GIN in 3 out of 6 datasets, e.g., a  $1.64\%$  relative improvement on GIN for the NCI1 dataset. Our approach outperforms the state-of-the-art graph contrastive learning approaches. For example, compared to SIMGRACE, which is one of the best SOTA methods, GALOPA has a relative improvement of  $1.63\%$  on average across all datasets. GALOPA outperforms GRAPHCL and INFOGRAPH with a relative improvement of  $2.25\%$  and  $3.01\%$  on average, respectively. To summarize, our newly proposed GALOPA for graph self-supervised has achieved SOTA performance.
Conclusion. In this paper, we investigated the self-supervised graph learning problem, addressing the challenges posed by label-invariant issues in contrasting graph learning. Unlike existing methods that adopt contrastive or distance-based regularization approaches, we propose a novel paradigm based on optimal transport for self-supervised graph learning. Our approach focuses on aligning optimal transport plans between the graph space and the representation space. By aligning the transport plans, our method enforces the backbone model to learn representations that precisely preserve the matching relationships in the original graph space. Our observations reveal several noteworthy findings: (i) The optimal transport plan serves as a more informative calibration signal for the encoder compared to the transport distance, capturing essential characteristics; (ii) It is feasible to utilize only node attributes as a correction signal for the backbone model, without relying on edge information; (iii) Our proposed graph self-supervised model eliminates the need to distinguish between positive and negative samples and overcomes the label-invariant assumption; Furthermore, extensive experiments conducted on multiple benchmark datasets demonstrate the state-of-the-art performance of our proposed framework in terms of generalizability and robustness.
Limitations and Future Work. Although the transport plan opens the door for direct communication between the input graph space and the output representation space, it also becomes a computational bottleneck for the model to some extent due to the limitation of optimal transport computation complexity. To reduce the time complexity, we can utilize the properties of the proposed model and/or the scaling optimal transport techniques that can reduce the time complexity from cubic to square or even to linear, we provide 4 ways to do this below: (i) Unlike general OT settings, where the two graphs are typically quite different and the matching relationship between them is completely unknown, the difference between the original and augmented graphs in GALOPA is quite small and the matching relations for subgraph components except with different part (i.e., complementary set of difference part) is known. This means that we can utilize the matching prior to reduce the computational cost. Hence, we can split the difference part with its neighborhood from the two graphs and compute the optimal transport plan only for that part. Since the percentage of that part is very small, it can greatly reduce the time complexity; (ii) According to the observation in Section 6, we can avoid the cubic complexity of optimizing GW by using only the node attributes for computing the optimal plan in graph space, while retaining similar performance with near-square time complexity; (iii) Alternatively, we can reduce the computational cost by utilizing sparsity [27] or graph partitioning [10, 67]. In particular, we can employ the most recent work on linear optimal transport [9, 38], which computes FGW term and/or Wasserstein term in linear time; (iv) We have the option to combine the aforementioned methods. For instance, by merging insights from the first point, a significant portion of subgraph pairs acquired via graph partitioning in the third point turns out to be identical. This realization can further pare down the complexity of partitioning methods.
As the main goal of this paper is to propose an alternative self-supervised graph learning paradigm beyond the label-invariant assumption that accurately links/communicates the input and output spaces, we leave the scalability issue as our future work.
We thank Edouard Pauwels and Samuel Vaiter for their valuable work and for proving the convergence of the derivatives of Sinkhorn-Knopp. We also thank the anonymous reviewers for their constructive suggestions. This project was in part supported by the following projects: the National Natural Science Foundation of China (No.62032013, No.92267206); Singapore Institute of Technology Ignition Grant (No.R-IE2-A405-0001).