SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 1 implies the faster convergence of marginal constraints than the strict joint constraint. This entails that with  $t$  Sinkhorn iterations of solving FGW, the solution of the relaxed FGW solver moves further than the strict one. This will benefit the convergence rate of the whole algorithm with a larger step size of  $\pi_i^{(k)}$  in each outer iteration.
Proposition 2. Let  $C_1, C_2$  be two convex sets, and  $f(\pi)$  denote the FGW function w.r.t.  $\pi$ . We denote  $\mathcal{X}$  as the critical point set of strict FGW that solves  $\min_{\pi} f(\pi) + \mathbb{I}_{\pi \in C_1} + \mathbb{I}_{\pi \in C_2}$ , defined by:  $\mathcal{X} = \{\pi \in C_1 \cap C_2 : 0 \in \nabla f(\pi) + \mathcal{N}_{C_1}(\pi) + \mathcal{N}_{C_2}(\pi)\}$ , and  $\mathcal{N}_C(\pi)$  is the normal cone to  $C$ .
at  $\pi$ . The fix-point set  $\mathcal{X}_{rel}$  of the relaxed FGW solving  $\min_{\pi, \omega} f(\pi) + f(\omega) + h(\pi) + h(\omega)$  is defined by:  $\mathcal{X}_{rel} = \{\pi \in C_1, \omega \in C_2 : 0 \in \nabla f(\pi) + \rho (\nabla h(\omega) - \nabla h(\pi)) + \mathcal{N}_{C_1}(\pi), \text{and } 0 \in \nabla f(\omega) + \rho (\nabla h(\pi) - \nabla h(\omega)) + \mathcal{N}_{C_2}(\omega)\}$  where  $h(\cdot)$  is the Bregman projection function. Then, the gap between  $\mathcal{X}_{rel}$  and  $\mathcal{X}$  satisfies:
$$
\exists \tau \in \mathbb {R}, \forall \left(\pi^ {*}, \omega^ {*}\right) \in \mathcal {X} _ {r e l}, \operatorname {d i s t} \left(\frac {\pi^ {*} + \omega^ {*}}{2}, \mathcal {X}\right) := \min  _ {x \in \mathcal {X}} \left\| \frac {\pi^ {*} + \omega^ {*}}{2} - x \right\| \leq \tau / \rho . \tag {7}
$$
This bounded gap ensures the correctness of FGwMixup* that as long as we select a step size  $\gamma = 1 / \rho$  that is small enough, the scale of the upper bound  $\tau / \rho$  will be sufficiently small to ensure the convergence to the ideal optimum of our algorithm. The detailed proofs of Propositions 1 and 2 are presented in Appendix B.
3.1 Experimental Settings
Datasets We evaluate our methods with five widely-used graph classification tasks from the graph benchmark dataset collection TUDataset [35]: NCI1 and NCI109 [36, 37] for small molecule classification, PROTEINS [38] for protein categorization, and IMDB-B and IMDB-M [5] for social networks classification. Noted that there are no node features in IMDB-B and IMDB-M datasets, we augment the two datasets with node degree features as in [2, 19, 20]. Detailed statistics on these datasets are reported in Appendix D.1.
Backbones Most existing works select traditional MPNNs such as GCN and GIN as the backbone. However, traditional MPNNs exhibit limited expressive power and sensitivity to graph structures (upper bounded by 1-Weisfeiler-Lehman (1-WL) test [17, 37]), while there exist various advanced GNN architectures [13, 14, 16] with stronger expressive power (upper bound promoted to 3-WL test). Moreover, the use of a global pooling layer (e.g., mean pooling) in graph classification models may further deteriorate their perception of intricate graph topologies. These challenges may undermine the reliability of the conclusions regarding the effectiveness of graph structure augmentation. Therefore, we attempt to alleviate the problems from two aspects. 1) We modify the READOUT approach of GIN and GCN to save as much structural information as we can. Following [39], we apply a virtual node that connects with all other nodes and use the final latent representation of the virtual node to conduct READOUT. The two modified backbones are dubbed vGIN and vGCN. 2) We select two Transformer-based GNNs with stronger expressive power as the backbones, namely Graphormer [39] and Graphormer-GD [16]. They are proven to be more sensitive to graph structures, and Graphormer-GD is even capable of perceiving cut edges and cut vertices in graphs. Detailed information about the selected backbones is introduced in Appendix D.2.
Comparison Baselines We select the following data augmentation methods as the comparison baselines. DropEdge [40] randomly removes a certain ratio of edges from the input graphs. DropNode [41] randomly removes a certain portion of nodes as well as the edge connections. M-Mixup [42] conducts Euclidean mixup in the latent spaces, which interpolates the graph representations after the READOUT function. ifMixup [19] applies an arbitrary node matching strategy to conduct mixup on graph node signals, without preserving key topologies of original graphs.  $\mathcal{G}$ -Mixup [20] conducts Euclidean addition on the estimated graphons of different classes of graphs to conduct class-level graph mixup. We also present the performances of the vanilla backbones, as well as our proposed method w/ and w/o acceleration, denoted as FGWMixup* and FGWMixup respectively.
Experimental Setups For a fair comparison, we employ the same set of hyperparameter configurations for all data augmentation methods in each backbone architecture. For all datasets, We randomly hold out a test set comprising  $10\%$  of the entire dataset and employ 10-fold cross-validation on the remaining data. We report the average and standard deviation of the accuracy on the test set over the best models selected from the 10 folds. This setting is more realistic than reporting results from validation sets in a simple 10-fold CV and allows a better understanding of the generalizability [43]. We implement our backbones and mixup algorithms based on Deep Graph Library (DGL) [44] and Python Optimal Transport (POT) [45] open-source libraries. More experimental and implementation details are introduced in Appendix D.3 and D.4.
Methods	PROTEINS	NCI1	NCI109	IMDB-B	IMDB-M
vGIN	vGCN	vGIN	vGCN	vGIN	vGCN	vGIN	vGCN	vGIN	vGCN
vanilla	74.93(3.02)	74.75(2.60)	76.98(1.87)	76.91(1.80)	75.70(1.85)	75.89(1.35)	71.30(4.96)	72.30(4.34)	49.00(2.64)	49.47(3.76)
DropEdge	73.59(2.50)	74.48(4.18)	76.47(2.85)	76.16(2.04)	75.38(2.05)	75.77(1.55)	73.30(3.85)	73.30(3.29)	49.47(2.66)	49.40(3.52)
DropNode	74.48(2.91)	75.11(3.00)	76.89(1.25)	77.42(1.71)	73.98(2.16)	75.45(1.90)	71.50(3.23)	73.20(5.58)	49.80(3.29)	50.00(3.41)
M-Mixup	74.40(3.00)	75.65(4.51)	76.45(3.39)	77.76(2.75)	75.41(2.78)	75.79(1.85)	72.20(4.83)	72.80(4.45)	49.13(3.25)	49.47(2.56)
ifMixup	74.76(3.71)	74.04(2.27)	76.16(1.78)	77.37(2.56)	76.13(1.87)	76.74(1.56)	72.50(3.98)	72.40(5.14)	49.07(3.16)	49.73(4.67)
G-Mixup	74.84(2.99)	74.57(2.88)	76.42(1.79)	77.79(1.88)	75.55(2.32)	76.38(1.79)	72.40(4.82)	72.20(6.45)	49.47(4.73)	49.60(3.90)
FGWmixup	75.02(3.86)	76.01(3.19)	78.32(2.65)	78.37(2.40)	76.40(1.65)	76.79(1.81)	73.00(3.69)	73.40(5.12)	49.80(2.63)	50.80(4.06)
FGWmixup*	75.20(3.30)	75.20(3.03)	77.27(2.71)	78.47(1.74)	76.64(2.60)	76.52(1.59)	73.50(4.54)	74.00(2.90)	49.20(3.38)	50.47(5.44)
Methods	Graphomer GraphomerGD	Graphomer GraphomerGD	Graphomer GraphomerGD	Graphomer GraphomerGD	Graphomer GraphomerGD
vanilla	75.47(3.16)	76.01(2.02)	61.56(3.70)	77.49(2.01)	65.54(3.04)	74.99(1.23)	70.40(5.00)	71.50(4.20)	48.87(4.10)	47.47(2.98)
DropEdge	75.20(4.02)	75.12(3.22)	63.07(3.21)	74.94(2.44)	66.73(3.50)	74.73(3.22)	71.10(5.65)	72.30(3.93)	49.60(4.09)	46.67(3.85)
DropNode	75.20(2.13)	76.28(3.49)	64.96(2.18)	76.20(1.95)	63.73(3.46)	74.78(2.07)	71.60(5.18)	71.30(5.18)	48.47(4.08)	47.67(2.83)
M-Mixup	75.11(3.78)	74.39(3.83)	62.31(3.48)	75.47(1.45)	66.54(2.70)	74.61(1.86)	71.10(4.83)	70.50(4.70)	49.67(4.25)	48.00(3.85)
G-Mixup	75.74(3.12)	74.85(3.52)	63.07(4.40)	76.06(3.12)	65.03(2.98)	74.90(2.04)	72.10(6.38)	71.10(5.01)	46.93(5.18)	46.80(4.41)
FGWmixup	76.82(2.35)	77.18(3.48)	66.45(2.58)	78.20(1.88)	67.36(3.21)	76.01(3.04)	72.60(5.08)	72.40(4.48)	49.73(3.80)	48.87(4.03)
FGWmixup*	76.19(3.20)	76.46(3.41)	64.26(3.25)	76.62(3.06)	67.46(2.82)	75.45(1.80)	71.70(4.17)	71.90(4.35)	50.27(4.26)	48.53(2.95)
Table 1: Test set classification accuracy  $(\%)$  from 10-fold CV on five benchmark datasets. Results are presented with the form of avg.(stddev.), and the best-performing method is highlighted in boldface. Note: ifMixup has no results on Graphormers because it generates graphs with continuous edge weights and is inapplicable for the Spatial Encoding module in the Graphormer-based architectures.
3.2 Experimental Results
Main Results We compare the performance of various GNN backbones on five benchmark datasets equipped with different graph data augmentation methods and summarize the results in Table 1. As is shown in the table, FGWMixup and FGWMixup* consistently outperform all other SOTA baseline methods, which obtain 13 and 7 best performances out of all the 20 settings respectively. The superiority is mainly attributed to the adoption of the FGW metric. Existing works hardly consider the node matching problem to align two unregistered graphs embedded in the signal-structure fused metric spaces, whereas the FGW metric searches the optimum from all possible couplings and conducts semantic-invariant augmentation guided by the optimal node matching strategy. Remarkably, despite introducing some infeasibility for acceleration, FGWMixup* maintains consistent performance due to the theoretically controlled gap with the ideal optimum, as demonstrated in Prop. 1. Moreover, FGWMixup and FGWMixup* both effectively improve the performance and generalizability of various GNNs. Specifically, our methods achieve an average relative improvement of  $1.79\%$  on MPNN backbones and  $2.67\%$  on Graphormer-based backbones when predicting held-out unseen samples. Interestingly, most SOTA graph data augmentation methods fail to bring notable improvements and even degrade the performance of Graphormers, which exhibit stronger expressive power and conduct more comprehensive interactions between graph signal and structure spaces. For instance, none of the baseline methods improve GraphormerGD on NCI109 and NCI1 datasets. However, our methods show even larger improvements on Graphormers compared to MPNNs, as we explicitly consider the interactions between the graph signal and structure spaces during the mixup process. In particular, the relative improvements of FGWMixup reach up to  $7.94\%$  on Graphormer and  $2.95\%$  on GraphormerGD. All the results above validate the effectiveness of our methods. Furthermore, we also conduct experiments on large-scale OGB [46] datasets, and the results are provided in Appendix E.5.
Robustness against Label Corruptions In this subsection, we evaluate the robustness of our methods against noisy labels. Practically, we introduce random label corruptions (i.e., switching to another random label) with ratios of  $20\%$ ,  $40\%$ , and  $60\%$  on the IMDB-B and NCI1 datasets. We employ vGCN as the backbone model and summarize the results in Table 2. The results evidently demonstrate that the mixup-series methods consistently outperform the in-sample augmentation method (DropEdge) by a significant margin. This improvement can be attributed to the soft-labeling strategy employed by mixup, which reduces the model's sensitivity to individual mislabeled instances and encourages the model to conduct more informed predictions based on the overall distribution of the blended samples. Notably, among all the mixup methods, FGWMixup and FGWMixup* exhibit stable and consistently superior performance under noisy label conditions. In conclusion, our methods effectively enhance the robustness of GNNs against label corruptions.
Analysis on Mixup Efficiency We run FGwMixup and FGwMixup* individually with identical experimental settings (including stopping criteria and hyper-parameters such as mixup ratio, etc.) on the same computing device and investigate the run-time computational efficiencies. Table 3 illustrates
Methods	IMDB-B	NCI1
20%	40%	60%	20%	40%	60%
vanilla	70.00(5.16)	59.70(5.06)	47.90(4.30)	70.58(1.29)	61.95(2.19)	48.25(4.87)
DropEdge	68.30(5.85)	59.40(5.00)	50.10(1.92)	69.51(2.27)	60.32(2.60)	49.61(1.28)
M-Mixup	70.70(5.90)	59.70(5.87)	50.90(1.81)	71.53(2.75)	63.24(2.59)	48.66(3.02)
G-Mixup	67.50(4.52)	59.10(4.74)	49.40(2.87)	72.46(1.95)	63.26(4.39)	50.01(1.26)
FGWMixup	70.10(4.39)	61.90(6.17)	50.80(3.19)	72.92(1.56)	62.99(1.35)	50.12(3.51)
FGWMixup*	70.80(3.97)	61.80(5.69)	51.00(1.54)	72.75(2.29)	63.55(2.60)	50.02(3.38)
Table 2: Experimental results of robustness against label corruption with different ratios.
Datasets	Avg. Mixup Time (s) / Fold
PROTEINS	NCI1	NCI109	IMDB-B	IMDB-M
FGWMixup	802.24	1711.45	1747.24	296.62	212.53
FGWMixup*	394.57	637.41	608.61	85.69	74.53
Speedup	2.03×	2.67×	2.74×	3.46×	2.85×
the average time spent on the mixup procedures of FGWMixup and FGWMixup* per fold. We can observe that FGWMixup* decreases the mixup time cost by a distinct margin, providing at least  $2.03 \times$  and up to  $3.46 \times$  of efficiency promotion. The results are consistent with our theoretical analysis of the convergence rate improvements, as shown in Proposition 1. More detailed efficiency statistics and discussions are introduced in Appendix E.3.
Infeasibility Analysis on the Single-loop FGW Solver in FGwMixup* We conduct an experiment to analyze the infeasibility of our single-loop FGW solver compared with the strict CG solver. Practically, we randomly select 1,000 pairs of graphs from PROTEINS dataset and apply the two solvers to calculate the FGW distance between each pair of graphs. The distances of the  $i$ -th pair of graphs calculated by the strict solver and the relaxed solver are denoted as  $d_i$  and  $d_i^*$ , respectively. We report the following metrics for comparison:
- MAE: Mean absolute error of FGW distance, i.e.,  $\frac{1}{N}\sum |d_i - d_i^*|$ .
- MAPE: Mean absolute percentage error of FGW distance, i.e.,  $\frac{1}{N} \sum \frac{|d_i - d_i^*|}{d_i}$ .
- mean-FGW: Mean FGW distance given by the strict CG solver, i.e.,  $\frac{1}{N}\sum d_i$ .
- mean-FGW*: Mean FGW distance given by the single-loop solver, i.e.,  $\frac{1}{N}\sum d_i^*$
- T-diff: L2-norm of the difference between two transportation plan matrices (divided by the size of the matrix for normalization).
Table 3: Comparisons of algorithm execution efficiency between FGWMixup and FGWMixup*.
MAE	MAPE	mean-FGW	mean-FGW*	T-diff
0.0126(0.0170)	0.0748(0.1022)	0.2198	0.2143	0.0006(0.0010)
Table 4: Infeasibility analysis on the single-loop FGW solver in FGWMixup*.
The results are shown in Table 4. We can observe that the MAPE is only  $0.0748$ , which means the FGW distance estimated by the single-loop relaxed solver is only  $7.48\%$  different from the strict CG solver. Moreover, the absolute error is around 0.01, which is quite small compared with the absolute value of FGW distances ( $\sim 0.21$ ). We can also find that the L2-norm of the difference between two transportation plan matrices is only 0.0006, which means two solvers give quite similar transportation plans. All the results imply that the single-loop solver will not produce huge infeasibility or make the estimation of FGW distance inaccurate.
3.3 Further Analyses
Effects of the Trade-off Coefficient  $\alpha$  We provide sensitivity analysis w.r.t. the trade-off coefficient  $\alpha$  of our proposed mixup methods valued from  $\{0.05, 0.5, 0.95, 1.0\}$  on two backbones and two
Methods	PROTEINS	NCI1
α=0.95	α=0.5	α=0.05	α=1.0	α=0.95	α=0.5	α=0.05	α=1.0
vGIN-FGWMixup	75.02(3.86)	75.38(2.58)	74.86(2.40)	74.57(2.62)	78.32(2.65)	77.42(1.93)	77.62(2.37)	75.91(2.93)
vGCN-FGWMixup	76.01(3.19)	75.47(3.56)	74.93(2.74)	74.40(3.57)	78.37(2.40)	77.93(1.68)	78.00(1.00)	77.27(0.92)
vGIN-FGWMixup*	75.20(3.30)	74.57(3.30)	74.39(3.01)	73.94(3.86)	77.27(2.71)	77.23(2.47)	76.59(2.14)	77.20(1.69)
vGCN-FGWMixup*	75.20(3.03)	74.57(3.52)	74.84(3.16)	74.66(2.91)	78.47(1.74)	77.66(1.48)	78.93(1.91)	77.71(1.97)
Table 5: Experimental results of different  $\alpha$  on PROTEINS and NCI1 datasets.
Figure 1: Test performance of our methods with different graph size settings on NCI1 and PROTEINS datasets using vGCNs as the backbone.
datasets. Note that  $\alpha$  controls the weights of the node feature alignment and graph structure alignment costs, and  $\alpha = 1.0$  falls back to the case of GW metric where node features are not incorporated. From the results shown in Table 5, we can observe that: 1) when FGW falls back to GW ( $\alpha = 1$ ), where node features are no longer taken into account, the performance will significantly decay (generally the worst among all investigated  $\alpha$  values). This demonstrates the importance of solving the joint modeling problem in graph mixup tasks. 2)  $\alpha = 0.95$  is the best setting in most cases. This empirically implies that it is better to conduct more structural alignment in graph mixup. In practice, we set  $\alpha$  to 0.95 for all of our reported results.
Effects of Mixup Graph Sizes We investigate the performance of FGWMixup and FGWMixup* with various mixup graph sizes (node numbers), including adaptive graph sizes (i.e., weighted average size of the mixup source graphs,  $\tilde{n} = \lambda n_1 + (1 - \lambda)n_2$ , which is selected as our method) and fixed graph sizes (i.e., the median size of all training graphs,  $0.5 \times$  and  $2 \times$  the median). The ablation studies are composed on NCI1 and PROTEINS datasets using vGCN backbone as motivating examples, and the results are illustrated in Fig.1. We can observe that the best performance is steadily obtained when selecting graph sizes with an adaptive strategy, whereas the fixed graph sizes lead to unstable results. Specifically, selecting the median or larger size may occasionally yield comparable performances, yet selecting the smaller size can result in an overall performance decay of over  $1\%$ . This phenomenon is associated with the graph size generalization problem [47, 48] that describes the performance degradation of GNNs caused by the graph size distributional shift between training and testing data. The fixed strategy may aggravate this problem, particularly for small graphs that struggle to generalize to larger ones. In contrast, the adaptive strategy can potentially combat this distributional shift by increasing the data diversity and reach better test time performance.
Effects of GNN Depths To validate the improvement of our methods across various model depths, we evaluate the performance of FGWMixup and FGWMixup\* using vGCNs equipped with different numbers (3-8) of layers. We experiment on NCI1 and PROTEINS datasets, and the results are illustrated in Fig.2. We can observe that our methods consistently improve the performance on NCI1 under all GCN depth settings by a significant margin. The same conclusion is also true for most cases on PROTEINS except for the 7-layer vGCN. These results indicate a universal improvement of our methods on GNNs with various depths.
Other Discussions More further analyses are introduced in the Appendix, including qualitative analyses of our mixup results (see Appendix E.1), further discussions on  $\mathcal{G}$ -Mixup (see Appendix E.2), and sensitivity analysis of the hyperparameter  $k$  in Beta distribution where mixup weights are sampled from (see Appendix E.4).
Figure 2: Test performance of our methods using vGCNs with varying numbers of layers on NCI1 and PROTEINS datasets.
Graph Data Augmentation There are currently two mainstream perspectives of graph data augmentation for graph-level classifications. A line of research concentrates on graph signal augmentation [41, 18, 19]. Node Attribute Masking [41] assigns random node features for a certain ratio of graph nodes. IfMixup [19] conducts Euclidean mixup on node features from different graphs with arbitrary node alignment strategy, whereas it damages the critical topologies (e.g., rings, bipartibility) of the source graphs. Another line of work focuses on graph structure augmentation [49, 20, 21, 22, 50, 41, 51]. Approaches like Subgraph[49, 41], DropEdge[40] and GAug[50] conduct removals or additions of graph nodes or edges to generate new graph structures. Graph Transplant[21] and Submix[22] realize CutMix-like augmentations on graphs, which cut off a subgraph from the original graph and replace it with another.  $\mathcal{G}$ -Mixup [20] estimates a graph structure generator (i.e., graphon) for each class of graphs and conducts Euclidean addition on graphons to realize structural mixup. However, as GNNs can capture the complex interaction between the two entangled yet complementary input spaces, it is essential to model this interaction for a comprehensive graph data augmentation. Regrettably, current works have devoted little attention to this interaction.
Optimal Transport on Graphs Building upon traditional Optimal Transport (OT) [25] methods (e.g., Wasserstein distance[52]), graph OT allows defining a very general distance metric between the structured/relational data points embedded in different metric spaces, where the data points are modeled as probability distributions. It proceeds by solving a coupling between the distributions that minimizes a specific cost. The solution of graph OT hugely relies on the (Fused) Gromov-Wasserstein (GW) [27, 26] distance metric. Further works employ the GW couplings for solving tasks such as graph node matching and partitioning [53, 32, 54], and utilize GW distance as a common metric in graph metric learning frameworks [55, 31, 56]. Due to the high complexity of the GW-series algorithm, another line of research [28, 57, 58, 29] concentrates on boosting the computational efficiency. In our work, we formulate graph mixup as an FGW-based OT problem that solves the optimal node matching strategy minimizing the alignment costs of graph structures and signals. Meanwhile, we attempt to accelerate the mixup procedure with a faster convergence rate.
In this work, we introduce a novel graph data augmentation method for graph-level classifications dubbed FGWMixup. FGWMixup formulates the mixup of two graphs as an optimal transport problem aiming to seek a "midpoint" of two graphs embedded in the graph signal-structure fused metric space. We employ the FGW distance metric to solve the problem and further propose an accelerated algorithm that improves the convergence rate from  $\mathcal{O}(t^{-1})$  to  $\mathcal{O}(t^{-2})$  for better scalability of our method. Comprehensive experiments demonstrate the effectiveness of FGWMixup in terms of enhancing the performance, generalizability, and robustness of GNNs, and also validate the efficiency and correctness of our acceleration strategy.
Despite the promising results obtained in our work, it is important to acknowledge its limitations. Our focus has primarily been on graph data augmentation for graph-level classifications. However, it still remains a challenging question to better exploit the interactions between the graph signal and structure spaces for data augmentation in other graph prediction tasks, such as link prediction and node classification. Moreover, we experiment with four classic (MPNNs) and advanced (Graphormers) GNNs, while there remain other frameworks that could be taken into account. We expect to carry out a graph classification benchmark with more comprehensive GNN frameworks in our future work.
This work was supported by the National Natural Science Foundation of China (No.82241052).