SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- vGCN [1]: Graph Convolution Network is developed from the spectral GNNs with 1-order approximation of Chebychev polynomial expansion. The update of features can be regarded as message passing procedure. The graph convolution operator of each layer in the vGCN is
defined as:  $\mathbf{X}^{(l + 1)} = \sigma (\tilde{\mathbf{A}}\mathbf{X}^{(l)}\mathbf{W}^{(l)})$ , where  $\tilde{\pmb{A}}$  is the renormalized adjacency matrix, and  $\mathbf{W}$  is the learnable parameters. We apply ReLU as the activation function  $\sigma (\cdot)$ .
- vGIN [2]: Graph Isomorphism Network takes the idea from the Weisfeiler-Lehman kernel, and define the feature update of each layer through explicit message passing procedure. The GIN layer takes the form of:  $x_{v}^{(l + 1)} = \mathrm{MLP}^{(l + 1)}\left(\left(1 + \epsilon^{(l + 1)}\right)\cdot x_{v}^{(l)} + \sum_{u\in \mathcal{N}(v)}x_{u}^{(l)}\right)$ , where  $\epsilon$  is a learnable scalar. We apply two-layer MLP with ReLU activation in our implementation.
- Graphormer [39]: Graphormer adopts the idea of the Transformer [62] architecture, which designs a multi-head global QKV-attention across all graph nodes, and encode graph topologies as positional encodings (spatial encoding in Graphormers). A Graphormer layer conducts feature updating as follows:
$$
\mathbf {A} ^ {h} \left(\mathbf {X} ^ {(l)}\right) = \operatorname {s o f t m a x} \left(\mathbf {X} ^ {(l)} \mathbf {W} _ {Q} ^ {l, h} \left(\mathbf {X} ^ {(l)} \mathbf {W} _ {K} ^ {l, h}\right) ^ {\top} + \phi^ {l, h} (\mathbf {D})\right);
$$
$$
\begin{array}{l} \hat {\mathbf {X}} ^ {(l)} = \mathbf {X} ^ {(l)} + \sum_ {h = 1} ^ {H} \mathbf {A} ^ {h} \left(\mathbf {X} ^ {(l)}\right) \mathbf {X} ^ {(l)} \mathbf {W} _ {V} ^ {l, h} \mathbf {W} _ {O} ^ {l, h}; \\ \mathbf {X} ^ {(l + 1)} = \hat {\mathbf {X}} ^ {(l)} + \mathrm {G E L U} \left(\hat {\mathbf {X}} ^ {(l)} \mathbf {W} _ {1} ^ {l}\right) \mathbf {W} _ {2} ^ {l}, \\ \end{array}
$$
where  $\phi (\cdot)$  is the spatial encoding module implemented as embedding tables, and  $\mathbf{D}$  is the shortest path distance of the graph.  $\mathbf{A}$  is the attention matrix.
- GraphormerGD [16]: GraphormerGD is an enhanced version of Graphormer, where the graph topologies are encoded with a General Distance metric defined by the combination of the shortest path distance (SPD) and the resistance distance [63] (RD). It mainly modifies the calculation of the attention matrix  $\mathbf{A}$  in Graphormer:
$$
\mathbf {A} ^ {h} \left(\mathbf {X} ^ {(l)}\right) = \phi_ {1} ^ {l, h} (\mathbf {D} _ {s p}, \mathbf {D} _ {r}) \odot \operatorname {s o f t m a x} \left(\mathbf {X} ^ {(l)} \mathbf {W} _ {Q} ^ {l, h} \left(\mathbf {X} ^ {(l)} \mathbf {W} _ {K} ^ {l, h}\right) ^ {\top} + \phi_ {2} ^ {l, h} \left(\mathbf {D} _ {s p}, \mathbf {D} _ {r}\right)\right)
$$
where  $\phi (\cdot)$  is an MLP taking SPD embedding and RD gaussian kernel embedding as inputs.
For all the backbones, we apply the virtual node READOUT approach (which is the official READOUT function of Graphormers) instead of traditional global pooling. We apply 6 network layers and take 64 as the hidden dimension for all the backbones on all datasets. GraphormerGD has a special model design on PROTEINS dataset, which applies 5 layers and 32 as the hidden dimension due to the GPU memory limitation.
D.3 Experimental Environment
Hardware Environment The experiments in this work are conducted on two machines: one with 8 Nvidia RTX3090 GPUs and Intel Xeon E5-2680 CPUs, one with 2 Nvidia RTX8000 GPUs and Intel Xeon Gold 6230 CPUs.
Software Environment Our experiments are implemented with Python 3.9, PyTorch 1.11.0, Deep Graph Library (DGL) [44] 1.0.2, and Python Optimal Transport (POT) [45] 0.8.2. The implementation of all the backbone layers are based on their DGL implementations. For the FGW solver, FGWMixup uses the official FGW distance solver implemented in POT, and FGWMixup* uses the accelerated FGW algorithm implemented on our own.
D.4 Implementation Details
For our mixup algorithm FGWMixup, we follow [31] and apply the Euclidean distance as the metric measuring the distance of the node features and the graph adjacency matrix as the structural distances between nodes. We sample the mixup weight  $\lambda$  from the distribution  $\mathrm{Beta}(0.2, 0.2)$  and the trade-off coefficient of the structure and signal costs  $\alpha$  are tuned in  $\{0.05, 0.5, 0.95\}$ . We set the size of the generated mixup graphs as the weighted average of the source graphs, which is  $\tilde{n} = \lambda n_1 + (1 - \lambda)n_2$ . The maximum number of iterations of FGWMixupis set to 200 for the outer loop optimizing  $X$  and  $A$ , and 300 for the inner loop optimizing couplings  $\pi$ . The stopping criteria of our algorithm are the relative update of the optimization objective reaching below a threshold, which is set to 5e-4. For
FGWMixup\*, we select the step size of MD  $\gamma$  from  $\{0.1, 1, 10\}$ . The mixup ratio (i.e., the proportion of mixup samples to the original training samples) is set to 0.25.
For the training of GNNs, MPNNs are trained for 400 epochs and Graphormers are trained for 300 epochs, both using AdamW optimizer with a weight decay rate of 5e-4. The batch size of GNNs are chosen from  $\{32,128\}$ , and the learning rate is chosen from  $\{1e - 3,5e - 4,1e - 4\}$ . Dropout is employed with a fixed dropout rate 0.5 to prevent overfitting. All the hyperparameters are fine-tuned by grid search on validation sets. For a fair comparison, we employ the same set of hyperparameter configurations for all data augmentation methods in each backbone architecture. For more details, please check our code published online at https://github.com/ArthurLeoM/FGWMixup.
E Further Experimental Analyses
E.1 Qualitative Analyses
We present two mixup examples of FGWMixup in Figure 4 to demonstrate that augmented graphs by FGWMixup can preserve key topologies of the original graphs and have semantically meaningful node features simultaneously. In Example 1, we can observe that the mixup graph adopts an overall trident structure and a substructure (marked green) from  $G_{1}$ , and adopts several substructures from  $G_{2}$  (marked red), finally formulating a new graph sample combined properties from both graphs. In Example 2, we can observe that the mixup graph is quite similar to  $G_{2}$ , but breaks the connection of two marked (red arrow pointed) edges and formulates two disconnected subgraphs, which is identical to the overall structure of  $G_{1}$ . Moreover, in both examples, we can observe that the preserved substructures are not only topologically alike, but also highly consistent in node features. The two examples demonstrate that FGWMixup can both preserve key topologies and generate semantically meaningful node features.
Figure 4: Two examples of FGWMixup. In each example, the subfigures on the left and middle are the original graphs to be mixed up, denoted as  $G_{1}$  and  $G_{2}$ , respectively. The subtitle denotes the mixup ratio  $\lambda$ . The subfigure on the right is the synthetic mixup graph. The node features are one-hot encoded and distinguished with the feature ID and corresponding color.
G1
Example 1
G2
Example 2
G2
E.2 Further Discussions on  $\mathcal{G}$ -Mixup
In this subsection, we conduct further analyses on  $\mathcal{G}$ -Mixup for a better understanding of its differences from our methods.
$\mathcal{G}$ -Mixup with GW Graphon Estimator  $\mathcal{G}$ -Mixup does not originally apply GW metrics to estimate the graphons (as they introduce in Table 1 and 6 in their paper [20]). In our implementation, we select the USVT method as the graphon estimator for  $\mathcal{G}$ -Mixup. Yet it is also practical to apply GW metric to estimate graphons as an ablation study. Here, we also provide the experimental results of  $\mathcal{G}$ -Mixup+GW and  $\mathcal{G}$ -Mixup+GW* (GW with our single-loop solver) on PROTEINS as shown in Table 7.
Backbone	G-Mixup	G-Mixup+GW	G-Mixup+GW*	FGWMixup
vGIN	74.84(2.99)	74.48(2.54)	74.03(4.46)	75.02(3.86)
vGCN	74.57(2.88)	74.57(3.18)	74.84(2.15)	76.01(3.19)
No matter what metrics are applied for the graphon estimation, we want to emphasize that  $\mathcal{G}$ -Mixup does not model the joint distribution of graph structure and signal spaces, but regards them as two disentangled and independent factors. However, our main contribution and the greatest advantage compared with  $\mathcal{G}$ -Mixup comes from the joint modeling of graph structure and signal spaces. This also explains why FGwMixup can outperform.
Extend  $\mathcal{G}$ -Mixup with FGW-based Attributed Graphon Estimator In [64], Xu et al. introduce an attributed graphon estimation algorithm, which makes us naturally think of an extended version of  $\mathcal{G}$ -Mixup. With the attributed graphon,  $\mathcal{G}$ -Mixup can also consider the joint modeling problem. Though it is not the contribution and the core idea from  $\mathcal{G}$ -Mixup to address our proposed joint modeling problem, we are also interested in whether this extension can improve the performance of  $\mathcal{G}$ -Mixup. Hence, we conduct experiments with the extended version of  $\mathcal{G}$ -Mixup using FGW as the attributed graphon estimator (denoted as  $\mathcal{G}$ -Mixup+FGW) on PROTEINS and NCI1 datasets, and the results are shown in Table 8.
Table 7: Experimental results of different graphon estimators of  $\mathcal{G}$ -Mixup on PROTEINS.
Dataset	Backbone	G-Mixup	G-Mixup+FGW	FGWMixup
PROTEINS	vGIN	74.84(2.99)	74.66(3.51)	75.02(3.86)
vGCN	74.57(2.88)	74.30(2.85)	76.01(3.19)
NCI1	vGIN	77.79(1.88)	78.18(1.73)	78.37(2.40)
vGCN	76.42(1.79)	75.91(1.54)	78.32(2.65)
Table 8: Experimental results of  $\mathcal{G}$ -Mixup extended with FGW attributed graphon estimator on PROTEINS and NCI1.
We can observe that FGW does not significantly improve the performance of  $\mathcal{G}$ -Mixup and still cannot outperform our methods. The main reason lies in that the node matching problem remains unsolved using the linear interpolation strategy of two graphons introduced in  $\mathcal{G}$ -Mixup. Though the intra-class node matching has been done with FGW graphon estimation, the graphons of different classes are not ensured with an aligned node distribution, which requires the inter-class node matching for an appropriate mixup. This is the core reason for the limited performance of  $\mathcal{G}$ -Mixup series methods.
E.3 Algorithm Efficiency
In this subsection, we provide additional details regarding the runtime efficiency analyses.
Further Comparisons between FGWMixup and FGWMixup. Table 9 presents the average time spent on a single iteration of  $\pi$  update (i.e, FGW solver convergence) and the average iterations taken for updating  $X$  and  $A$  until convergence. We can observe that the convergence of a single FGW procedure does not appear to be significantly accelerated. This is because the gradient step
has been taken twice for an alternating projection process, which incurs double time for the gradient calculation (with a complexity of  $O(n^{3})$ ,  $n$  is the graph size). In contrast, the strict solver only takes one gradient step for each projection. Therefore, despite a faster convergence rate of the relaxed FGW, the extra gradient step makes the overall convergence time of the relaxed FGW solver similar to that of the strict FGW solver. However, a notable improvement is observed in the convergence rate of the outer loop responsible for updating  $\mathbf{X}$  and  $\mathbf{A}$ , resulting from the larger step of  $\pi$  that the relaxed FGW provides, as shown in Prop.1. Considering the two factors above, the overall mixup time has been efficiently decreased by up to  $3.46 \times$  with FGWMixup*.
Dataset	Single FGW Iter. Time (s)	Avg. Outer Loop Converge Iter. (#)
FGWmixup*	FGWmixup	FGWmixup*	FGWmixup	Speedup
PROTEINS	0.0181	0.0225	80.64	153.69	1.91×
NCI1	0.0140	0.0120	54.25	170.23	3.14×
NCI109	0.0136	0.0123	53.12	169.98	3.20×
IMDB-B	0.0162	0.0119	20.03	95.48	4.77×
IMDB-M	0.0081	0.0060	14.26	60.91	4.27×
Comparisons between Our Methods and Compared Baselines We present the averaged efficiencies of different mixup methods and time spent on training vanilla backbones of each fold on PROTEINS and NCI1 datasets in Table 10. As the efficiency of  $\mathcal{G}$ -Mixup hugely relies on the graphon estimation approach, we also include  $\mathcal{G}$ -Mixup with GW graphon estimators (denoted as G-Mixup+GW) in the table.
We can observe that our methods and  $\mathcal{G}$ -Mixup+GW are slower than the other data augmentation methods. The main reason is that the complexity of calculating (F)GW distances between two graphs is cubic ( $\mathcal{O}(mn^2 + nm^2)$ )[28], where  $m, n$  are the sizes of two graphs. Moreover, when calculating barycenters, we need an outer loop with  $T$  iterations and  $M$  graphs. In total, the time complexity of mixing up two graphs of size  $n$  is  $\mathcal{O}(MTn^3)$ . FGWMixup* boosts the efficiency by enhancing the convergence rate and reducing the required iterations  $T$  (see Table 9 for more details), whereas  $\mathcal{G}$ -Mixup+GW will have to go over the whole dataset to calculate graphons, which is much more time-consuming than FGWMixup.
However, sacrificing complexity to pursue higher performance has been the recent trend of technical development, e.g. GPT-4. Moreover, we believe that the current time complexity of FGWMixup* is still acceptable compared with our performance improvements, as most compared graph augmentation methods cannot effectively enhance the model performance as shown in Table 1.
More importantly, in practice, the main computational bottleneck of (F)GW-based method is the OT network flow CPU solver in the current implementation based on the most widely used POT lib. In other words, GPU-based network flow algorithms have not been applied in current computation frameworks. Moreover, mini-batch parallelization is not yet deployed in POT. However, recent works [65] from NVIDIA have focused on accelerating network flow algorithms on GPU, which may probably be equipped on CUDA and allow a huge acceleration for GW solvers in the near future. Hence, we firmly believe that the fact that our method is not yet optimized in parallel on GPUs is only a temporary problem. Just as some other works (e.g., MLP, LSTM, etc.) that have brought enormous contributions in the past era, they are initially slow when proposed, but have become efficient with fast-following hardware supports.
Table 9: More algorithm execution efficiency details of FGwMixup and FGwMixup*.
Dataset	DropEdge	DropNode	G-Mixup	ifMixup	G-Mixup+GW	FGWMixup	FGWMixup*
PROTEINS	0.192	0.229	6.34	2.08	2523.78	802.24	394.57
NCI1	0.736	0.810	10.31	5.67	9657.48	1711.45	637.41
Table 10: Average mixup efficiency (clock time spent, seconds) of all compared baselines on each fold of PROTEINS and NCI1 datasets.
E.4 Sensitivity Analysis on Varying Beta Distribution Parameter  $k$
We empirically follow the setting in [24] to select the beta distribution parameter  $k = 0.2$  where the mixup method is first proposed. We also provide the sensitivity analysis of the Beta distribution parameter  $k$  on PROTEINS and NCI1 datasets in Table 11.
Methods	PROTEINS	NCI1
k=0.2	k=0.5	k=1.0	k=2.5	k=0.2	k=0.5	k=1.0	k=2.5
vGIN-FGWMixup	75.02(3.86)	75.02(2.67)	74.93(2.93)	74.39(1.07)	78.32(2.65)	76.37(2.06)	76.59(2.39)	77.71(2067)
vGCN-FGWMixup	76.01(3.19)	75.47(3.12)	75.95(2.58)	74.30(4.12)	78.37(2.40)	78.00(1.40)	77.98(1.50)	77.96(1.73)
vGIN-FGWMixup*	75.20(3.30)	73.59(2.38)	73.86(2.81)	76.10(2.97)	77.27(2.71)	77.25(2.09)	77.32(1.78)	77.01(2.14)
vGCN-FGWMixup*	75.20(3.03)	74.29(4.62)	75.38(3.41)	74.21(4.52)	78.47(1.74)	78.71(1.49)	78.10(1.71)	77.96(1.02)
Table 11: Experimental results of different  $k$  on PROTEINS and NCI1 datasets.
From the results, we can find that  $\mathrm{Beta}(0.2, 0.2)$  is the overall best-performed setting. There are also a few circumstances where the other settings outperform. In our opinion, different datasets and backbones prefer different optimal settings of  $k$ . However, we should choose the one that is overall the best across various settings.
E.5 Experiments on OGB Datasets
We also conduct experiments on several OGB benchmark datasets[46], including ogbg-molhiv and ogbg-molvace. We conduct binary classification (molecular property prediction) on these datasets with AUROC (Area Under Receiver Operating Characteristic) as the reported metric. The split of training, validating, and testing datasets are provided by the OGB benchmark. In spite of the existence of edge features in OGB datasets, in this work, both of our GNN backbones and augmentation methods do not encode or consider the edge features. We select vGCN and vGIN (5 layers and 256 hidden dimensions) as the backbones and train the models five times with different random seeds, and we report the average and standard deviation of AUROCs as the results. The dataset information and the predicting performances are listed in Table 12.
From the table, we can observe evident improvements on both datasets and backbones with our methods. FGWMixup* obtains the best performance on ogbg-molhiv dataset, with  $2.69\%$  and  $3.46\%$  relative improvements on vGCN and vGIN respectively, and FGWMixup obtains the best on ogbg-molbace dataset with  $1.14\%$  and  $8.86\%$  relative improvements. Meanwhile, our methods provide a much more stable model performance where we reduce the variance by a significant margin, which demonstrates that our methods can resist the potential noises underlying the data and increase the robustness of GNNs. These additional experimental results further indicate the effectiveness of our methods in improving the performance of GNNs in terms of their generalizability and robustness.
Stats	ogbg-molhiv	ogbg-molbace
Graphs	41,127	1,513
Avg. Nodes	25.5	34.1
Avg. Edges	27.5	36.9
Feature Dim.	9	9
Backbones	Methods	ogbg-molhiv	ogbg-molbace
vGCN	vanilla	73.72(2.45)	76.62(4.59)
DropEdge	74.07(2.68)	75.31(4.57)
G-Mixup	74.65(1.42)	69.80(6.41)
FGWMixup	75.24(2.78)	77.49(2.71)
FGWMixup*	75.70(1.15)	76.40(2.45)
vGIN	vanilla	72.61(1.01)	69.77(1.33)
DropEdge	72.97(1.61)	72.68(4.92)
G-Mixup	74.52(1.58)	73.74(7.07)
FGWMixup	72.68(1.47)	75.95(2.42)
FGWMixup*	75.12(1.02)	74.02(2.18)
Table 12: Statistics and experimental results on OGB benchmark datasets. The reported metrics are AUROCs taking the form of avg.(stddev.).