Title: Fused Gromov-Wasserstein Graph Mixup for Graph-level Classifications
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Data Augmentation

Graph Mixup

Fused Gromov Wasserstein
Abstract: Graph data augmentation has shown superiority in enhancing generalizability and robustness of GNNs in graph-level classifications. However, existing methods primarily focus on the augmentation in the graph signal space and the graph structure space independently, neglecting the joint interaction between them. In this paper, we address this limitation by formulating the problem as an optimal transport problem that aims to find an optimal inter-graph node matching strategy considering the interactions between graph structures and signals. To solve this problem, we propose a novel graph mixup algorithm called FGWMixup, which seeks a "midpoint" of source graphs in the Fused Gromov-Wasserstein (FGW) metric space. To enhance the scalability of our method, we introduce a relaxed FGW solver that accelerates FGWMixup by improving the convergence rate from $\mathcal{O}(t^{-1})$ to $\mathcal{O}(t^{-2})$. Extensive experiments conducted on five datasets using both classic (MPNNs) and advanced (Graphormers) GNN backbones demonstrate that \mname\xspace effectively improves the generalizability and robustness of GNNs. Codes are available at https://github.com/ArthurLeoM/FGWMixup.

Fused Gromov-Wasserstein Graph Mixup for Graph-level Classifications

Abstract

Graph data augmentation has shown superiority in enhancing generalizability and robustness of GNNs in graph-level classifications. However, existing methods primarily focus on the augmentation in the graph signal space and the graph structure space independently, neglecting the joint interaction between them. In this paper, we address this limitation by formulating the problem as an optimal transport problem that aims to find an optimal inter-graph node matching strategy considering the interactions between graph structures and signals. To solve this problem, we propose a novel graph mixup algorithm called FGWMixup , which seeks a "midpoint" of source graphs in the Fused Gromov-Wasserstein (FGW) metric space. To enhance the scalability of our method, we introduce a relaxed FGW solver that accelerates FGWMixup by improving the convergence rate from O ( t -1 ) to O ( t -2 ) . Extensive experiments conducted on five datasets using both classic (MPNNs) and advanced (Graphormers) GNN backbones demonstrate that FGWMixup effectively improves the generalizability and robustness of GNNs. Codes are available at https://github.com/ArthurLeoM/FGWMixup .

1 Introduction

In recent years, Graph Neural Networks (GNNs) [1, 2] have demonstrated promising capabilities in graph-level classifications, including molecular property prediction [3, 4], social network classification [5], healthcare prediction [6, 7], etc. Nevertheless, similar to other successfully deployed deep neural networks, GNNs also suffer from data insufficiency and perturbation, requiring the application of regularization techniques to improve generalizability and robustness [8]. Data augmentation is widely adopted for regularization in deep learning. It involves creating new training data by applying various semantic-invariant transformations to the original data, such as cropping or rotating images in computer vision [9], randomly inserting and rephrasing words in natural language [10, 11], etc. The augmented data fortify deep neural networks (DNNs) against potential noise and outliers underlying insufficient samples, enabling DNNs to learn more robust and representative features.

Data augmentation for GNNs requires a unique design due to the distinctive properties of attributed graphs [12], such as irregular sizes, misaligned nodes, and diverse topologies, which are not encountered when dealing with data in the Euclidean spaces such as images and tabular data. Generally, the augmentation for GNNs necessitates the consideration of two intertwined yet complementary input spaces, namely the graph signal space X and the graph structure space A , which are mapped to an aligned latent space H with GNNs. The graph signal space X consists of node features. Current

GNNs rely on parameterized nonlinear transformations to process graph signals, which can be efficiently encoded and serve as crucial inputs for downstream predictions. Therefore, the augmentation of graph signals is significant for regularizing the GNN parameter space. On the other hand, the graph structure space A consists of information about graph topology. Traditional Message Passing Neural Networks (MPNNs), such as GCN and GIN, perform feature aggregation based on edge connectivity. Great efforts [13, 14, 15, 16] have been further exerted on enhancing the expressive power of GNNs [17], which carry out new GNN architectures with stronger sensitivity to graph topology compared with MPNNs. Hence, a good augmentation method should also consider the graph structure space.

Recently, huge efforts have been made to design graph data augmentation methods based on the two spaces. Mainstream research [18, 19, 20, 21, 22] considers the augmentation in the graph signal space and the graph structure space independently . For instance, ifMixup [19] conducts Euclidean mixup in the graph signal space, yet fails to preserve key topologies of the original graphs. G -Mixup [20] realizes graph structure mixup based on the estimated graphons, yet fails to assign semantically meaningful graph signals. In fact, the graph signal and structure spaces are not isolated from each other, and there are strong entangling relations between them [23]. Therefore, a joint modeling of the interaction between the two spaces is essential for conducting graph data augmentation (joint modeling problem for short).

Aiming at graph data augmentation and addressing the joint modeling problem, we design a novel graph mixup [24] method considering the interaction of the two spaces during the mixup procedure. The key idea is to solve a proper inter-graph node matching strategy in a metric space that measures the distance with respect to both graph signals and structures. We propose to compute the distance metric by solving an optimal transport (OT) problem [25]. The OT problem solves the optimal coupling between nodes across graphs in the fused Gromov-Wasserstein metric space [26], wherein the distance between points takes the interaction between the graph signals and structures into account. Specifically, following [26], graphs can be modeled as probability distributions embedded in the product metric space X × A . Our objective is to solve an augmented graph that minimizes the weighted sum of transportation distances between the distributions of the source graphs and the objective graph in this metric space. Developing from the Gromov-Wasserstein metric [27], Fused Gromov-Wasserstein (FGW) distance [26] has been designed to calculate the transportation distance between two unregistered probability distributions defined on different product metric spaces comprising two components, such as graph signals and structures, which defines a proper distance metric for attributed graphs. In short words, the solved graph is the augmented mixup graph in a space considering the interaction between the graph signals and structures.

However, trivially adopting FGW distance solvers [28, 26] is not scalable to large graph datasets due to a heavy computation burden. The computational bottleneck of FGW-based solvers is the nested triple-loop optimization [29], mainly due to the polytope constraint for the coupling matrix in the FGW solver. Inspired by [30], we disentangle the polytope constraint into two simplex constraints for rows and columns respectively, thence executing mirror descent with projections on the two simplexes in an alternating manner to approximate the original constraint. We prove that with a bounded gap with the ideal optimum, we may optimize the entire algorithm into a double-loop structure at convergence rate O ( t -2 ) , improving the O ( t -1 ) rate of traditional FGW solvers.

In summary, we highlight the contributions of this paper: We address the challenge of enhancing the generalizability and robustness of GNNs by proposing a novel graph data augmentation method that models the interaction between the graph signal and structure spaces. We formulate our objective as an OT problem and propose a novel graph mixup algorithm dubbed FGWMixup that seeks a "midpoint" of two graphs defined in the graph structure-signal product metric space. We employ FGW as the distance metric and speed up FGWMixup by relaxing the polytope constraint into disentangled simplex constraints, reducing the complexity from nested triple loops to double loops and meanwhile improve the convergence rate of FGWMixup . Extensive experiments are conducted on five datasets and four classic (MPNNs) and advanced (Graphormers) backbones. The results demonstrate that our method substantially improves the performance of GNNs in terms of their generalizability and robustness.

2 Methodology

In this section, we formally introduce the proposed graph data augmentation method dubbed FGWMixup . We first introduce Fused Gromov-Wasserstein (FGW) distance that presents a distance metric between graphs considering the interaction of graph signal and structure spaces.

Then we propose our optimization objective of graph mixup based on the FGW metric and the algorithmic solutions. Finally, we present our acceleration strategy. In the following we denote ∆ n := { µ ∈ R n + | ∑ i µ i = 1 } as the probability simplex with n -bins, and S n ( A ) as the set of symmetric matrices of size n taking values in A ⊂ R .

2.1 Fused Gromov-Wasserstein Distance

In OT problems, an undirected attributed graph G with n nodes is defined as a tuple ( µ , X , A ) . µ ∈ ∆ n denotes the probability measure of nodes within the graph, which can be modeled as the relative importance weights of graph nodes. Common choices of µ are uniform distributions [31] ( µ = 1 n /n ) and degree distributions [32] ( µ = [deg( v i )] i / ( ∑ i deg( v i )) ) . X = ( x (1) , · · · , x ( n ) ) ⊤ ∈ R n × d denotes the node feature matrix with d -dimensional feature on each node. A ∈ S n ( R ) denotes a matrix that encodes structural relationships between nodes, which can be selected from adjacency matrix, shortest path distance matrix or other distance metrics based on the graph topologies. Given two graphs G 1 = ( µ 1 , X 1 , A 1 ) and G 2 = ( µ 2 , X 2 , A 2 ) of sizes n 1 and n 2 respectively, Fused Gromov-Wasserstein distance can be defined as follows:

where Π( µ 1 , µ 2 ) := { π ∈ R n 1 × n 2 + | π 1 n 2 = µ 1 , π ⊤ 1 n 1 = µ 2 } is the set of all valid couplings between node distributions µ 1 and µ 2 , d ( · , · ) is the distance metric in the feature space, and α ∈ [0 , 1] is the weight that trades off between the Gromov-Wasserstein cost on graph structure and Wasserstein cost on graph signals. The FGW distance is formulated as an optimization problem that aims to determine the optimal coupling between nodes in a fused metric space considering the interaction of graph structure and node features. The optimal coupling matrix π ∗ serves as a soft node matching strategy that tends to match two pairs of nodes from different graphs that have both similar graph structural properties (such as k -hop connectivity, defined by A ) and similar node features (such as Euclidean similarity, defined by d ( · , · ) ). In fact, FGW also defines a strict distance metric on the graph space (∆ , X , A ) when q = 1 and A s are distance matrices, and defines a semi-metric whose triangle inequality is relaxed by 2 q -1 for q > 1 . In practice, we usually choose q = 2 and Euclidean distance for d ( · , · ) to calculate FGW distance.

Solving FGW Distance Solving FGW distance is a non-convex optimization problem, whose non-convexity comes from the quadratic term of the GW distance. There has been a line of research contributing to solving this problem. [26] proposes to apply the conditional gradient (CG) algorithm to solve this problem, and [28] presents an entropic approximation of the problem and solves the optimization through Mirror Descent (MD) algorithm according to KL-divergence.

2.2 Solving Graph Mixup in the FGW Metric Space

Building upon the FGW distance and its properties, we propose a novel graph mixup method dubbed FGWMixup . Formally, our objective is to solve a synthetic graph ˜ G = (˜ µ , ˜ X , ˜ A ) of size ˜ n that minimizes the weighted sum of FGW distances between ˜ G and two source graphs G 1 = ( µ 1 , X 1 , A 1 ) and G 2 = ( µ 2 , X 2 , A 2 ) respectively. The optimization objective is as follows:

where λ ∈ [0 , 1] is a scalar mixing ratio, usually sampled from a Beta( k, k ) distribution with hyperparameter k . This optimization problem formulates the graph mixup problem as an OT problem that aims to find the optimal graph ˜ G at the "midpoint" of G 1 and G 2 in terms of both graph signals and structures. When the optimum ˜ G ∗ is reached, the solutions of FGW distances between ˜ G ∗ and the source graphs G 1 , G 2 provide the node matching strategies that minimize the costs of aligning graph structures and graph signals. The label of ˜ G is then assigned as y ˜ G = λy G 1 +(1 -λ ) y G 2 .

In practice, we usually fix the node probability distribution of ˜ G with a uniform distribution (i.e., ˜ µ = 1 ˜ n / ˜ n ) [26]. Then, Eq.2 can be regarded as a nested bi-level optimization problem, which composes the upper-level optimization w.r.t. the node feature matrix ˜ X and the graph structure ˜ A ,

and the lower-level optimization w.r.t. the couplings between current ˜ G and G 1 , G 2 denotes as π 1 , π 2 . Inspired by [28, 26, 33], we propose to solve Eq.2 using a Block Coordinate Descent algorithm, which iteratively minimizes the lower-level and the upper-level with a nested loop framework. The algorithm is presented in Alg.1. The inner loop solves the lower-level problem (i.e., FGW distance and the optimal couplings π ( k ) 1 , π ( k ) 2 ) based on ˜ X ( k ) , ˜ A ( k ) at the k -th outer loop iteration, which is non-convex and requires optimization algorithms introduced in Section 2.1. The outer loop solves the upper-level problem (i.e., minimizing the weighted sum of FGW distances), which is a convex quadratic optimization problem w.r.t. ˜ X ( k ) and ˜ A ( k ) with exact analytical solution (i.e., Line 7,8).

2.3 Accelerating FGWMixup

Algorithm 1 provides a practical solution to optimize Eq.2, whereas the computation complexity is relatively high. Specifically, Alg.1 adopts a nested double-loop framework, where the inner loop is the FGW solver that optimizes the couplings π , and the outer loop updates the optimal feature matrix ˜ X and graph structure ˜ A accordingly. However, the most common FGW solvers, such as MD and CG, require another nested double-loop algorithm. This algorithm invokes (Bregman) projected gradient descent type methods to address the non-convex optimization of FGW, which involves taking a gradient step in the outer loop and projecting to the polytope-constrained feasible set w.r.t. the couplings in the inner loop (e.g., using Sinkhorn [30] or Dykstra [34] iterations). Consequently, this makes the entire mixup algorithm a triple-loop framework, resulting in a heavy computation burden.

Therefore, we attempt to design a method that efficiently accelerates the algorithm. There lie two efficiency bottlenecks of Alg.1: 1) the polytope constraints of couplings makes the FGW solver a nested double-loop algorithm, 2) the strict projection of couplings to the feasible sets probably modifies the gradient step to another direction that deviates from the original navigation of the gradient, possibly leading to a slower convergence rate. In order to alleviate the problems, we are motivated to slightly relax the feasibility constraints of couplings to speed up the algorithm. Inspired by [29, 30], we do not strictly project π to fit the polytope constraint Π( µ i , µ j ) := { π ∈ R n 1 × n 2 + | π 1 n 2 = µ 1 , π ⊤ 1 n 1 = µ 2 } after taking each gradient step. Instead, we relax the constraint into two simplex constraints of rows and columns respectively (i.e., Π 1 := { π ∈ R n 1 × n 2 + | π 1 n 2 = µ 1 } , Π 2 := { π ∈ R n 1 × n 2 + | π ⊤ 1 n 1 = µ 2 } ), and project π to the relaxed constraints Π 1 and Π 2 in an alternating fashion. The accelerated algorithm is presented in Alg.2, dubbed FGWMixup ∗ .

Alg.2 mainly substitutes Line 5 and Line 6 of Alg.1 with a single-loop FGW distance solver (Line 7-12 in Alg.2) that relaxes the joint polytope constraints of the couplings. Specifically, we remove the constant term in FGW distance, and denote the equivalent metric as FGW( G 1 , G 2 ) . The optimization objective of FGW( G 1 , G 2 ) can be regarded as a function of π , and we name it the FGW function f ( π ) (See Appendix A.1 for details). Then we employ entropic regularization on f ( π ) and select Mirror Descent as the core algorithm for the FGW solver. With the negative entropy ϕ ( x ) = ∑ i x i log x i as the Bregman projection, the MD update takes the form of:

where γ is the step size. The subgradient of FGW w.r.t. the coupling π can be calculated as:

where D = ( d ( X 1 [ i ] , X 2 [ j ])) n 1 × n 2 is the distance matrix of node features between two graphs. The detailed derivation can be found in Appendix A.2.

Our relaxation is conducted in Line 9 and 11, where the couplings are projected to the row and column simplexes alternately instead of directly to the strict polytope. Although this relaxation may sacrifice some feasibility due to the relaxed projection, the efficiency of the algorithm has been greatly promoted. On the one hand, noted that Π 1 and Π 2 are both simplexes, the Bregman projection w.r.t. the negative entropy of a simplex can be extremely efficiently conducted without invoking extra iterative optimizations (i.e., Line 9, 11), which simplifies the FGW solver from a nested double-loop framework to a single-loop one. On the other hand, the relaxed constraint may also increase the convergence efficiency due to a closer optimization path to the unconstrained gradient descent.

Wealso provide some theoretical results to justify our algorithm. Proposition 1 presents a convergence rate analysis on our algorithm. Taking 1 /γ as the entropic regularization coefficient, our FGW solver can be formulated as Sinkhorn iterations, whose convergence rate can be optimized from O ( t -1 ) to O ( t -2 ) by conducting marginal constraint relaxation. Proposition 2 presents a controlled gap between the optima given by the relaxed single-loop FGW solver and the strict FGW solver.

Proposition 1. Let ( X , µ ) , ( Y , ν ) be Polish probability spaces, and π ∈ Π( µ, ν ) the probability measure on X × Y with marginals µ, ν . Let π t be the Sinkhorn iterations π 2 t = arg min Π( ∗ ,ν ) H ( ·| π 2 t -1 ) , π 2 t +1 = arg min Π( µ, ∗ ) H ( ·| π 2 t ) , where H ( p | q ) = -∑ i p i log q i p i is the Kullback-Leibler divergence, and Π( ∗ , ν ) is the set of measures with second marginal ν and arbitrary first marginal ( Π( µ, ∗ ) is defined analogously). Let π ∗ be the unique optimal solution. We have the convergence rate as follows:

Proposition 1 implies the faster convergence of marginal constraints than the strict joint constraint. This entails that with t Sinkhorn iterations of solving FGW, the solution of the relaxed FGW solver moves further than the strict one. This will benefit the convergence rate of the whole algorithm with a larger step size of π ( k ) i in each outer iteration.

Proposition 2. Let C 1 , C 2 be two convex sets, and f ( π ) denote the FGW function w.r.t. π . We denote X as the critical point set of strict FGW that solves min π f ( π ) + I π ∈ C 1 + I π ∈ C 2 , defined by: X = { π ∈ C 1 ∩ C 2 : 0 ∈ ∇ f ( π ) + N C 1 ( π ) + N C 2 ( π ) } , and N C ( π ) is the normal cone to C

at π . The fix-point set X rel of the relaxed FGW solving min π,ω f ( π ) + f ( ω ) + h ( π ) + h ( ω ) is defined by: X rel = { π ∈ C 1 , ω ∈ C 2 : 0 ∈ ∇ f ( π ) + ρ ( ∇ h ( ω ) - ∇ h ( π )) + N C 1 ( π ) , and 0 ∈ ∇ f ( ω ) + ρ ( ∇ h ( π ) -∇ h ( ω )) + N C 2 ( ω ) } where h ( · ) is the Bregman projection function. Then, the gap between X rel and X satisfies:

This bounded gap ensures the correctness of FGWMixup ∗ that as long as we select a step size γ = 1 /ρ that is small enough, the scale of the upper bound τ/ρ will be sufficiently small to ensure the convergence to the ideal optimum of our algorithm. The detailed proofs of Propositions 1 and 2 are presented in Appendix B.

3 Experiments

3.1 Experimental Settings

Datasets We evaluate our methods with five widely-used graph classification tasks from the graph benchmark dataset collection TUDataset [35]: NCI1 and NCI109 [36, 37] for small molecule classification, PROTEINS [38] for protein categorization, and IMDB-B and IMDB-M [5] for social networks classification. Noted that there are no node features in IMDB-B and IMDB-M datasets, we augment the two datasets with node degree features as in [2, 19, 20]. Detailed statistics on these datasets are reported in Appendix D.1.

Backbones Most existing works select traditional MPNNs such as GCN and GIN as the backbone. However, traditional MPNNs exhibit limited expressive power and sensitivity to graph structures (upper bounded by 1-Weisfeiler-Lehman (1-WL) test [17, 37]), while there exist various adavanced GNN architectures [13, 14, 16] with stronger expressive power (upper bound promoted to 3-WL test). Moreover, the use of a global pooling layer (e.g., mean pooling) in graph classification models may further deteriorate their perception of intricate graph topologies. These challenges may undermine the reliability of the conclusions regarding the effectiveness of graph structure augmentation. Therefore, we attempt to alleviate the problems from two aspects. 1) We modify the READOUT approach of GIN and GCN to save as much structural information as we can. Following [39], we apply a virtual node that connects with all other nodes and use the final latent representation of the virtual node to conduct READOUT. The two modified backbones are dubbed vGIN and vGCN. 2) We select two Transformer-based GNNs with stronger expressive power as the backbones, namely Graphormer [39] and Graphormer-GD [16]. They are proven to be more sensitive to graph structures, and GraphormerGD is even capable of perceiving cut edges and cut vertices in graphs. Detailed information about the selected backbones is introduced in Appendix D.2.

Comparison Baselines We select the following data augmentation methods as the comparison baselines. DropEdge [40] randomly removes a certain ratio of edges from the input graphs. DropNode [41] randomly removes a certain portion of nodes as well as the edge connections. M-Mixup [42] conducts Euclidean mixup in the latent spaces, which interpolates the graph representations after the READOUT function. ifMixup [19] applies an arbitrary node matching strategy to conduct mixup on graph node signals, without preserving key topologies of original graphs. G -Mixup [20] conducts Euclidean addition on the estimated graphons of different classes of graphs to conduct class-level graph mixup. We also present the performances of the vanilla backbones, as well as our proposed method w/ and w/o acceleration, denoted as FGWMixup ∗ and FGWMixup respectively.

Experimental Setups For a fair comparison, we employ the same set of hyperparameter configurations for all data augmentation methods in each backbone architecture. For all datasets, We randomly hold out a test set comprising 10% of the entire dataset and employ 10-fold cross-validation on the remaining data. We report the average and standard deviation of the accuracy on the test set over the best models selected from the 10 folds. This setting is more realistic than reporting results from validation sets in a simple 10-fold CV and allows a better understanding of the generalizability [43]. We implement our backbones and mixup algorithms based on Deep Graph Library (DGL) [44] and Python Optimal Transport (POT) [45] open-source libraries. More experimental and implementation details are introduced in Appendix D.3 and D.4.

PROTEINS

NCI1

NCI109

IMDB-B

IMDB-M

Methods

vGIN

vGCN

vGIN

vGCN

vGIN

vGCN

vGIN

vGCN

vGIN

vGCN

MPNNs

vanilla

74.93(3.02)

74.75(2.60)

76.98(1.87)

76.91(1.80)

75.70(1.85)

75.89(1.35)

71.30(4.96)

72.30(4.34)

49.00(2.64)

49.47(3.76)

DropEdge

73.59(2.50)

74.48(4.18)

76.47(2.85)

76.16(2.04)

75.38(2.05)

75.77(1.55)

73.30(3.85)

73.30(3.29)

49.47(2.66)

49.40(3.52)

DropNode

74.48(2.91)

75.11(3.00)

76.89(1.25)

77.42(1.71)

73.98(2.16)

75.45(1.90)

71.50(3.23)

73.20(5.58)

49.80(3.29)

50.00(3.41)

M-Mixup

74.40(3.00)

75.65(4.51)

76.45(3.39)

77.76(2.75)

75.41(2.78)

75.79(1.85)

72.20(4.83)

72.80(4.45)

49.13(3.25)

49.47(2.56)

ifMixup

74.76(3.71)

74.04(2.27)

76.16(1.78)

77.37(2.56)

76.13(1.87)

76.74(1.56)

72.50(3.98)

72.40(5.14)

49.07(3.16)

49.73(4.67)

G

-Mixup

74.84(2.99)

74.57(2.88)

76.42(1.79)

77.79(1.88)

75.55(2.32)

76.38(1.79)

72.40(4.82)

72.20(6.45)

49.47(4.73)

49.60(3.90)

FGWMixup

75.02(3.86)

76.01(3.19)

78.32(2.65)

78.37(2.40)

76.40(1.65)

76.79(1.81)

73.00(3.69)

73.40(5.12)

49.80(2.63)

50.80(4.06)

FGWMixup

∗

75.20(3.30)

75.20(3.03)

78.47(1.74)

76.64(2.60)

76.52(1.59)

73.50(4.54)

74.00(2.90)

49.20(3.38)

50.47(5.44)

Methods

Graphormer GraphormerGD Graphormer GraphormerGD Graphormer GraphormerGD Graphormer GraphormerGD Graphormer GraphormerGD

Graphormers

vanilla

75.47(3.16)

76.01(2.02)

61.56(3.70)

77.49(2.01)

65.54(3.04)

74.99(1.23)

70.40(5.00)

71.50(4.20)

48.87(4.10)

47.47(2.98)

DropEdge

75.20(4.02)

75.12(3.22)

63.07(3.21)

74.94(2.44)

66.73(3.50)

74.73(3.22)

71.10(5.65)

72.30(3.93)

49.60(4.09)

46.67(3.85)

DropNode

75.20(2.13)

76.28(3.49)

64.96(2.18)

76.20(1.95)

63.73(3.46)

74.78(2.07)

71.60(5.18)

71.30(5.18)

48.47(4.08)

47.67(2.83)

M-Mixup

75.11(3.78)

74.39(3.83)

62.31(3.48)

75.47(1.45)

66.54(2.70)

74.61(1.86)

71.10(4.83)

70.50(4.70)

49.67(4.25)

48.00(3.85)

G

-Mixup

75.74(3.12)

74.85(3.52)

63.07(4.40)

76.06(3.12)

65.03(2.98)

74.90(2.04)

72.10(6.38)

71.10(5.01)

46.93(5.18)

46.80(4.41)

FGWMixup

76.82(2.35)

77.18(3.48)

66.45(2.58)

78.20(1.88)

67.36(3.21)

76.01(3.04)

72.60(5.08)

72.40(4.48)

49.73(3.80)

48.87(4.03)

FGWMixup

∗

76.19(3.20)

76.46(3.41)

64.26(3.25)

76.62(3.06)

67.46(2.82)

75.45(1.80)

71.70(4.17)

71.90(4.35)

50.27(4.26)

48.53(2.95)

3.2 Experimental Results

Main Results We compare the performance of various GNN backbones on five benchmark datasets equipped with different graph data augmentation methods and summarize the results in Table 1. As is shown in the table, FGWMixup and FGWMixup ∗ consistently outperform all other SOTA baseline methods, which obtain 13 and 7 best performances out of all the 20 settings respectively. The superiority is mainly attributed to the adoption of the FGW metric. Existing works hardly consider the node matching problem to align two unregistered graphs embedded in the signal-structure fused metric spaces, whereas the FGW metric searches the optimum from all possible couplings and conducts semantic-invariant augmentation guided by the optimal node matching strategy. Remarkably, despite introducing some infeasibility for acceleration, FGWMixup ∗ maintains consistent performance due to the theoretically controlled gap with the ideal optimum, as demonstrated in Prop. 1. Moreover, FGWMixup and FGWMixup ∗ both effectively improve the performance and generalizability of various GNNs. Specifically, our methods achieve an average relative improvement of 1.79% on MPNN backbones and 2.67% on Graphormer-based backbones when predicting held-out unseen samples. Interestingly, most SOTA graph data augmentation methods fail to bring notable improvements and even degrade the performance of Graphormers, which exhibit stronger expressive power and conduct more comprehensive interactions between graph signal and structure spaces. For instance, none of the baseline methods improve GraphormerGD on NCI109 and NCI1 datasets. However, our methods show even larger improvements on Graphormers compared to MPNNs, as we explicitly consider the interactions between the graph signal and structure spaces during the mixup process. In particular, the relative improvements of FGWMixup reach up to 7.94% on Graphormer and 2.95% on GraphormerGD. All the results above validate the effectiveness of our methods. Furthermore, we also conduct experiments on large-scale OGB [46] datasets, and the results are provided in Appendix E.5.

Robustness against Label Corruptions In this subsection, we evaluate the robustness of our methods against noisy labels. Practically, we introduce random label corruptions (i.e., switching to another random label) with ratios of 20%, 40%, and 60% on the IMDB-B and NCI1 datasets. We employ vGCN as the backbone model and summarize the results in Table 2. The results evidently demonstrate that the mixup-series methods consistently outperform the in-sample augmentation method (DropEdge) by a significant margin. This improvement can be attributed to the soft-labeling strategy employed by mixup, which reduces the model's sensitivity to individual mislabeled instances and encourages the model to conduct more informed predictions based on the overall distribution of the blended samples. Notably, among all the mixup methods, FGWMixup and FGWMixup ∗ exhibit stable and consistently superior performance under noisy label conditions. In conclusion, our methods effectively enhance the robustness of GNNs against label corruptions.

Analysis on Mixup Efficiency We run FGWMixup and FGWMixup ∗ individually with identical experimental settings (including stopping criteria and hyper-parameters such as mixup ratio, etc.) on the same computing device and investigate the run-time computational efficiencies. Table 3 illustrates

the average time spent on the mixup procedures of FGWMixup and FGWMixup ∗ per fold. We can observe that FGWMixup ∗ decreases the mixup time cost by a distinct margin, providing at least 2.03 × and up to 3.46 × of efficiency promotion. The results are consistent with our theoretical analysis of the convergence rate improvements, as shown in Proposition 1. More detailed efficiency statistics and discussions are introduced in Appendix E.3.

Infeasibility Analysis on the Single-loop FGW Solver in FGWMixup ∗ We conduct an experiment to analyze the infeasibility of our single-loop FGW solver compared with the strict CG solver. Practically, we randomly select 1,000 pairs of graphs from PROTEINS dataset and apply the two solvers to calculate the FGW distance between each pair of graphs. The distances of the i -th pair of graphs calculated by the strict solver and the relaxed solver are denoted as d i and d ∗ i , respectively. We report the following metrics for comparison:

MAE : Mean absolute error of FGW distance, i.e., 1 N ∑ | d i -d ∗ i | .

mean-FGW : Mean FGW distance given by the strict CG solver, i.e., 1 N ∑ d i .

MAPE : Mean absolute percentage error of FGW distance, i.e., 1 N ∑ | d i -d ∗ i | d i .

mean-FGW* : Mean FGW distance given by the single-loop solver, i.e., 1 N ∑ d ∗ i .

T-diff : L2-norm of the difference between two transportation plan matrices (divided by the size of the matrix for normalization).

The results are shown in Table 4. We can observe that the MAPE is only 0.0748, which means the FGW distance estimated by the single-loop relaxed solver is only 7.48% different from the strict CG solver. Moreover, the absolute error is around 0.01, which is quite small compared with the absolute value of FGW distances (~0.21). We can also find that the L2-norm of the difference between two transportation plan matrices is only 0.0006, which means two solvers give quite similar transportation plans. All the results imply that the single-loop solver will not produce huge infeasibility or make the estimation of FGW distance inaccurate.

3.3 Further Analyses

Effects of the Trade-off Coefficient α Weprovide sensitivity analysis w.r.t. the trade-off coefficient α of our proposed mixup methods valued from {0.05, 0.5, 0.95, 1.0} on two backbones and two

FGWMixup

FGWMixup

*

76.0

76.5

77.0

77.5

78.0

78.5

79.0

Accuracy (%)

/glyph1197CI1

76.0

76.5

77.0

77.5

78.0

78.5

79.0

FGWMixup

FGWMixup

*

74.0

74.5

75.0

75.5

76.0

Accuracy (%)

PROTEI/glyph1197S

74.0

74.5

75.0

75.5

76.0

Adaptive

Median

2

×

Med.

0.5

×

Med.

Vanilla

datasets. Note that α controls the weights of the node feature alignment and graph structure alignment costs, and α = 1 . 0 falls back to the case of GW metric where node features are not incorporated. From the results shown in Table 5, we can observe that: 1) when FGW falls back to GW ( α =1), where node features are no longer taken into account, the performance will significantly decay (generally the worst among all investigated α values). This demonstrates the importance of solving the joint modeling problem in graph mixup tasks. 2) α =0.95 is the best setting in most cases. This empirically implies that it is better to conduct more structural alignment in graph mixup. In practice, we set α to 0.95 for all of our reported results.

Effects of Mixup Graph Sizes We investigate the performance of FGWMixup and FGWMixup ∗ with various mixup graph sizes (node numbers), including adaptive graph sizes (i.e., weighted average size of the mixup source graphs, ˜ n = λn 1 +(1 -λ ) n 2 , which is selected as our method) and fixed graph sizes (i.e., the median size of all training graphs, 0.5 × and 2 × the median). The ablation studies are composed on NCI1 and PROTEINS datasets using vGCN backbone as motivating examples, and the results are illustrated in Fig.1. We can observe that the best performance is steadily obtained when selecting graph sizes with an adaptive strategy, whereas the fixed graph sizes lead to unstable results. Specifically, selecting the median or larger size may occasionally yield comparable performances, yet selecting the smaller size can result in an overall performance decay of over 1%. This phenomenon is associated with the graph size generalization problem [47, 48] that describes the performance degradation of GNNs caused by the graph size distributional shift between training and testing data. The fixed strategy may aggravate this problem, particularly for small graphs that struggle to generalize to larger ones. In contrast, the adaptive strategy can potentially combat this distributional shift by increasing the data diversity and reach better test time performance.

Effects of GNN Depths To validate the improvement of our methods across various model depths, we evaluate the performance of FGWMixup and FGWMixup ∗ using vGCNs equipped with different numbers (3-8) of layers. We experiment on NCI1 and PROTEINS datasets, and the results are illustrated in Fig.2. We can observe that our methods consistently improve the performance on NCI1 under all GCN depth settings by a significant margin. The same conclusion is also true for most cases on PROTEINS except for the 7-layer vGCN. These results indicate a universal improvement of our methods on GNNs with various depths.

Other Discussions More further analyses are introduced in the Appendix, including qualitative analyses of our mixup results (see Appendix E.1), further discussions on G -Mixup (see Appendix E.2), and sensitivity analysis of the hyperparameter k in Beta distribution where mixup weights are sampled from (see Appendix E.4).

3

4

5

6

7

8

GCN Depth

76.5

77.0

77.5

78.0

78.5

79.0

Accuracy (%)

/glyph1197CI1

3

4

5

6

7

8

GCN Depth

72.5

73.0

73.5

74.0

74.5

75.0

75.5

76.0

Accuracy (%)

PROTEI/glyph1197S

vanilla

FGWMixup

FGWMixup

*

4 Related Works

Graph Data Augmentation There are currently two mainstream perspectives of graph data augmentation for graph-level classifications. A line of research concentrates on graph signal augmentation [41, 18, 19]. Node Atrribute Masking [41] assigns random node features for a certain ratio of graph nodes. IfMixup [19] conducts Euclidean mixup on node features from different graphs with arbitrary node alignment strategy, whereas it damages the critical topologies (e.g., rings, bipartibility) of the source graphs. Another line of work focuses on graph structure augmentation [49, 20, 21, 22, 50, 41, 51]. Approaches like Subgraph[49, 41], DropEdge[40] and GAug[50] conduct removals or additions of graph nodes or edges to generate new graph structures. Graph Transplant[21] and Submix[22] realize CutMix-like augmentations on graphs, which cut off a subgraph from the original graph and replace it with another. G -Mixup [20] estimates a graph structure generator (i.e., graphon) for each class of graphs and conducts Euclidean addition on graphons to realize structural mixup. However, as GNNs can capture the complex interaction between the two entangled yet complementary input spaces, it is essential to model this interaction for a comprehensive graph data augmentation. Regrettably, current works have devoted little attention to this interaction.

Optimal Transport on Graphs Building upon traditional Optimal Transport (OT) [25] methods (e.g., Wasserstein distance[52]), graph OT allows defining a very general distance metric between the structured/relational data points embedded in different metric spaces, where the data points are modeled as probability distributions. It proceeds by solving a coupling between the distributions that minimizes a specific cost. The solution of graph OT hugely relies on the (Fused) Gromov-Wasserstein (GW) [27, 26] distance metric. Further works employ the GW couplings for solving tasks such as graph node matching and partitioning [53, 32, 54], and utilize GW distance as a common metric in graph metric learning frameworks [55, 31, 56]. Due to the high complexity of the GW-series algorithm, another line of research [28, 57, 58, 29] concentrates on boosting the computational efficiency. In our work, we formulate graph mixup as an FGW-based OT problem that solves the optimal node matching strategy minimizing the alignment costs of graph structures and signals. Meanwhile, we attempt to accelerate the mixup procedure with a faster convergence rate.

5 Conclusion and Limitation

In this work, we introduce a novel graph data augmentation method for graph-level classifications dubbed FGWMixup . FGWMixup formulates the mixup of two graphs as an optimal transport problem aiming to seek a "midpoint" of two graphs embedded in the graph signal-structure fused metric space. Weemploy the FGW distance metric to solve the problem and further propose an accelerated algorithm that improves the convergence rate from O ( t -1 ) to O ( t -2 ) for better scalability of our method. Comprehensive experiments demonstrate the effectiveness of FGWMixup in terms of enhancing the performance, generalizability, and robustness of GNNs, and also validate the efficiency and correctness of our acceleration strategy.

Despite the promising results obtained in our work, it is important to acknowledge its limitations. Our focus has primarily been on graph data augmentation for graph-level classifications. However, it still remains a challenging question to better exploit the interactions between the graph signal and structure spaces for data augmentation in other graph prediction tasks, such as link prediction and node classification. Moreover, we experiment with four classic (MPNNs) and advanced (Graphormers) GNNs, while there remain other frameworks that could be taken into account. We expect to carry out a graph classification benchmark with more comprehensive GNN frameworks in our future work.

References

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations .

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning , pages 1263-1272. PMLR, 2017.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical Science , 9(2):513-530, 2018.

Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1365-1374, 2015.

Xinyu Ma, Yasha Wang, Xu Chu, Liantao Ma, Wen Tang, Junfeng Zhao, Ye Yuan, and Guoren Wang. Patient health representation learning via correlational sparse prior of medical features. IEEE Transactions on Knowledge and Data Engineering , 2022.

Yongxin Xu, Xu Chu, Kai Yang, Zhiyuan Wang, Peinie Zou, Hongxin Ding, Junfeng Zhao, Yasha Wang, and Bing Xie. Seqcare: Sequential training with external medical knowledge graph for diagnosis prediction in healthcare data. In Proceedings of the ACM Web Conference 2023 , pages 2819-2830, 2023.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107115, 2021.

Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. Image data augmentation for deep learning: A survey. arXiv preprint arXiv:2204.08610 , 2022.

Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6382-6388, 2019.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems , 28, 2015.

Zhiqiang Xu, Yiping Ke, Yi Wang, Hong Cheng, and James Cheng. A model-based approach to attributed graph clustering. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data , pages 505-516, 2012.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in Neural Information Processing Systems , 32, 2019.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. arXiv preprint arXiv:2110.03753 , 2021.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. arXiv preprint arXiv:2110.02910 , 2021.

Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of gnns via graph biconnectivity. arXiv preprint arXiv:2301.09505 , 2023.

Pan Li and Jure Leskovec. The expressive power of graph neural networks. Graph Neural Networks: Foundations, Frontiers, and Applications , pages 63-98, 2022.

Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, Juncheng Liu, and Bryan Hooi. Nodeaug: Semi-supervised node classification with data augmentation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 207-217, 2020.

Hongyu Guo and Yongyi Mao. ifmixup: Towards intrusion-free graph mixup for graph classification. arXiv e-prints , pages arXiv-2110, 2021.

Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning , pages 8230-8248. PMLR, 2022.

Joonhyung Park, Hajin Shim, and Eunho Yang. Graph transplant: Node saliency-guided graph mixup with local structure preservation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7966-7974, 2022.

Jaemin Yoo, Sooyeon Shim, and U Kang. Model-agnostic augmentation for accurate graph classification. In Proceedings of the ACM Web Conference 2022 , pages 1281-1291, 2022.

Antonio Ortega, Pascal Frossard, Jelena Kovaˇ cevi´ c, José MF Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE , 106(5):808-828, 2018.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

Vayer Titouan, Nicolas Courty, Romain Tavenard, and Rémi Flamary. Optimal transport for structured data with application on graphs. In International Conference on Machine Learning , pages 6275-6284. PMLR, 2019.

Facundo Mémoli. Gromov-wasserstein distances and the metric approach to object matching. Foundations of Computational Mathematics , 11:417-487, 2011.

Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-wasserstein averaging of kernel and distance matrices. In International Conference on Machine Learning , pages 2664-2672. PMLR, 2016.

Jiajin Li, Jianheng Tang, Lemin Kong, Huikang Liu, Jia Li, Anthony Man-Cho So, and Jose Blanchet. A convergent single-loop algorithm for relaxation of gromov-wasserstein in graph data. arXiv preprint arXiv:2303.06595 , 2023.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems , 26, 2013.

Cédric Vincent-Cuaz, Rémi Flamary, Marco Corneli, Titouan Vayer, and Nicolas Courty. Template based graph neural network with optimal transport distances. arXiv preprint arXiv:2205.15733 , 2022.

Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wasserstein learning for graph matching and node embedding. In International Conference on Machine Learning , pages 6932-6941. PMLR, 2019.

Rémi Flamary, Nicolas Courty, Devis Tuia, and Alain Rakotomamonjy. Optimal transport with laplacian regularization: Applications to domain adaptation and shape matching. In NIPS Workshop on Optimal Transport and Machine Learning OTML , 2014.

Meyer Scetbon, Marco Cuturi, and Gabriel Peyré. Low-rank sinkhorn factorization. In International Conference on Machine Learning , pages 9344-9354. PMLR, 2021.

Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663 , 2020.

Nikil Wale and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classification. In Sixth International Conference on Data Mining (ICDM'06) , pages 678-689. IEEE, 2006.

Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12(9), 2011.

Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics , 21(suppl_1):i47-i56, 2005.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems , 34:28877-28888, 2021.

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXiv:1907.10903 , 2019.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems , 33:5812-5823, 2020.

Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. Mixup for node and graph classification. In Proceedings of the Web Conference 2021 , pages 3663-3674, 2021.

Yoshua Bengio and Yves Grandvalet. No unbiased estimator of the variance of k-fold crossvalidation. Advances in Neural Information Processing Systems , 16, 2003.

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315 , 2019.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research , 22(78):1-8, 2021.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Davide Buffelli, Pietro Lió, and Fabio Vandin. Sizeshiftreg: a regularization method for improving size-generalization in graph neural networks. In Advances in Neural Information Processing Systems , 2022.

Beatrice Bevilacqua, Yangze Zhou, and Bruno Ribeiro. Size-invariant graph representations for graph classification extrapolations. In International Conference on Machine Learning , pages 837-851. PMLR, 2021.

Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. Graphcrop: Subgraph cropping for graph classification. arXiv preprint arXiv:2009.10564 , 2020.

Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11015-11023, 2021.

Hyeonjin Park, Seunghun Lee, Sihyeon Kim, Jinyoung Park, Jisu Jeong, Kyung-Min Kim, Jung-Woo Ha, and Hyunwoo J Kim. Metropolis-hastings data augmentation for graph neural networks. Advances in Neural Information Processing Systems , 34:19010-19020, 2021.

Cédric Villani. Topics in Optimal Transportation . Number 58. American Mathematical Soc., 2003.

Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wasserstein learning for graph partitioning and matching. Advances in Neural Information Processing Systems , 32, 2019.

Samir Chowdhury and Tom Needham. Generalized spectral clustering via gromov-wasserstein learning. In International Conference on Artificial Intelligence and Statistics , pages 712-720. PMLR, 2021.

Cédric Vincent-Cuaz, Titouan Vayer, Rémi Flamary, Marco Corneli, and Nicolas Courty. Online graph dictionary learning. In International Conference on Machine Learning , pages 1056410574. PMLR, 2021.

Benson Chen, Gary Bécigneul, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Optimal transport graph neural networks. arXiv preprint arXiv:2006.04804 , 2020.

Cédric Vincent-Cuaz, Rémi Flamary, Marco Corneli, Titouan Vayer, and Nicolas Courty. Semi-relaxed gromov-wasserstein divergence with applications on graphs. arXiv preprint arXiv:2110.02753 , 2021.

Meyer Scetbon, Gabriel Peyré, and Marco Cuturi. Linear-time gromov wasserstein distances using low rank couplings and costs. In International Conference on Machine Learning , pages 19347-19365. PMLR, 2022.

Promit Ghosal and Marcel Nutz. On the convergence rate of sinkhorn's algorithm. arXiv preprint arXiv:2212.06000 , 2022.

Heinz H Bauschke. Projection algorithms and monotone operators. 1996.

Zhi-Quan Luo and Paul Tseng. Error bound and convergence analysis of matrix splitting algorithms for the affine variational inequality problem. SIAM Journal on Optimization , 2(1):4354, 1992.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Douglas J Klein and Milan Randi´ c. Resistance distance. Journal of Mathematical Chemistry , 12:81-95, 1993.

Hongteng Xu, Peilin Zhao, Junzhou Huang, and Dixin Luo. Learning graphon autoencoders for generative graph modeling. arXiv preprint arXiv:2105.14244 , 2021.

Nikolay Sakharnykh and Braun Hugo. Efficient maximum flow allgorithm.

A Problem Properties

A.1 FGWDistance Metric

where D = { d ( x ( i ) 1 , x ( j ) 2 ) } ij is the distance matrix of node features, and L ( A 1 , A 2 ) = {| A 1 ( i, k ) -A 2 ( j, ℓ ) |} i,j,k,ℓ is a 4-dimensional tensor that depicts the structural distances. We denote ⟨ U , V ⟩ = tr( U ⊤ V ) as the matrix scalar product. The ⊗ operator conducts tensor-matrix multiplication as: L⊗ T def. = ( ∑ k,ℓ L i,j,k,ℓ T k,ℓ ) i,j . Then according to Proposition 1 in [28], when we take the ℓ 2 -norm to calculate the structural distance, we have:

where c A 1 , A 2 = A ⊙ 2 1 µ 1 1 ⊤ n 2 + 1 n 1 µ ⊤ 2 A ⊙ 2 2 , and ⊙ 2 denotes the Hadamard square (i.e., U ⊙ 2 = U ⊙ U ). Taking Eq.9 into Eq.8, we can rewrite the minimizer as:

The items in the overbrace become a constant because the marginals of π are fixed as µ 1 and µ 2 , respectively. Therefore, we can remove the constant term and formulate an equivalent optimization objective denoted as FGW( G 1 , G 2 ) , which can be rewritten as:

We denote the above optimization objective as a function f ( π ) w.r.t. π , and we call it the FGW function in the main text.

A.2 Mirror Descent and the Relaxed FGW Solver in FGWMixup ∗

Definition 1 (Normal Cone) . Given any set C and point x ∈ C , we can define normal cone as:

There are some properties of normal cone. The normal cone is always convex, and Proj C ( x + z ) = x, ∀ x ∈ C , z ∈ N C ( x ) always holds.

Proposition 3 (The Update of Mirror Descent) . The update of Mirror Descent takes the form of x ( k +1) = arg min x ∈X D ϕ ( x, y ( k +1) )) , where ϕ ( · ) is a Bregman Projection, and ∇ ϕ ( y ( k +1) ) = ∇ ϕ ( ω ( k ) ) -∇ f ( π ) /ρ .

Proof. For a constrained optimization problem min x ∈X f ( x ) , X ⊂ D , Mirror Descent iteratively updates x with x ( k +1) = argmin x ∈X ∇ f ( x ( k ) ) ⊤ x + ρD ϕ ( x, x ( k ) ) with Bregman divergence D ϕ ( x, y ) := ϕ ( x ) -ϕ ( y ) -⟨∇ ϕ ( y ) , x -y ⟩ . We further have:

Primal Space

Dual Space

在此处键入公式。

𝒳

𝒟

𝑥

(")

∇𝜙(𝒟)

∇𝜙

𝑥

("$%)

∇𝜙(𝑦

("#$)

)

∇𝜙(𝑥

(")

)

-∇𝑓(𝑥

!

)/𝜌

𝑦

("$%)

①

②

③

④

This updating procedure is vividly depicted by Fig.3. For each update of x ( k ) , MD conducts the following steps:

1 Apply mirror mapping ( ∇ ϕ ( · ) ) on x ( k ) from the primal space D to the dual space ∇ ϕ ( D ) .

2 Take a gradient step in the dual space, which is δ ( k +1) = ∇ ϕ ( x ( k ) ) -∇ f ( x ( k ) ) /ρ .

3 Inversely mapping ( ∇ ϕ -1 ( · ) ) the dual space back to the primal space, which is y ( k +1) = ∇ ϕ -1 ( δ ( k +1) ) , also written as ∇ ϕ ( y ( k +1) ) = ∇ ϕ ( x ( k ) ) -∇ f ( x ( k ) ) /ρ .

4 The updated point may fall out of the constraints of X . Thus, Bregman projection is conducted to project y ( k +1) to x ( k +1) ∈ X , where x ( k +1) = arg min x ∈X D ϕ ( x, y ( k +1) )) .

The relaxed FGW conducts projections to constraints of rows and columns alternately, which is written as min π f ( π ) + I { π ∈ Π 1 } + I { π ∈ Π 2 } , where I is the indicator function. We adopt the operator splitting strategy, and the optimization can be formulated as:

We apply Mirror Descent algorithm with Bregman divergence D ϕ ( x, y ) to solve this problem, whose update takes the form of:

Let's take the MD update of π as an example. Taking relative entropy as ϕ ( x ) = ∑ i x i log x i , we can solve y ( k +1) (conducting steps 1 to 3 ) with exact analytical solutions. Specifically, ∇ ϕ ( x ) = ∑ i (1 + log x i ) , and the update w.r.t. π writes:

Furthermore, under this setting, we also have an exact solution of the projection step 4 arg min π ∈ Π 1 D ϕ ( π, y ( k +1) )) = diag( µ 1 ./y ( k +1) 1 n 2 ) y ( k +1) . Then the MD update procedure can be formulated according to Prop. 3 as follows:

Furthermore, the sub-gradient of our FGW optimizing objective w.r.t. π is calculated as follows:

Taking Eq.17 into Eq.16, we have the update of π , and a similar procedure is conducted for ω . Setting ρ to 1 /γ , the final update procedure is shown in Line 8-11 in Alg.2.

B Proofs of Theoretical Results

B.1 Proof of Proposition 1

Proof. From Corollary 4.6 of [59] we have: for any Sinkhorn iteration optimizing π ∈ Π( µ, ν ) on two different marginals alternately µ, ν , let t 1 = inf { t ≥ 0 : H ( µ 2 t | µ ) ≤ 1 } -1 , when iterations t ≥ 2 t 1 :

where C 1 is a constant, and R is a fixed reference measure.

B.2 Proof of Proposition 2

Definition 2 (Bounded Linear Regularity, Definition 5.6 in [60]) . Let C 1 , C 2 , · · · , C N be closed convex subsets of R n with a non-empty intersection C . We call the set { C 1 , C 2 , · · · , C N } is bounded linearly regular if for every bounded subset B of R n , there exists a constant κ > 0 such that

The BLR condition naturally holds if all the C i are polyhedral sets.

Definition 3 (Strong Convexity) . A differentiable function f ( · ) is strongly convex with the constant m , if the following inequality holds for all points x, y in its domain:

where ∥·∥ is the corresponding norm.

In Mirror Descent algorithm, the Bregman divergence D h ( x, y ) requires h to be strongly convex.

Proof. Recall that the fixed point set of relaxed FGW is X rel = { π ∗ ∈ C 1 , ω ∗ ∈ C 2 : 0 ∈ ∇ f ( π ∗ )+ ρ ( ∇ h ( ω ∗ ) -∇ h ( π ∗ ))+ N C 1 ( π ∗ ) , and 0 ∈ ∇ f ( ω ∗ )+ ρ ( ∇ h ( π ∗ ) -∇ h ( ω ∗ ))+ N C 2 ( ω ∗ ) } . Let p ∈ N C 1 ( π ∗ ) and q ∈ N C 2 ( ω ∗ ) , X rel is denoted as:

We denote ˆ π = Proj C 1 ∩ C 2 ( π ∗ ) . According to Definition 2, as C 1 and C 2 are both polyhedral constraints and satisfy the BLR condition, we have:

The first inequality holds based on the triangle inequality of the norm distances, and the second inequality holds based on the definition of point-to-space distance in Def.2. Moreover, according to the definition of X rel , for any ( π ∗ , ω ∗ ) ∈ X rel , we have:

Summing up the equations above, we have:

The ( a ) equation holds based on the definition of Bregman Divergence D h ( x, y ) := h ( x ) -h ( y ) -⟨∇ h ( y ) , x -y ⟩ . Furthermore, due to the property of normal cones N C 1 ( π ∗ ) , N C 2 ( ω ∗ ) , we have p ⊤ ( π -π ∗ ) ≤ 0 , ∀ π ∈ C 1 and q ⊤ ( ω -ω ∗ ) ≤ 0 , ∀ ω ∈ C 2 . Taking π and ω as ˆ π ∈ C 1 ∩ C 2 , we have p ⊤ (ˆ π -π ∗ ) + q ⊤ (ˆ π -ω ∗ ) ≤ 0 . Therefore, from 22 we have the following inequality:

The ( b ) inequality holds as the optimization objective is a quadratic function with bounded constraints, which means the gradient norm has a natural constant upper bound C f . For the left hand side of the inequality above, the sum of Bregman divergences is also lower bounded by

as h ( · ) is σ -strongly convex in the definition of Bregman Divergence. Therefore, combining (24) and (23), we have:

Based on this important result, we analyze the distance between X and X rel , which is given by dist( π ∗ + ω ∗ 2 , X ) := min x ∈X ∥ ∥ ∥ π ∗ + ω ∗ 2 -x ∥ ∥ ∥ , π ∗ ∈ C 1 , ω ∗ ∈ C 2 . By adding up the two equations in 19 depicting the conditions on π ∗ and ω ∗ of X rel , we have:

The ( c ) derivation holds due to the linear property of the subgradient of FGW w.r.t. π (see Eq.17). Invoking this equality, we present the following uppper bound analysis on dist( π ∗ + ω ∗ 2 , X ) :

(27)

The ( d ) inequality comes from the Luo-Tseng error bound condition [61] and Prop.3.1 in [29]. The ( e ) holds based on Eq.26. The ( f ) equality holds based on the property of the normal cone, which is Proj C ( x + z ) = x, ∀ x ∈ C, z ∈ N C ( x ) . The ( g ) inequality holds by invoking Lemma 3.2 in [29], which tells ∥ ∥ proj C 1 ( x ) + proj C 2 ( y ) -2 proj C 1 ∩ C 2 ( x + y 2 )∥ ∥ ⩽ M ∥ ∥ proj C 1 ( x ) -proj C 2 ( y ) ∥ ∥ . Finally, combining (25) and (27), we can obtain the desired result, i.e.,

where τ := (2 κ +1) ϵC f M 2 σ is a constant.

C Complete Mixup Procedure

We provide the complete mixup procedure of our method in Algorithm 3. We conduct augmentation only on the training set, and we mixup graphs from every pair of classes. For a specific mixup of two

graphs, we apply our proposed algorithm FGWMixup ( ∗ ) (Alg.1, 2) to obtain the synthetic mixup graph ˜ G = (˜ µ , ˜ X , ˜ A ) . However, the numerical solution does not ensure a discrete adjacency matrix ˜ A . Thus, we adopt a thresholding strategy to discretize ˜ A , setting entries below and above the threshold to 0 and 1, respectively. The threshold is linearly searched between the maximum and minimum entries of ˜ A , aiming to minimize the density difference between the mixup graph and the original graphs. The mixup graphs are added to the training set for augmentation, and the training, validating and testing procedures follow the traditional paradigms.

̸

D Experimental Settings

D.1 Dataset Information

PROTEINS, NCI1, NCI109, IMDB-BINARY (IMDB-B) and IMDB-MULTI (IMDB-M) are the five benchmark graph classification datasets used in our experiments. When preprocessing the datasets, we remove all the isolated nodes (i.e., 0-degree nodes) in order to avoid invalid empty message passing of MPNNs. We present the detailed statistics of the preprocessed datasets in Table 6, including the number of graphs, the average/median node numbers, the average/median edge numbers, the dimension of node features, and the number of classes.

D.2 Backbones

We adopt two categories of GNN frameworks as the backbones. The first category is the virtual-nodeenhanced Message Passing Neural Networks (MPNNs), including vGCN and vGIN. The second category is the Graph Transformer-based networks, including Graphormer and GraphormerGD. The details of these GNN layers are listed as follows:

vGCN [1]: Graph Convolution Network is developed from the spectral GNNs with 1-order approximation of Chebychev polynomial expansion. The update of features can be regarded as message passing procedure. The graph convolution operator of each layer in the vGCN is

defined as: X ( l +1) = σ ( ˜ AX ( l ) W ( l ) ) , where ˜ A is the renormalized adjacency matrix, and W is the learnable parameters. We apply ReLU as the activation function σ ( · ) .

vGIN [2]: Graph Isomorphism Network takes the idea from the Weisfeiler-Lehman kernel, and define the feature update of each layer through explicit message passing procedure. The GIN layer takes the form of: x ( l +1) v = MLP ( l +1) ( ( 1 + ϵ ( l +1) ) · x ( l ) v + ∑ u ∈N ( v ) x ( l ) u ) , where ϵ is a learnable scalar. We apply two-layer MLP with ReLU activation in our implementation.

Graphormer [39]: Graphormer adopts the idea of the Transformer [62] architecture, which designs a multi-head global QKV-attention across all graph nodes, and encode graph topologies as positional encodings (spatial encoding in Graphormers). A Graphormer layer conducts feature updating as follows:

where ϕ ( · ) is the spatial encoding module implemented as embedding tables, and D is the shortest path distance of the graph. A is the attention matrix.

GraphormerGD [16]: GraphormerGD is an enhanced version of Graphormer, where the graph topologies are encoded with a General Distance metric defined by the combination of the shortest path distance (SPD) and the resistance distance [63] (RD). It mainly modifies the calculation of the attention matrix A in Graphormer:

where ϕ ( · ) is an MLP taking SPD embedding and RD gaussian kernel embedding as inputs.

For all the backbones, we apply the virtual node READOUT approach (which is the official READOUT function of Graphormers) instead of traditional global pooling. We apply 6 network layers and take 64 as the hidden dimension for all the backbones on all datasets. GraphormerGD has a specical model design on PROTEINS dataset, which applies 5 layers and 32 as the hidden dimension due to the GPU memory limitation.

D.3 Experimental Environment

Hardware Environment The experiments in this work are conducted on two machines: one with 8 Nvidia RTX3090 GPUs and Intel Xeon E5-2680 CPUs, one with 2 Nvidia RTX8000 GPUs and Intel Xeon Gold 6230 CPUs.

Software Environment Our experiments are implemented with Python 3.9, PyTorch 1.11.0, Deep Graph Library (DGL) [44] 1.0.2, and Python Optimal Transport (POT) [45] 0.8.2. The implementation of all the backbone layers are based on their DGL implementations. For the FGW solver, FGWMixup uses the official FGW distance solver implemented in POT, and FGWMixup ∗ uses the accelerated FGW algorithm implemented on our own.

D.4 Implementation Details

For our mixup algorithm FGWMixup , we follow [31] and apply the Euclidean distance as the metric measuring the distance of the node features and the graph adjacency matrix as the structural distances between nodes. We sample the mixup weight λ from the distribution Beta(0 . 2 , 0 . 2) and the trade-off coefficient of the structure and signal costs α are tuned in {0.05, 0.5, 0.95}. We set the size of the generated mixup graphs as the weighted average of the source graphs, which is ˜ n = λn 1 +(1 -λ ) n 2 . The maximum number of iterations of FGWMixup is set to 200 for the outer loop optimizing X and A , and 300 for the inner loop optimizing couplings π . The stopping criteria of our algorithm are the relative update of the optimization objective reaching below a threshold, which is set to 5e-4. For

FGWMixup ∗ , we select the step size of MD γ from {0.1, 1, 10}. The mixup ratio (i.e., the proportion of mixup samples to the original training samples) is set to 0.25.

For the training of GNNs, MPNNs are trained for 400 epochs and Graphormers are trained for 300 epochs, both using AdamW optimizer with a weight decay rate of 5e-4. The batch size of GNNs are chosen from {32, 128}, and the learning rate is chosen from {1e-3, 5e-4, 1e-4}. Dropout is employed with a fixed dropout rate 0.5 to prevent overfitting. All the hyperparameters are fine-tuned by grid search on validation sets. For a fair comparison, we employ the same set of hyperparameter configurations for all data augmentation methods in each backbone architecture. For more details, please check our code published online at https://github.com/ArthurLeoM/FGWMixup .

E Further Experimental Analyses

E.1 Qualitative Analyses

We present two mixup examples of FGWMixup in Figure 4 to demonstrate that augmented graphs by FGWMixup can preserve key topologies of the original graphs and have semantically meaningful node features simultaneously. In Example 1, we can observe that the mixup graph adopts an overall trident structure and a substructure (marked green) from G 1 , and adopts several substructures from G 2 (marked red), finally formulating a new graph sample combined properties from both graphs. In Example 2, we can observe that the mixup graph is quite similar to G 2 , but breaks the connection of two marked (red arrow pointed) edges and formulates two disconnected subgraphs, which is identical to the overall structure of G 1 . Moreover, in both examples, we can observe that the preserved substructures are not only topologically alike, but also highly consistent in node features. The two examples demonstrate that FGWMixup can both preserve key topologies and generate semantically meaningful node features.

G

1

G

2

(a)

(a)

(b)

(c)

(c)

(b)

G

1

G

2

Example 1

Example 2

0.61617

9.59195

E.2 Further Discussions on G -Mixup

In this subsection, we conduct further analyses on G -Mixup for a better understanding of its differences from our methods.

G -Mixup with GW Graphon Estimator G -Mixup does not originally apply GW metrics to estimate the graphons (as they introduce in Table 1 and 6 in their paper [20]). In our implementation, we select the USVT method as the graphon estimator for G -Mixup. Yet it is also practical to apply GWmetric to estimate graphons as an ablation study. Here, we also provide the experimental results of G -Mixup+GW and G -Mixup+GW* (GW with our single-loop solver) on PROTEINS as shown in Table 7.

No matter what metrics are applied for the graphon estimation, we want to emphasize that G -Mixup does not model the joint distribution of graph structure and signal spaces, but regards them as two disentangled and independent factors. However, our main contribution and the greatest advantage compared with G -Mixup comes from the joint modeling of graph structure and signal spaces. This also explains why FGWMixup can outperform.

Extend G -Mixup with FGW-based Attributed Graphon Estimator In [64], Xu et al. introduce an attributed graphon estimation algorithm, which makes us naturally think of an extended version of G -Mixup. With the attributed graphon, G -Mixup can also consider the joint modeling problem. Though it is not the contribution and the core idea from G -Mixup to address our proposed joint modeling problem, we are also interested in whether this extension can improve the performance of G -Mixup. Hence, we conduct experiments with the extended version of G -Mixup using FGW as the attributed graphon estimator (denoted as G -Mixup+FGW) on PROTEINS and NCI1 datasets, and the results are shown in Table 8.

Table 8: Experimental results of G -Mixup extended with FGW attributed graphon estimator on PROTEINS and NCI1.

We can observe that FGW does not significantly improve the performance of G -Mixup and still cannot outperform our methods. The main reason lies in that the node matching problem remains unsolved using the linear interpolation strategy of two graphons introduced in G -Mixup. Though the intra-class node matching has been done with FGW graphon estimation, the graphons of different classes are not ensured with an aligned node distribution, which requires the inter-class node matching for an appropriate mixup. This is the core reason for the limited performance of G -Mixup series methods.

E.3 Algorithm Efficiency

In this subsection, we provide additional details regarding the runtime efficiency analyses.

Further Comparisons between FGWMixup and FGWMixup ∗ Table 9 presents the average time spent on a single iteration of π update (i.e, FGW solver convergence) and the average iterations taken for updating X and A until convergence. We can observe that the convergence of a single FGW procedure does not appear to be significantly accelerated. This is because the gradient step

has been taken twice for an alternating projection process, which incurs double time for the gradient calculation (with a complexity of O ( n 3 ) , n is the graph size). In contrast, the strict solver only takes one gradient step for each projection. Therefore, despite a faster convergence rate of the relaxed FGW, the extra gradient step makes the overall convergence time of the relaxed FGW solver similar to that of the strict FGW solver. However, a notable improvement is observed in the convergence rate of the outer loop responsible for updating X and A , resulting from the larger step of π that the relaxed FGW provides, as shown in Prop.1. Considering the two factors above, the overall mixup time has been efficiently decreased by up to 3.46 × with FGWMixup ∗ .

Comparisons between Our Methods and Compared Baselines We present the averaged efficiencies of different mixup methods and time spent on training vanilla backbones of each fold on PROTEINS and NCI1 datasets in Table 10. As the efficiency of G -Mixup hugely relies on the graphon estimation approach, we also include G -Mixup with GW graphon estimators (denoted as G-Mixup+GW) in the table.

We can observe that our methods and G -Mixup+GW are slower than the other data augmentation methods. The main reason is that the complexity of calculating (F)GW distances between two graphs is cubic ( O ( mn 2 + nm 2 ) )[28], where m,n are the sizes of two graphs. Moreover, when calculating barycenters, we need an outer loop with T iterations and M graphs. In total, the time complexity of mixing up two graphs of size n is O ( MTn 3 ) . FGWMixup ∗ boosts the efficiency by enhancing the convergence rate and reducing the required iterations T (see Table 9 for more details), whereas G -Mixup+GW will have to go over the whole dataset to calculate graphons, which is much more time-consuming than FGWMixup .

However, sacrificing complexity to pursue higher performance has been the recent trend of technical development, e.g. GPT-4. Moreover, we believe that the current time complexity of FGWMixup ∗ is still acceptable compared with our performance improvements, as most compared graph augmentation methods cannot effectively enhance the model performance as shown in Table 1.

More importantly, in practice, the main computational bottleneck of (F)GW-based method is the OT network flow CPU solver in the current implementation based on the most widely used POT lib. In other words, GPU-based network flow algorithms have not been applied in current computation frameworks. Moreover, mini-batch parallelization is not yet deployed in POT. However, recent works [65] from NVIDIA have focused on accelerating network flow algorithms on GPU, which may probably be equipped on CUDA and allow a huge acceleration for GW solvers in the near future. Hence, we firmly believe that the fact that our method is not yet optimized in parallel on GPUs is only a temporary problem. Just as some other works (e.g., MLP, LSTM, etc.) that have brought enormous contributions in the past era, they are initially slow when proposed, but have become efficient with fast-following hardware supports.

E.4 Sensitivity Analysis on Varying Beta Distribution Parameter k

We empirically follow the setting in [24] to select the beta distribution parameter k = 0 . 2 where the mixup method is first proposed. We also provide the sensitivity analysis of the Beta distribution parameter k on PROTEINS and NCI1 datasets in Table 11.

From the results, we can find that Beta(0 . 2 , 0 . 2) is the overall best-performed setting. There are also a few circumstances where the other settings outperform. In our opinion, different datasets and backbones prefer different optimal settings of k . However, we should choose the one that is overall the best across various settings.

E.5 Experiments on OGB Datasets

We also conduct experiments on several OGB benchmark datasets[46], including ogbg-molhiv and ogbg-molbace. We conduct binary classification (molecular property prediction) on these datasets with AUROC (Area Under Receiver Operating Characteristic) as the reported metric. The split of training, validating, and testing datasets are provided by the OGB benchmark. In spite of the existence of edge features in OGB datasets, in this work, both of our GNN backbones and augmentation methods do not encode or consider the edge features. We select vGCN and vGIN (5 layers and 256 hidden dimensions) as the backbones and train the models five times with different random seeds, and we report the average and standard deviation of AUROCs as the results. The dataset information and the predicting performances are listed in Table 12.

From the table, we can observe evident improvements on both datasets and backbones with our methods. FGWMixup ∗ obtains the best performance on ogbg-molhiv dataset, with 2.69% and 3.46% relative improvements on vGCN and vGIN respectively, and FGWMixup obtains the best on ogbgmolbace dataset with 1.14% and 8.86% relative improvements. Meanwhile, our methods provide a much more stable model performance where we reduce the variance by a significant margin, which demonstrates that our methods can resist the potential noises underlying the data and increase the robustness of GNNs. These additional experimental results further indicate the effectiveness of our methods in improving the performance of GNNs in terms of their generalizability and robustness.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a novel graph mixup strategy, which focuses on synthesizing a 'midpoint' graph between two graphs based on the graph structure-signal product metric space. The authors utilize the Fused Gromov-Wasserstein (FGW) distance to achieve this and also propose a method to accelerate the computation of FGW distance solvers by relaxing the polytope constraint. The effectiveness of the proposed method is verified through experiments on molecule and social network datasets.\n\nSTRENGTHS:\n- The draft is well-written, and the proposed methods are technically sound  with theoretical analysis.\n\n- Performance improvements compared to the recent baseline are significant.\n\nWEAKNESSES:\n- Efficient computation is a key aspect when deploying augmentation strategies in real-world applications. However, Table 3 is not conclusive to me in its current form. Please provide the computation cost for the vanilla model and other baselines to better understand the tradeoff between performance gain and efficiency.\n\n- Lack of qualitative analysis. It remains somewhat unclear whether the synthetic 'midpoint' graphs possess semantic meaning. For instance, in the case of molecules, specific functional groups determine their properties. If these key groups are missing in the synthetic 'midpoint' graphs, it might mislead the network. To provide readers with more insights regarding the 'midpoint' graphs, including real samples of synthetic graphs is recommended.\n\n- The experiments are currently limited to small-scale graph datasets (1K~4K). To demonstrate the applicability of the proposed method to medium/large-scale graphs, conducting experiments on more large-scale datasets such as the OGB benchmark datasets is highly required.\n\nQUESTIONS:\nI noticed that Table 1 shows that the recent state-of-the-art baseline, G-Mixup, performs worse than the vanilla model in many cases. Is there a reason that the authors conjecture for these results?\n\nLIMITATIONS:\nThe authors addressed the limitations of their work in the draft.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors study the problem of graph data augmentation for graph-level classifications. They propose a mix-up strategy based on the computation of Fused Gromov-Wasserstein(FGW) “mid-point” (or barycenter) between a pair of graphs from the training dataset. To solve these optimization problems, they adapt a recent single-loop GW solver [28] to the FGW setting (i.e including node features in the OT problem) and provide a theoretical analysis of the resulting algorithm. Finally, they benchmark their novel mix-up strategy against SOTA mix-up approaches on five real-world datasets using 4 different GNN architectures and show that their method achieves better performances than its competitors.\n\nSTRENGTHS:\n- Authors propose an interesting mix-up strategy capable of incorporating both structure and feature information via the FGW distance.\n- They investigate two FGW solvers to estimate the inherent soft (attributed) graph matching problems in a novel task of graph dataset augmentation which is also interesting for the active optimization literature on (F)GW.\n- They partially extend the analysis done in [28] for GW to FGW.\n- They benchmark their FGWmixup with various SOTA mix-up strategies with 4 different backbones on 5 relatively small real-world datasets (small graphs with around 1k to 4k samples). The supplementary material also contains concluding experiments on the ogbg-molhiv datasets containing +40k samples. Plus study the robustness to label corruption of FGWmixup.\n- They study different mix-up strategies w.r.t the size of the FGW barycenters with pairwise (local) and global strategies (e.g proportional to the median size in the dataset)\n- Overall I find the paper well-written\n\nWEAKNESSES:\n- 1. *[addressed by authors]* **The FGWmixup is badly positioned with respect to the G-mixup** and the current literature on GW (resp. FGW) for the estimation of graphons (resp. attributed graphons): \nFrom my understanding, the G-mixup [18] strategy comes down to estimate 1 graphon per class then perform mix-up by sampling a graph from a linear interpolation between a pair of these graphons. [A] actually showed that a GW barycenter better estimates graphon than all methods considered in G-mixup paper. As discussed in [B], we can consider that an analog result holds for estimating attributed graphons using FGW barycenters. These observations highlight the following weaknesses / missing points in authors’ overall reasoning currently illustrated in the papier:\n\n\n     - a) G-mixup should be benchmarked using GW solvers which correspond to the FGW solvers studied by the authors. Note that [A] rather studied the Proximal Point solver introduced in [C], which seems to lead to lower performances on several tasks than the single-loop solver in [28] which coincides with your FGWmixup*.\n\n     - b) The extension of the G-mixup strategy using FGW instead of GW should also be discussed and benchmarked. In the same spirit, an ablation study with $\\alpha = 1$ should be considered for the FGWmixup. Basically, I would expect from your paper to identity what is better between a global mix-up strategy (G-mixup) or a pairwise one (FGWmixup) for data augmentation. From my point of view, the first one is favorably biased by a class (as a unique barycenter represents a class) but in the meantime the generated graphs are poorly representative of the graph dataset manifold as only one graphon is taken per class. Whereas your FGWmixup is not really biased by a class (potentially too local) but the generated graphs are intrinsically more representative of the input graphs.\n\n- 2. *[partially addressed by authors, the effect of removing marginals' cost is still unclear]* The single loop solver for FGW problems used in Algorithm 2, should be better positioned w.r.t to the one in [28] for GW. It is an exact adaptation from [28] adding the linear OT cost from node features. Moreover you propose to solve for an equivalent problem, where you removed the fixed terms on the marginals. This should be clear in the paper.\n\n     - a) As you remove fixed terms on the marginals, you solve an equivalent problem to get an OT solution. De facto, equation 4 is wrong as you do not consider exactly the gradient of FGW. Make this clearer in the paper please (maybe via different notations for both objective functions).\n\n     - b) In the spirit of [28, Table 7], the feasibility error i.e errors on marginals should be benchmarked across used solvers. I’m worried that removing the marginal terms in the FGW cost makes this feasibility error problem worse. Also this analysis should be completed with an analysis of the estimated FGW distances across used FGW solvers, over a dataset considered in the mix-up experiments.\n\n     - c) The theoretical results, especially proposition 2, are fairly easily proven by following [28]. This reduces the credit we can give to your theoretical contributions.\n\n- 3. [*partially addressed by the authors, the extension to full learning of the 'midpoint' distribution requires more in-depth analysis and will be the subject of future work.*] On the form there are still some points which can be easily improved:\n\n     - a) L93. Mu is not a probability measure, but a probability vector. + L95, references for ‘normalized’ degree distributions ?\n\n     - b) FGW (semi-)metric properties hold on the space of metric spaces quotiented by the notion of strong-isomorphism, like graphs depicted by distance matrices and invariant to permutations. Otherwise, following [D] it can be extended to any graph representations but w.r.t the notion of weak-isomorphism, hence FGW rather defines a pseudo-metric. \n\n     - c) I find the optimization parts confusing and heavy to read. I believe that their readability could be clearly improved by referring efficiently to specific steps in the algorithm and/or specific updates, instead of the confusing nested loop vocabulary. \n\n     - d) I got confused with Equation 2 being a nested bi-level optimization problem, could you further elaborate ? In my opinion it is more simple than that knowing that Atilde and Xtilde are not involved in the coupling constraints.\n\n     - e) Equation 2. In practice it seems that you do not consider learning the masses of the barycenter which removes a degree of freedom in your model. Learning those has actually been showed to be beneficial for many graph representation learning tasks (see [51, 52, 54]). Could you further justify your choice ? and I guess address that as a current limitation of your work. (Considering that it is not difficult to deduce a subgradient for the masses when using bregman projection based solvers.)\n\n     - f)  I believe that the paragraph ‘Effects of Mixup Graph Sizes’ could also take into consideration [E] where graph size also is a current limitation and does not exactly fall into the mentioned problem of graph size generalization. I believe that the latter problem mostly focuses on the question: can we generalize to big graphs when learning on small graphs ? (maybe I’m wrong)\n\n[A] Xu, Hongteng, et al. \"Learning graphons via structured gromov-wasserstein barycenters.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\n[B] Xu, Hongteng, et al. \"Learning Graphon Autoencoders for Generative Graph Modeling.\" arXiv preprint arXiv:2105.14244(2021).\n\n[C] Xu, Hongteng, et al. \"Gromov-wasserstein learning for graph matching and node embedding.\" International conference on machine learning. PMLR, 2019.\n\n[D] Chowdhury, Samir, and Facundo Mémoli. \"The gromov–wasserstein distance between networks and stable network invariants.\" Information and Inference: A Journal of the IMA 8.4 (2019): 757-787.\n\n[E] Brogat-Motte, Luc, et al. \"Learning to predict graphs with fused Gromov-Wasserstein barycenters.\" International Conference on Machine Learning. PMLR, 2022.\n\n*Update after rebuttal : The authors have fully or partially addressed most of my concerns through their rebuttals and discussions. Considering that the authors have undertaken to amend the paper and supplementary material accordingly, I increase my grade from 4 (borderline reject) to 6 (weak accept).*\n\nQUESTIONS:\nI refer the authors to the weaknesses' list above for first suggestions and questions. Follows some questions/suggestions to potentially make the paper clearer:\n\nQ1. Could you please explicit the thresholding method used to deduce new sample from the FGWmixup strategy ?\n\nQ2. Could you report an ablation study with respect to the alpha parameter in FGW for the validated values {0.05, 0.5, 0.95} ?\n\nLIMITATIONS:\nI think discussing the weaknesses I've mentioned and answering my questions detailed above would help identify and work around the other limitations of their current work. If the authors manage to address these, I will happily increase my rating.\n\n This work has no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new graph data augmentation method for graph-level classifications. To address the limitation of existing methods, the authors consider the joint interaction between the graph structure and node features by finding an optimal inter-graph node matching strategy. Furthermore, the authors introduce a relaxed FGW solver to accelerate the proposed method FGWMixup. Experiments show that the FGWMixup outperforms multiple baselines on five datasets using different GNN backbones.\n\nSTRENGTHS:\nOverall the paper is well-written and easy to read. \n\nUsing the Fused Gromov-Wasserstein distance metric space seems to be an interesting solution for mixing graphs. \n\nThe authors provide theoretical analysis for accelerating FGWMixup by improving the convergence rate.\n\nWEAKNESSES:\nMy major concern is about the claim that most existing graph data augmentation methods only consider one of the graph structure space and node feature space. There exist many graph augmentation methods (adversarial augmentation, local graph augmentation, automated graph augmentation and etc). Many of them change both graph structure and node features to generate augmented graphs. I don't think this work is the only one considering the joint modeling problem. \n\nBesides, the authors claim the importance of the interaction between graph structure space and node features space. I think more explanations/analyses are needed to show that augmented graphs by FGWMixup can preserve key topologies of the original graphs and have semantically meaningful node features simultaneously.\n\nFor the experiments, the improvement of the performance is not significant compared to the high std. I would suggest the authors include experiments on the OGB benchmark(https://ogb.stanford.edu/docs/graphprop/) since the std on OGB is typically small compared to TUDataset. Furthermore, except for some graph mixup methods, the authors only compare with DropEdge and DropNode. It would make the experimental results more convincing if the authors can include some advanced graph data augmentation methods.\n\nQUESTIONS:\nThere are multiple hyperparameters in the proposed FGWMixup, ex. $\\alpha$ in equation 1. How to choose the hyperparameters? How sensitive is the performance gain to the hyperparameter tuning?\n\nThe authors provide experiments about the running time between FGWMixup and FGWMixup*. How does FGWMixup compare to the other baselines? Does the improvement of the performance come from a much higher computational cost?\n\nLIMITATIONS:\nYes. The authors have discussed the limitation of their method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a gap in graph data augmentation for graph-level classifications, where existing methods mainly focus on augmenting graph signal space and graph structure space separately, overlooking their mutual interactions. The authors formulate the issue as an optimal transport problem that considers both graph structures and signals, and propose a novel graph mixup algorithm called FGWMixup. FGWMixup seeks the \"midpoint\" of source graphs in the Fused Gromov-Wasserstein (FGW) metric space. The authors further introduce a relaxed FGW solver to improve the scalability and performance of FGWMixup, and experimental results across five datasets and various GNN backbones demonstrate its effectiveness in enhancing the generalizability and robustness of GNNs.\n\nSTRENGTHS:\nS1. Innovation in Graph Mixup Method: The paper introduces FGWMixup, a novel graph mixup method that is formulated as an optimal transport problem. This innovative approach aims to find the optimal graph at the \"midpoint\" of two source graphs in terms of both graph signals and structures. This unique formulation could potentially provide a more effective way to combine graphs compared to traditional methods.  \nS2. Comprehensive Evaluation: The paper comprehensively evaluates the proposed method on five widely-used graph classification tasks from the graph benchmark dataset. The use of four different types of backbones for the evaluation further demonstrates the versatility and robustness of the FGWMixup method. This extensive evaluation provides strong evidence of the effectiveness of the proposed method across various tasks and settings.  \nS3. Theoretical Analysis and Optimization: The paper provides a thorough theoretical analysis of the proposed FGWMixup method, including the convergence of the algorithm and the correctness of FGWMixup. This rigorous theoretical foundation strengthens the credibility of the proposed method. Additionally, using an algorithm to optimize the computation complexity is a commendable effort to address potential efficiency concerns.\n\nWEAKNESSES:\nW1. High Time Complexity: The proposed FGWMixup and Accelerated FGWMixup methods have a relatively high time complexity. This could limit their applicability in scenarios where computational resources or time are constrained. The paper would benefit from a more detailed analysis of the time complexity of these methods, including how it scales with the size and complexity of the input graphs.  \nW2. Lack of Results for Varying Beta Distribution Parameter: The paper needs results showing the impact of changing the beta distribution parameter 'k' on the performance of the proposed methods. This parameter could significantly influence the distribution, and it would be informative to understand how its variation affects the results. This could also provide insights into how to choose the best 'k' for different scenarios.  \nW3. Effects of Relaxed Projection: The paper could provide more discussion about the potential negative effects due to the relaxed projection. While this approach may improve the algorithm's efficiency, it could also introduce inaccuracies or instability in the results. A deeper exploration of these trade-offs would be beneficial.\n\nQUESTIONS:\nQ1. The algorithm may sacrifice some feasibility due to the relaxed projection. Can you provide some potential effects?  \nQ2. How does the computational complexity of FGWMixup compare to other graph mixup methods?\n\nLIMITATIONS:\nThe paper does not explicitly address the potential limitation of the proposed approach. The authors can test their method on node classification or link prediction tasks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study a new method \"FGWMixup\" for graph data augmentation. \n\nFor the two input graphs $G_1, G_2$, they propose to construct a synthetic graph $\\tilde G$ through optimizing the weighted distance sum (2). To further improve the efficiency of the algorithm, they relax the polytope constraint (row sum and column sum meet the source and target distribution) on the transport matrix $\\pi$, to two alternatively enforced constraints (separately on row sum and column sum); they summarize the new solver in Algorithm 2.\n\nThey also study the performance of the proposed method/solver through empirical experiments.\n\nSTRENGTHS:\n- originality: \n    - While the rough idea to use OT for data augmentation is regular, this paper proposes a different solution and a new relaxed FGW solver, which is novel to me.\n\n- quality:\n    - The new method/solver they propose does work and can provide comparable performance to existing mixup methods.\n\n- significance: \n    - This paper provides a OT-based mixup method, which can be a good reference for self-supervised learning on graph.\n\nWEAKNESSES:\n- quality\n    - Some empirical results may be unconvincing due to the small size of datasets. I leave a related comment below.\n\n- clarity: \n    - Some concepts are not clearly illustrated. See the questions below. With the unclear illustration, the theoretical justifications for some claims are hard to follow.\n\n- significance: \n    - Even with the new approximate solver, I doubt the new FGWmixup is much slower than previous mixup methods, which can make the work less attractive to practitioners.\n\nQUESTIONS:\n1. In solving (2), they \"fix the node probability distribution of $\\tilde G$ with a uniform distribution\" (Line 129). It makes sense, while it can be better to add more justification, either empirically or theoretically.\n2. The concept of “Bregman projection” in Line 164, 171 is not well explained.\n3. The statement of Proposition 1 is confusing. \n        - \"Let $\\pi_t$ be the Sinkhorn iterations\" is unclear. What's the relationship between $\\pi_t$ and the following $\\pi_{2t}, \\pi_{2t+1}$?\n        - \"Let $\\pi^*$ be the unique optimal solution\" needs more explanation, optimal w.r.t. what? \n        - What's the definition of $\\mu_t, \\nu_t$?\n        - Does the proposition mean the new solver may not converge to $\\pi^*$?\n4. The statement of Proposition 2 is also confusing. \n        - The concept of “FGW function” is not defined.\n        - The concept of “normal cone” is not well explained.\n        - Actually the follow-up remark after Prop 2 is not very informative. Please provide more details and explain why the distance bound implies FGWMixup∗ \"will converge close to the ideal optimum\". What's the scale of $\\tau/\\rho$?\n5. The experiments only involve small datasets. Can the authors add some medium to large datasets, and also report the runtime of different mixup methods?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}