Title: Outlier-Robust Gromov-Wasserstein for Graph Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gromov Wasserstein

Robust Optimization

Nonconvex Optimization
Abstract: Gromov-Wasserstein (GW) distance is a powerful tool for comparing and aligning probability distributions supported on different metric spaces. Recently, GW has become the main modeling technique for aligning heterogeneous data for a wide range of graph learning tasks. However, the GW distance is known to be highly sensitive to outliers, which can result in large inaccuracies if the outliers are given the same weight as other samples in the objective function. To mitigate this issue, we introduce a new and robust version of the GW distance called RGW. RGW features optimistically perturbed marginal constraints within a Kullback-Leibler divergence-based ambiguity set. To make the benefits of RGW more accessible in practice, we develop a computationally efficient and theoretically provable procedure using Bregman proximal alternating linearized minimization algorithm. Through extensive experimentation, we validate our theoretical results and demonstrate the effectiveness of RGW on real-world graph learning tasks, such as subgraph matching and partial shape correspondence.

Outlier-Robust Gromov-Wasserstein for Graph Data

Abstract

Gromov-Wasserstein (GW) distance is a powerful tool for comparing and aligning probability distributions supported on different metric spaces. Recently, GW has become the main modeling technique for aligning heterogeneous data for a wide range of graph learning tasks. However, the GW distance is known to be highly sensitive to outliers, which can result in large inaccuracies if the outliers are given the same weight as other samples in the objective function. To mitigate this issue, we introduce a new and robust version of the GW distance called RGW. RGW features optimistically perturbed marginal constraints within a KullbackLeibler divergence-based ambiguity set. To make the benefits of RGW more accessible in practice, we develop a computationally efficient and theoretically provable procedure using Bregman proximal alternating linearized minimization algorithm. Through extensive experimentation, we validate our theoretical results and demonstrate the effectiveness of RGW on real-world graph learning tasks, such as subgraph matching and partial shape correspondence.

1 Introduction

Gromov-Wasserstein distance (GW) [26, 28] acts as a main model tool in data science to compare data distributions on unaligned metric spaces. Recently, it has received much attention across a host of applications in data analysis, e.g., shape correspondence [24, 31, 36, 27], graph alignment and partition [37, 38, 15, 18, 14, 44], graph embedding and classification [41, 43], unsupervised word embedding and translation [3, 19], generative modeling across incomparable spaces [8, 45].

In practice, the robustness of GW distance suffers heavily from its sensitivity to outliers. Here, outliers mean the samples with large noise, which usually are far away from the clean samples or have different structures from the clean samples. The hard constraints on the marginals in the Gromov-Wasserstein distance require all the mass in the source distribution to be entirely transported to the target distribution, making it highly sensitive to outliers. When the outliers are weighted similarly as other clean samples, even a small fraction of outliers corrupted can largely impact the GWdistance value and the optimal coupling, which is unsatisfactory in real-world applications.

To overcome the above issue, some recent works are trying to relax the marginal constraints of GWdistance. [33] introduces a L 1 relaxation of mass conservation of the GW distance. However, this reformulation replaces the strict marginal constraint that the transport plan should be a joint distribution with marginals as specific distributions by the constraint that only requires the transport plan to be a joint distribution, which can easily lead to over-relaxation. On another front, [10] propose a so-called partial GW distance (PGW), which only transports a fraction of mass from source distribution to target distribution. The formulation of PGW is limited to facilitating mass

GW = 0.0

(a) Clean dataset

GW = 0.148

(b) Dataset with outliers

RGW = 0.002

(c) Dataset with outliers

destruction, which restricts its ability to handle situations where outliers exist predominantly on one side. A formulation that allows both mass destruction and creation is proposed in [35] called unbalanced GW (UGW). The UGW relaxes the marginal constraint via adding the quadratic φ -divergence as the penalty function in the objective function and extends GW distance to compare metric spaces equipped with arbitrary positive measures. Additionally, [40] proved that UGW is robust to outliers and can effectively remove the mass of outliers with high transportation costs. However, UGW is sensitive to the penalty parameter as it balances the reduction of outlier impact and the control of marginal distortion in the transport plan. On the computational side, an alternate Sinkhorn minimization method is proposed to calculate the entropy-regularized UGW. Note that the algorithm does not exactly solve UGW but approximates the lower bound of the entropic regularized UGWinstead. From a statistical viewpoint, these works do not establish a direct link between the reformulated GW distance and the GW distance in terms of uncontaminated levels.

In this work, we propose the robust Gromov-Wasserstein (RGW) to estimate the GW distance robustly when dealing with outliers. To achieve this, RGW simultaneously optimizes the transport plan and selects the best marginal distribution from a neighborhood of the given marginal distributions, avoiding contaminated distributions. Perturbed marginal distributions help to re-weight the samples and lower the weight assigned to the outliers. The introduction of relaxed distributions to handle outliers draws inspiration from robust OT techniques [4, 29, 23]. Unlike robust OT, which is convex, RGWis non-convex, posing algorithmic challenges. This idea is also closely related to optimistic modelings of distribution ambiguity in data-driven optimization, e.g., upper confidence bound in the multi-armed bandit problem and reinforcement learning [7, 30, 1], data-driven distributionally robust decision-making with outliers [21, 9], etc.

Moreover, inspired by UGW, RGW relaxes the marginal constraint via adding the Kullback-Leibler (KL) divergence between the marginals of the transport plan and the perturbed distributions as the penalty function in the objective function to lessen the impact of the outliers further. Instead of utilizing the quadratic KL divergence as employed in unbalanced GW, we opt for KL divergence due to its computational advantages. It allows for convex subproblems with closed-form solutions, as opposed to the linearization required for non-convex quadratic KL divergence, which could be challenging algorithmically. Furthermore, we leverage the convexity of KL divergence to establish the statistical properties of RGW, leading to an upper bound by the true GW distance with explicit control through marginal relaxation parameters and marginal penalty parameters. This statistical advantage is disrupted by the non-convex nature of quadratic KL divergence. Additionally, KL divergence aligns with our goal of outlier elimination and is less sensitive to outliers compared to quadratic KL divergence, which is more outlier-sensitive due to its quadratic nature. Overall, RGW combines the introduction of perturbed marginal distributions with the relaxation of hard marginal constraints to achieve greater flexibility, allowing control over marginal distortion through marginal penalty parameters and reduction of outlier impact using marginal relaxation parameters.

To realize the modeling benefits of RGW, we further propose an efficient algorithm based on the Bregman proximal alternating linearized minimization (BPALM) method. The updates in each

iteration of BPALM can be computed in a highly efficient manner. On the theoretical side, we prove that the BPALM algorithm converges to a critical point of the RGW problem. Empirically, we demonstrate the effectiveness of RGW and the proposed BPALM algorithm through extensive numerical experiments on subgraph alignment and partial shape correspondence tasks. The results reveal that RGW surpasses both the balanced GW-based method and the reformulations of GW, including PGW and UGW.

Our Contributions We summarize our main contributions as follows:

We develop a new robust model called RGW to alleviate the impact of outliers on the GW distance. The key insight is to simultaneously optimize the transport plan and perturb marginal distributions in the most efficient way.

On the statistical side, we demonstrate that the robust Gromov-Wasserstein is bounded above by the true GW distance under the Huber ϵ -contamination model.

On the computational side, we propose an efficient algorithm for solving RGW using the BPALM method and prove that the algorithm converges to a critical point of the RGW problem.

Empirical results on subgraph alignment and partial shape correspondence tasks demonstrate the effectiveness of RGW. This is the first successful attempt to apply GW-based methods to partial shape correspondence, a challenging problem pointed out in [36].

2 Problem Formulation

In this section, we review the definition of Gromov-Wasserstein distance and formally formulate the robust Gromov-Wasserstein. Following that, we discuss the statistical properties of the proposed robust Gromov-Wasserstein model under the Huber ϵ -contamination model.

For the rest of the paper, we will use the following notation. Let ( X,d X ) be a complete separable metric space and denote the finite, positive Borel measure on X by M + ( X ) . Let P ( X ) ⊂ M + ( X ) denotes the space of Borel probability measures on X . We use ∆ n to denote the simplex in R n . We use 1 n and 1 n × m to denote the n -dimensional all-one vector and n × m all-one matrix. We use S n to denote the set of n × n symmetric matrice. The indicator function of set C is denoted as I C ( · ) .

2.1 Robust Gromov-Wasserstein

The Gromov-Wasserstein (GW) distance aims at matching distributions defined in different metric spaces. It is defined as follows:

Definition 2.1 (Gromov-Wasserstein) . Suppose that we are given two unregistered complete separable metric spaces ( X,d X ) , ( Y, d Y ) accompanied with Borel probability measures µ, ν respectively. The GWdistance between µ and ν is defined as

where Π( µ, ν ) is the set of all probability measures on X × Y with µ and ν as marginals.

As shown in the definition, the sensitivity to outliers of Gromov-Wasserstein distance is due to its hard constraints on marginal distributions. This suggests relaxing the marginal constraints such that the weight assigned to the outliers by the transport plan can be small. To do it, we invoke the Kullback-Leibler divergence, defined as d KL ( α, µ ) = ∫ X α ( x ) log ( α ( x ) /µ ( x )) dx , to soften the constraint on marginal distributions. We also introduce an optimistically distributionally robust mechanism to perturb the marginal distributions and reduce the weight assigned to outliers. Further details on this mechanism will be discussed later.

Definition 2.2 (Robust Gromov-Wasserstein) . Suppose that we are given two unregistered complete separable metric spaces ( X,d X ) , ( Y, d Y ) accompanied with Borel probability measures µ, ν respectively. The Robust GW between µ and ν is defined as

where F ( α, β ) =

and ( π 1 , π 2 ) are two marginals of the joint distribution π , defined by π 1 ( A ) = π ( A × Y ) for any Borel set A ⊂ X and π 2 ( B ) = π ( X × B ) for any Borel set B ⊂ Y .

The main idea of our formulation is to optimize the transport plan and perturbed distribution variables in the ambiguity set of the observed marginal distributions jointly. This formulation aims to find the perturbed distributions that approximate the clean distributions and compute the transport plan based on the perturbed distributions. However, incorporating the constraints of equal marginals between the transport plan π and the perturbed distributions α and β directly poses challenges in developing an algorithm due to potential non-smoothness issues. Inspired by [35], we address this challenge by relaxing these marginal constraints and incorporating the KL divergence terms, denoted as d KL ( π 1 , α ) and d KL ( π 2 , β ) , into the objective function as penalty functions. Different from [35], we use KL divergence instead of quadratic KL divergence due to its joint convexity, which is more amenable to algorithm development, as quadratic KL divergence is typically non-convex. Besides, transforming the hard marginal constraints into penalty functions can further lessen the impact of outliers on the transport plan.

Our new formulation extends the balanced GW distance and can recover it by choosing ρ 1 = ρ 2 = 0 and letting τ 1 and τ 2 tend to infinity. When properly chosen, ρ 1 and ρ 2 can encompass the clean distributions within the ambiguity sets. In this scenario, the relaxed reformulation closely approximates the original GW distance in a certain manner. Building on this concept, we prove that RGW can serve as a robust approximation of the GW distance without outliers, given some mild assumptions on the outliers.

2.2 Robustness Guarantees

Robust Gromov-Wasserstein aims at mitigating the sensitivity of the GW distance to outliers, which can result in large inaccuracies if the outliers are given the same weight as other samples in the objective function. Specifically, RGW is designed to address the issue of the GW distance exploding as the distance between the clean samples and the outliers goes to infinity. In general, even a small number of outliers can cause the GW distance to change dramatically when added to the marginal distributions. To formalize this, consider the Huber ϵ -contamination model popularized in robust statistics [22, 11, 12]. In that model, a base measure µ c is contaminated by an outlier distribution µ a to obtain a contaminated measure µ ,

Under this model, data are drawn from µ defined in (2).

Under the assumption of the Huber ϵ -contamination model, it can be demonstrated that by selecting suitable values of ρ 1 and ρ 2 , the robust Gromov-Wasserstein distance ensures that outliers are unable to substantially inflate the transportation distance. For robust Gromov-Wasserstein, we have the following bound:

Theorem 2.3. Let µ and ν be two distributions corrupted by fractions ϵ 1 and ϵ 2 of outliers, respectively. Specifically, µ is defined as (1 -ϵ 1 ) µ c + ϵ 1 µ a , and ν is defined as (1 -ϵ 2 ) ν c + ϵ 2 ν a , where µ c and ν c represent the clean distributions, and µ a and ν a represent the outlier distributions. Then,

In Appendix B, we provide a proof that constructs a feasible transport plan and relaxed marginal distributions. By relaxing the strict marginal constraints, we can find a feasible transport plan that closely approximates the transport plan between the clean distributions and obtain relaxed marginal distributions that approximate the clean distributions.

The derived bound indicates that robust Gromov-Wasserstein provides a provably robust estimate under the Huber ϵ -contamination model. If the fraction of outliers is known, the upper bound for the robust GW is determined by the true Gromov-Wasserstein distance, along with additional terms that account for the KL divergence between the clean distribution and the outlier distribution for both µ and ν . The impact of this factor is determined by the extent of relaxation in the marginal distributions ρ 1 and ρ 2 . By carefully choosing ρ 1 = ϵ 1 d KL ( µ a , µ c ) and ρ 2 = ϵ 2 d KL ( ν a , ν c ) , we can tighten the upper bound on the robust GW value, while still keeping it below the original GW distance (excluding outliers). Importantly, substituting these values of ρ 1 and ρ 2 yields GW rob ρ 1 ,ρ 2 ( µ, ν ) ≤ GW ( µ c , ν c ) , indicating that the robust GW between the contaminated distribution µ and ν is upper bounded by the original GW distance between the clean distribution µ c and ν c .

Remark 2.4. The following inequality for UGW under the Huber ϵ -contamination model can be derived using the same techniques as in Theorem 2.3:

We observe that the terms d ⊗ KL ( µ c , µ ) and d ⊗ KL ( µ c , µ ) cannot be canceled out unless τ 1 and τ 2 are set to zero, resulting in over-relaxation. However, RGW allows us to control the error terms d KL ( µ c , µ a ) and d KL ( ν c , ν a ) through the marginal relaxation parameters ρ 1 and ρ 2 .

3 Proposed Algorithm

3.1 Problem Setup

To start with our algorithmic developments, we consider the discrete case for simplicity and practicality, where µ and ν are two empirical distributions, i.e., µ = ∑ n i =1 µ i δ x i and ν = ∑ m j =1 ν j δ y j . Denote D ∈ S n , D ik = d X ( x i , x k ) and ¯ D ∈ S m and ¯ D jl = d Y ( y j , y l ) . We construct a 4-way tensor as follows:

We define the tensor-matrix multiplication as

Then, the robust GW admits the following reformulation:

Here, π 1 = π 1 m and π 2 = π T 1 n .

3.2 Bregman Proximal Alternating Linearized Minimization (BPALM) Method

As Problem (3) is non-convex and involves three variables, we employ BPALM [5, 2] to solve it. By choosing the KL divergence as Bregman distance, the updates of this algorithm are given by:

Here, t k , c k , and r k are stepsizes in BPALM.

In our algorithm updates, we employ distinct proximal operators for π and α (and β ). The use of d KL ( π, π k ) in π -subproblem (4) allows for the application of the Sinkhorn algorithm, while the introduction of d KL ( α k , α ) in α -subproblem (5) facilitates a closed-form solution, which we will detail in the following part.

To solve the π -subproblem, we can utilize the Sinkhorn algorithm for the entropic regularized unbalanced optimal transport problem. This algorithm, which has been previously introduced in [13, 32], is well-suited for our needs. As for the α -subproblem, we consider the case where ρ 1 is strictly larger than 0. Otherwise, when ρ 1 = 0 , α should simply equal µ , making the subproblem unnecessary. To solve the α -subproblem, we attempt to find the optimal dual multiplier w ∗ . Specifically, consider the problem:

Let ˆ α ( w ) represent the optimal solution to (7), and we define the function p : R + → R by p ( w ) = d KL ( µ, ˆ α ( w )) -ρ 1 . We prove the convexity, differentiability, and monotonicity of p , which are crucial for developing an efficient algorithm for (5) later.

Proposition 3.1. Problem (7) has a closed-form solution

If w satisfies (i) w = 0 and p ( w ) ≤ 0 , or (ii) w > 0 , p ( w ) = 0 , then ˆ α ( w ) is the optimal solution to the α -subproblem (5) . Moreover, p ( · ) is convex, twice differentiable, and monotonically non-increasing on R + .

Given Proposition 3.1, we begin by verifying p (0) ≤ 0 . If this condition is not met, and given that p (0) > 0 while lim w → + ∞ p ( w ) = -ρ 1 < 0 , it implies that p possesses at least one root within R + . The following proposition provides the framework to seek the root of p by employing Newton's method, with the initialization set at 0. Hence, the α -subproblem can be cast to search a root of p in one dimension, in which case it can be solved efficiently.

Proposition 3.2. Let p ( · ) : I → R be a convex, twice differentiable, and monotonically nonincreasing on the interval I ⊂ R . Assume that there exists an ˜ x , ¯ x ∈ I such that p (˜ x ) > 0 and p (¯ x ) < 0 . Then p has a unique root on I , and the sequence obtained from Newton's method with initial point x 0 = ˜ x will converge to the root of p .

Since the β -subproblem shares the same structure as the α -subproblem, we can apply this method to search for the optimal solution to the β -subproblem.

3.3 Convergence Analysis

To illustrate the convergence result of BPALM, we consider the compact form for simplicity:

where f ( π ) = ⟨L ( D, ¯ D ) ⊗ π, π ⟩ , q ( π ) = I { π ≥ 0 } ( π ) , g 1 ( π, α ) = τ 1 d KL ( π 1 m , α ) , g 2 ( π, β ) = τ 2 d KL ( π T 1 n , β ) , h 1 ( α ) = I { α ∈ ∆ n ,d KL ( µ,α ) ≤ ρ 1 } ( α ) , and h 2 ( β ) = I { β ∈ ∆ m ,d KL ( ν,β ) ≤ ρ 2 } ( β ) .

The following theorem states that any limit point of the sequence generated by BPALM belongs to the critical point set of problem (3).

Theorem 3.3 (Subsequence Convergence) . Suppose that in Problem (1) , the step size t k in (4) satisfies 0 < t ≤ t k < ¯ t ≤ σ/L f for k ≥ 0 where t , ¯ t are given constants and L f is the gradient Lipschitz constant of f . The step size c k in (5) and r k in (6) satisfy 0 < r ≤ c k , r k < ¯ r for k ≥ 0 where r , ¯ r are given constants. Any limit point of the sequence of solutions { π k , α k , β k } k ≥ 0 belongs to the critical point set X , where X is defined by

For the sake of brevity, we omit the proof. We refer the reader to Appendix C for further details.

4 Experiment Results

In this section, we present comprehensive experimental results to validate the effectiveness of our proposed RGW model and BPALM algorithm in various graph learning tasks, specifically subgraph alignment and partial shape correspondence. Traditionally, balanced GW has been applied successfully in scenarios where the source and target graphs have similar sizes. However, in our approach, we treat the missing part of the target graph as outliers and leverage RGW for improved performance. All simulations are conducted in Python 3.9 on a high-performance computing server running Ubuntu 20.10, equipped with an Intel(R) Xeon(R) Silver 4214R CPU. Our code is available at https://github.com/lmkong020/outlier-robust-GW.

4.1 Partial Shape Correspondence

In this subsection, we first investigate a toy matching problem in a 2D setting to support and validate our theoretical insights and results presented in Section 2. Figure 2 (a) illustrates an example where we aim to map a two-dimensional shape without symmetries to a rotated version of the same shape while accounting for outliers in the source domain. Here, we sample 300 points from the source shape and 400 points from the target shape. Additionally, we introduce 50 outliers by randomly adding points from a discrete uniform distribution on [ -3 , -2 . 5] × [0 , 0 . 5] to the source domain. The distance matrices, D and ¯ D are computed using pairwise Euclidean distances.

Figures 2 provide visualizations of the coupling matrices and objective values for all the models, highlighting the matching results. In Figure 2(c), it is evident that even a small number of outliers has a significant impact on the coupling and leads to an increased estimated GW distance. While unbalanced GW and partial GW attempt to handle outliers to some extent, they fall short in achieving accurate mappings. On the other hand, our robust GW formulation, RGW, effectively disregards outliers and achieves satisfactory performance. Additionally, the objective value of RGW closely approximates the true GW distance without outliers, as indicated by Theorem 2.3, approaching zero.

3

2

1

0

1

(a) Shape Geometry

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

0

100

200

300

0

50

100

150

200

250

300

0

100

200

300

(c) Balanced GW:

GW

=0.59

0

50

100

150

200

250

300

0

100

200

300

(d) Unbalanced GW:

GWub

=3.72

0

50

100

150

200

250

300

0

100

200

300

(e) Partial GW:

GWpartial

=0.08

0

50

100

150

200

250

300

0

100

200

300

(f) Robust GW:

GWrob

=0.04

0

50

100

150

200

250

300

We evaluate the matching performance of RGW on the TOSCA dataset [6, 34] for partial shape correspondence. For the initial estimation of the transport plan in both RGW and PGW, we utilize the partial functional map method [34], employing 30 eigenfunctions. This method establishes an initial matching relationship between the source and target shapes. This method establishes an initial matching relationship between the source and target shapes by setting π ik to 1 for matching pairs ( i, k ) and 0 otherwise. The resultant transport plan is scaled by ∥ π ∥ 1 = ∑ ij π ij . UGWis not suitable for this large-scale task due to its long execution time. As shown in Figure 3, RGW outperforms PGW, and it enhances the performance of the initial point.

4.2 Subgraph Alignment

The subgraph alignment problem, which involves determining the isomorphism between a query graph and a subgraph of a larger target graph, has been extensively studied [16, 20]. While the

(a) Shape Geometry

(e) Robust GW

(d) Partial GW

(c) Partial Functional Map

20

40

60

80

100

120

140

100

200

300

400

20

20

20

40

40

40

60

60

60

80

80

80

100

100

100

120

120

120

140

140

140

100

200

300

400

100

200

300

400

100

200

300

400

restricted quadratic assignment problem is commonly used for graphs of similar sizes, the GW distance provides an optimal probabilistic correspondence that preserves the isometric property. In the subgraph alignment context, the nodes in the target graph, excluding those in the source graph, can be considered outliers, making the RGW model applicable to this task. In our comparison, we evaluate RGW against various methods, including unbalanced GW, partial GW, semi-relaxed GW (srGW) [42], RGWD [25], and methods for computing balanced GW such as FW [39], BPG [46], SpecGW [15], eBPG [36], and BAPG [24].

Database Statistics We evaluate the methods on synthetic and real databases. In the synthetic database, we generate target graphs G t using Barabasi-Albert models with scales ranging from 100 to 500 nodes. The source graphs G s are obtained by sampling connected subgraphs of G t with a specified percentage of nodes. This process results in five synthetic graph pairs for each setup, totaling 200 graph pairs. The Proteins and Enzymes biological graph databases from [15] are also used, following the same subgraph generation routine. For the Proteins database, we evaluate the accuracy of matching the overlap between two subgraphs with 90% overlap and between a subgraph and the entire graph, presented in the "Proteins-2" and "Proteins-1" columns, respectively. We compute the matching accuracy by comparing the predicted correspondence set S pred to the ground truth correspondence set S gt, with accuracy calculated as Acc = |S gt ∩S pred | / |S gt | × 100% . In addition, we also evaluate our methods on the Douban Online-Offline social network dataset, which consists of online and offline graphs, representing user interactions and presence at social gatherings, respectively. The online graph includes all users from the offline graph, with 1,118 users serving as ground truth alignments. Node locations are used as features in both graphs. In line with previous works [18, 37], we gauge performance using the Hit@k metric, which calculates the percentage of nodes in set V t where the ground truth alignment includes V s among the top-k candidates.

0.2

0.3

0.4

0.5

Ratio of number of nodes

0

20

40

60

80

100

Accuracy (%)

SpecGW

BPG

BAPG

UGW

PGW

(a) Synthetic

0.2

0.3

0.4

0.5

Ratio of number of nodes

0

20

40

60

Accuracy (%)

SpecGW

BPG

BAPG

UGW

PGW

(b) Proteins

0.2

0.3

0.4

0.5

Ratio of number of nodes

0

20

40

60

80

Accuracy (%)

SpecGW

BPG

BAPG

UGW

PGW

(c) Enzymes

Parameters Setup We use unweighted symmetric adjacency matrices D and ¯ D as input distance matrices. SpecGW employs the heat kernel ( exp( -L ) ) with the normalized graph Laplacian matrix L . Both µ and ν are set as uniform distributions. For SpecGW, BPG, eBPG, BAPG, srGW, and RGWD, we follow their respective paper setups. FW is implemented using the default PythonOT package. UGW selects the best results from regularization parameter sets { 0 . 5 , 0 . 2 , 0 . 1 , 0 . 01 , 0 . 001 } and marginal penalty parameter sets { 0 . 1 , 0 . 01 , 0 . 001 } . PGW reports the best results in the range of transported mass from 0.1 to 0.9. RGW sets τ 1 = τ 2 = 0 . 1 and selects the best results from the ranges { 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 } for marginal relaxation parameters and { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 } for step size parameters ( t k , c k , r k ). The transport plan initialization uses different approaches depending on the dataset. For the Synthetic , Proteins , and Enzymes datasets, we employ the 1 n × m / ( nm ) approach. For the Douban Online-Offline dataset, we initialize the transport plan using classical optimal transport conducted on the feature space. Additionally, for all datasets, we initialize α and β with uniform distributions.

Result of All Methods Table 1 and Figure 4 provide a comprehensive comparison of matching accuracy and computation time across various methods on the Synthetic , Proteins , and Enzymes datasets. Notably, RGW demonstrates superior accuracy compared to other methods, while maintaining comparable computation time with UGW and PGW. On the other hand, methods for computing the balanced GW exhibit poor performance on all datasets, particularly on the extensive Synthetic graph database. This can be attributed to their struggle in fulfilling the hard marginal constraint on the source side and addressing the presence of outliers. Furthermore, the introduction of local minima in the balanced GW problem due to partial source graph structures contributes to the suboptimal results. UGWperforms well with low target graph partiality but suffers significant degradation as partiality increases. PGW, aimed at mitigating outlier impact, inadvertently affects the matching between clean samples by reducing the mass transported from the source domain. In addition, RGW achieved the best results with the highest Hit@1 and Hit@10 on the Douban Online-Offline dataset, improving performance from 4.04% and 14.9% respectively for the initial point created by features.

Selection of Hyperparameters ρ and τ The hyperparameters ρ 1 , ρ 2 , τ 1 , and τ 2 in our formulation serve distinct roles. Specifically, ρ 1 and ρ 2 control the extent of marginal relaxation, while τ 1 and τ 2 act as the marginal penalty parameters. As indicated by Theorem 2.3, we have the flexibility to fix the value of τ and adjust ρ to accommodate outlier effects. To determine optimal hyperparameter values, we conduct two analyses. First, we examine the effect of varying ρ while maintaining a fixed subgraph node ratio and constant τ on the Enzymes dataset. Second, we adjust τ while holding ρ constant and assess the impact across the Synthesis , Proteins , and Enzymes datasets in the task of matching a 50%

0

0.01

0.05

0.1

0.2

0.5

1

10

Marginal Relaxation Parameter

0

20

40

60

80

Accuracy (%)

ratio = 0.5

ratio = 0.4

ratio = 0.3

ratio = 0.2

0.01

0.1

0.2

0.5

1

10

Penalty Parameter

20

40

60

80

100

Accuracy (%)

Synthetic

Proteins

Enzymes

subgraph to the entire graph. In Figure 5, the left side demonstrates that while keeping the ratio and τ fixed but varying ρ , accuracy diminishes when ρ becomes either excessively large or too small. However, within the range of 0.05 to 1, accuracy remains resilient, highlighting the mitigating effect of marginal relaxation on outlier influence. Notably, accuracy significantly improves when ρ falls within the range of 0.05 to 1 compared to the scenario with ρ = 0, thus affirming the significance of marginal relaxation in our model. A similar trend is observed for τ on the right side of Figure 5, where maintaining a fixed ratio and ρ while adjusting τ results in stable accuracy within the range of 0.1 to 1 but declining beyond this range. Consequently, in RGW computations, we can initially set ρ to 0.2 and τ to 0.1 by default and subsequently adjust ρ based on the outlier ratio: increasing it for a large ratio and decreasing it for a small ratio.

4.3 Tightness of the Bound in Theorem 2.3

Our primary focus is on scenarios where the GW distance between clean samples is nearly zero or zero due to noise, such as in partial shape correspondence and subgraph alignment tasks. In such cases, it is possible to find an isometric mapping from the query subgraph to a portion of the entire graph. By appropriately selecting the value of ρ , as discussed in Section 2.2, the upper bound in RGW becomes the GW distance between clean samples, which is zero. As RGW is always nonnegative, this upper bound is tight in this context.

To empirically validate this, we conducted experiments on the toy example in Section 4.1, and Figure 6 (a) illustrates the function values of PGW, UGW, and RGW with varying outlier ratios. The results confirm that the value of RGW can remain close to zero as the ratio of outliers increases. Additionally, Figure 6 (b) shows the function value of RGW and its upper bound as ρ varies. Both the RGW value and its upper bound decrease, converging to zero as ρ increases. This observation provides empirical support for Theorem 2.3. Regarding UGW and its upper bound with changing τ , we observed that both the UGW value and its upper bound increase as τ becomes larger, as shown in Figure 6 (c). Unlike RGW, UGW's τ must strike a balance between reducing outlier impact and preserving marginal distortion in the transport plan. This demands a careful balance and caution against setting τ excessively close to zero, which could lead to over-relaxation and potentially deteriorate the performance.

0.0

0.1

0.2

0.3

0.4

0.5

Ratio of outliers

0.00

0.05

0.10

0.15

0.20

Function value

UGW

PGW

0.01

0.1

1.0

5.0

Relaxation parameter

0.00

0.04

0.08

0.12

Function value

RGW-UB

0.01

0.1

1.0

2.0

Penalty parameter

0.0

0.5

1.0

1.5

2.0

2.5

Function value

UGW-UB

UGW

5 Conclusion

In this paper, we introduce RGW, a robust reformulation of Gromov-Wasserstein that incorporates distributionally optimistic modeling. Our theoretical analysis demonstrates its robustness to outliers, establishing RGW as a reliable estimator. We propose a Bregman proximal alternating linearized minimization method to efficiently solve RGW. Extensive numerical experiments validate our theoretical results and demonstrate the effectiveness of the proposed algorithm. Regarding the robust estimation of Gromov-Wasserstein, a natural question is whether we can recover the transport plan from the RGW model. On the computational side, our algorithm suffers from the heavy computation cost due to the use of the unbalanced OT as our subroutine, which limits its application in largescale real-world settings. To address this issue, a natural future direction is to develop single-loop algorithms to leverage the model benefits of robust GW for real applications.

Acknowledgments and Disclosure of Funding

Anthony Man-Cho So is supported in part by the Hong Kong Research Grants Council (RGC) General Research Fund (GRF) project CUHK 14203920.

References

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pages 104-114. PMLR, 2020.

Masoud Ahookhosh, Le Thi Khanh Hien, Nicolas Gillis, and Panagiotis Patrinos. Multiblock Bregman proximal alternating linearized minimization and its application to orthogonal nonnegative matrix factorization. Computational Optimization and Applications , 79(3):681-715, 2021.

David Alvarez-Melis and Tommi S Jaakkola. Gromov-Wasserstein alignment of word embedding spaces. arXiv preprint arXiv:1809.00013 , 2018.

Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. Advances in Neural Information Processing Systems , 33:12934-12944, 2020.

Jérôme Bolte, Shoham Sabach, and Marc Teboulle. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Mathematical Programming , 146(1):459-494, 2014.

Alexander M Bronstein, Michael M Bronstein, and Ron Kimmel. Numerical geometry of non-rigid shapes . Springer Science & Business Media, 2008.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka. Learning generative models across incomparable spaces. In International Conference on Machine Learning , pages 851-861. PMLR, 2019.

Junyu Cao and Rui Gao. Contextual decision-making under parametric uncertainty and datadriven optimistic optimization. Available at Optimization Online , 2021.

Laetitia Chapel, Mokhtar Z Alaya, and Gilles Gasso. Partial optimal transport with applications on positive-unlabeled learning. Advances in Neural Information Processing Systems , 33:29032913, 2020.

Mengjie Chen, Chao Gao, and Zhao Ren. Robust covariance and scatter matrix estimation under Huber's contamination model. The Annals of Statistics , 46(5):1932-1960, 2018.

Sitan Chen, Frederic Koehler, Ankur Moitra, and Morris Yau. Online and distribution-free robustness: Regression and contextual bandits with huber contamination. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 684-695. IEEE, 2022.

Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation , 87(314):25632609, 2018.

Samir Chowdhury and Facundo Mémoli. The Gromov-Wasserstein distance between networks and stable network invariants. Information and Inference: A Journal of the IMA , 8(4):757-787, 2019.

Samir Chowdhury and Tom Needham. Generalized spectral clustering via Gromov-Wasserstein learning. In International Conference on Artificial Intelligence and Statistics , pages 712-720. PMLR, 2021.

Luigi P Cordella, Pasquale Foggia, Carlo Sansone, and Mario Vento. A (sub) graph isomorphism algorithm for matching large graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence , 26(10):1367-1372, 2004.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al. Pot: Python optimal transport. The Journal of Machine Learning Research , 22(1):3571-3578, 2021.

Ji Gao, Xiao Huang, and Jundong Li. Unsupervised graph alignment with Wasserstein distance discriminator. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Data Mining , pages 426-435, 2021.

Edouard Grave, Armand Joulin, and Quentin Berthet. Unsupervised alignment of embeddings with Wasserstein procrustes. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1880-1890. PMLR, 2019.

Myoungji Han, Hyunjoon Kim, Geonmo Gu, Kunsoo Park, and Wook-Shin Han. Efficient subgraph matching: Harmonizing dynamic programming, adaptive matching order, and failing set together. In Proceedings of the 2019 International Conference on Management of Data , pages 1429-1446, 2019.

Nan Jiang and Weijun Xie. DFO: A framework for data-driven decision-making with endogenous outliers. Preprint optimization-online.org , 2021.

S Kassam and J Thomas. Asymptotically robust detection of a known signal in contaminated non-gaussian noise. IEEE Transactions on Information Theory , 22(1):22-26, 1976.

Khang Le, Huy Nguyen, Quang M Nguyen, Tung Pham, Hung Bui, and Nhat Ho. On robust optimal transport: Computational complexity and barycenter computation. Advances in Neural Information Processing Systems , 34:21947-21959, 2021.

Jiajin Li, Jianheng Tang, Lemin Kong, Huikang Liu, Jia Li, Anthony Man-Cho So, and Jose Blanchet. A convergent single-loop algorithm for relaxation of Gromov-Wasserstein in graph data. In The Eleventh International Conference on Learning Representations , 2023.

Weijie Liu, Jiahao Xie, Chao Zhang, Makoto Yamada, Nenggan Zheng, and Hui Qian. Robust graph dictionary learning. In The Eleventh International Conference on Learning Representations , 2022.

Facundo Mémoli. On the use of Gromov-Hausdorff Distances for Shape Comparison. In M. Botsch, R. Pajarola, B. Chen, and M. Zwicker, editors, Eurographics Symposium on PointBased Graphics . The Eurographics Association, 2007.

Facundo Mémoli. Spectral Gromov-Wasserstein distances for shape matching. In 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops , pages 256263. IEEE, 2009.

Facundo Mémoli. Gromov-Wasserstein distances and the metric approach to object matching. Foundations of computational mathematics , 11:417-487, 2011.

Debarghya Mukherjee, Aritra Guha, Justin M Solomon, Yuekai Sun, and Mikhail Yurochkin. Outlier-robust optimal transport. In International Conference on Machine Learning , pages 7850-7860. PMLR, 2021.

Rémi Munos et al. From bandits to monte-carlo tree search: The optimistic principle applied to optimization and planning. Foundations and Trends® in Machine Learning , 7(1):1-129, 2014.

Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-Wasserstein averaging of kernel and distance matrices. In International Conference on Machine Learning , pages 2664-2672. PMLR, 2016.

Khiem Pham, Khang Le, Nhat Ho, Tung Pham, and Hung Bui. On unbalanced optimal transport: An analysis of sinkhorn algorithm. In International Conference on Machine Learning , pages 7673-7682. PMLR, 2020.

Emanuele Rodola, Alex M Bronstein, Andrea Albarelli, Filippo Bergamasco, and Andrea Torsello. A game-theoretic approach to deformable shape matching. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 182-189. IEEE, 2012.

Emanuele Rodolà, Luca Cosmo, Michael M Bronstein, Andrea Torsello, and Daniel Cremers. Partial functional correspondence. In Computer graphics forum , volume 36, pages 222-236. Wiley Online Library, 2017.

Thibault Séjourné, François-Xavier Vialard, and Gabriel Peyré. The unbalanced Gromov Wasserstein distance: Conic formulation and relaxation. Advances in Neural Information Processing Systems , 34, 2021.

Justin Solomon, Gabriel Peyré, Vladimir G Kim, and Suvrit Sra. Entropic metric alignment for correspondence problems. ACM Transactions on Graphics (ToG) , 35(4):1-13, 2016.

Jianheng Tang, Weiqi Zhang, Jiajin Li, Kangfei Zhao, Fugee Tsung, and Jia Li. Robust attributed graph alignment via joint structure learning and optimal transport. In 2023 IEEE 39th International Conference on Data Engineering (ICDE) , pages 1638-1651, 2023.

Jianheng Tang, Kangfei Zhao, and Jia Li. A fused Gromov-Wasserstein framework for unsupervised knowledge graph entity alignment. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 3320-3334.

Vayer Titouan, Nicolas Courty, Romain Tavenard, and Rémi Flamary. Optimal transport for structured data with application on graphs. In International Conference on Machine Learning , pages 6275-6284. PMLR, 2019.

Quang Huy Tran, Hicham Janati, Nicolas Courty, Rémi Flamary, Ievgen Redko, Pinar Demetci, and Ritambhara Singh. Unbalanced CO-Optimal transport. In Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI) , 2023.

Cédric Vincent-Cuaz, Titouan Vayer, Rémi Flamary, Marco Corneli, and Nicolas Courty. Online graph dictionary learning. In International Conference on Machine Learning , pages 1056410574. PMLR, 2021.

Cédric Vincent-Cuaz, Rémi Flamary, Marco Corneli, Titouan Vayer, and Nicolas Courty. Semi-relaxed Gromov Wasserstein divergence with applications on graphs. In International Conference on Learning Representations (ICLR) , 2022.

Hongteng Xu, Jiachang Liu, Dixin Luo, and Lawrence Carin. Representing graphs via GromovWasserstein factorization. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable Gromov-Wasserstein learning for graph partitioning and matching. Advances in Neural Information Processing Systems , 32, 2019.

Hongteng Xu, Dixin Luo, Lawrence Carin, and Hongyuan Zha. Learning graphons via structured Gromov-Wasserstein barycenters. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10505-10513, 2021.

Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-Wasserstein learning for graph matching and node embedding. In International Conference on Machine Learning , pages 6932-6941. PMLR, 2019.

A Organization of the Appendix

We organize the appendix as follows:

The proof details of Theorem 2.3 and discussion of Remark 2.4 are given in Section B.

The proof details of the algorithm, including the properties of p , the convergence analysis of Newton's method, and Bregman proximal alternating linearized minimization method are collected in C.

Additional experiment results are summarized in Section D.

B Proof of Robust Guarantee in Section 2.2

B.1 Proof of Theorem 2.3

Proof. GW rob ρ 1 ,ρ 2 ( µ, ν ) is defined as

Consider π c , the optimal transport plans for GW ( µ c , ν c ) . Notably, π c serves as a feasible solution for GW rob ρ 1 ,ρ 2 ( µ, ν ) . Consequently, we can deduce that:

To establish an upper bound for GW ( µ, ν ) , let us begin by addressing the following problem:

We consider the distribution of the form (1 -γ ) µ + γµ c , for γ ∈ [0 , 1] . Then we prove that if γ ≤ min ( ρ 1 ϵ 1 d KL ( µ a , µ c ) , 1 ) , then (1 -γ ) µ + γµ c is a feasible solution for problem (8).

By the joint convexity of KL divergence, we have

Therefore,

The largest value γ can take is ρ 1 ϵ 1 d KL ( µ a , µ c ) . This gives

Similarly, we can prove that inf β ∈P ( Y ) , d KL ( ν,β ) ≤ ρ 2 d KL ( ν c , β ) ≤ max ( 0 , 1 -ρ 2 ϵ 2 d KL ( ν a , ν c ) ) ϵ 2 d KL ( ν c , ν a ) . This completes the proof.

B.2 Proof and Discussion of Remark 2.4

Utilizing the same notations as in the proof of Theorem 2.3, and considering that π c is a feasible solution to the UGW problem, substituting it directly leads to the desired result.

Furthermore, referring to [40, Theorem 1] and adapting the notations to our framework, we have the following property: specifically, when considering only µ is corrupted with outliers, characterized by µ = (1 -ϵ 1 ) µ c + ϵ 1 µ a , while ν = ν c . If we let δ = 2( τ 1 + τ 2 ) ϵ 1 , and K = M + 1 M UGW ( µ c , ν )+ δ , where M represents the transported mass between clean data and ∆ ∞ signifies the maximal deviation between the contaminated source and the target, then the following inequality holds:

Given that UGW ( µ c , ν ) ≤ GW ( µ c , ν ) , we can derive the following relationship:

Comparing this result to Remark 2.4, (9) also incorporates GW ( µ c , ν ) in the exponential term and M , the transported mass between clean data. These variables may be influenced by the marginal parameters τ 1 and τ 2 . As a result, finding an optimal choice for τ 1 and τ 2 to establish a tight upper bound for UGW using the GW distance between clean samples, as presented in (9), proves challenging. As outlined in Remark 2.4, it is worth noting that setting τ 1 = τ 2 = 0 represents the sole means to attain a tight upper bound. Nevertheless, in real-world applications, this approach is impractical, as it could lead to over-relaxation and compromise the performance.

C Proof Details of Bregman Proximal Alternating Linearized Minimization Method for Robust GW

Given a vector x, we use ∥ x ∥ 2 to denote its ℓ 2 norm. We use ∥ X ∥ F to denote the Frobenius norm of matrix X . For a convex set C and a point x , we define the distance between C and x as

C.1 Proof of Proposition 3.1

Proof. Problem (7) can be written as

We first consider a relaxed problem of the problem above:

Consider the Lagrangian function of the relaxed problem

Let

Then we obtain that

Since ∑ n i =1 α i = 1 , ∑ n i =1 α k i = 1 , and ∑ n i =1 µ i = 1 , then 1 + λ = ∑ ij π k +1 ij + 1 c k + w . Thus, the optimal solution to the relaxed problem is

We can see that ˆ α ( w ) ≥ 0 . Hence, ˆ α ( w ) is also the optimal solution to problem (7).

Since if w satisfies (i) or (ii), (ˆ α ( w ) , w ) is a solution to KKT conditions of problem (5), therefore, ˆ α ( w ) is an optimal solution to problem (5). Next, we prove that p is differentiable when h is relative entropy. Problem (7) admits the closed-form solution

By substituting (10) into p , p ( w ) can be written as

Thus, p is twice differentiable. The first-order derivative and second-order of p are

and

Then we prove that p ′ ( w ) ≤ 0 and p ′′ ( w ) ≥ 0 for w ≥ 0 , so p is monotonically non-increasing and convex on R + . Let s i = ( π k +1 1 m ) i + 1 c k α k i + µ i w and s = ( π k +1 1 m ) i + 1 c k α k i + µ i w . Note that s = ∑ n i =1 s i . Then p ′ and p ′′ can be written as

and

Therefore, it is equivalent to show ∑ n i =1 µ i µ i s s i ≥ 1 and ∑ n i =1 µ i µ 2 i s 2 s 2 i ≥ 1 .

Recall that 1 x and 1 x 2 are convex on R ++ , then

C.2 Proof of Proposition 3.2

Proof. First, prove that p only has one root on I . Since p is continuous on I and there exists ˜ x , ¯ x ∈ I such that p (˜ x ) > 0 and p (¯ x ) < 0 , p contains at least one root on [˜ x, ¯ x ] . Since p is non-increasing, p cannot have roots outside [˜ x, ¯ x ] . Suppose that p have two different roots z 1 and z 2 on [˜ x, ¯ x ] and z 1 < z 2 . By convexity of p , we have

This is a contradiction. So p has a unique root on I .

p ′ ( x ) ≤ 0 since p is non-increasing on I . Denote the root of p as r . Claim that p ′ ( x ) < 0 for x ∈ [˜ x, r ] . Otherwise, there exist x ∈ [˜ x, r ] such that p ′ ( x ) = 0 , then

which leads to a contradiction. Especially, p ′ (˜ x ) < 0 , and we can set ˜ x as the initial point of Newton's method.

The update of Newton's method is

Therefore, x k +1 ≥ x k and { x k } k ≥ 1 is an increasing sequence. Since p is convex,

x k ≤ r because p is a monotonically non-increasing function. { x k } k ≥ 1 is an increasing sequence with an upper bound, so it has a limit x ∗ and lim k →∞ ( x k -x k +1 ) = 0 . Also, p ′ is bounded on [˜ x, r ] since it is continuous. Therefore,

Hence, the sequence generated by Newton's method converges to a root of p .

C.3 Proof of Theorem 3.3

Assumption C.1. The critical point set X is non-empty.

Before the proof of Theorem 3.3, we first prove that sequence { π k } k ≥ 0 generated by BPALM lies in a compact set.

Proposition C.2. Sequence { π k } k ≥ 0 generated by BPALM lies in a compact set.

Proof. We prove that { π k } k ≥ 0 lies in the compact set A := { π ∈ R n × m : 0 ≤ π ij ≤ 1 } by mathematical induction. For k = 0 , we can initialize π 0 with 0 ≤ π 0 ij ≤ 1 . Suppose that π k ∈ A and π k +1 / ∈ A . Then there exist i ∈ [ n ] and j ∈ [ m ] such that π k +1 ij > 1 . Recall that π k +1 is the optimal solution to the problem

Observe that function ϕ ( x ) := x log x a -x + a is a unimodal function on R + and achieve its minimum at x = a . Since α i , β j , and π k ij are smaller than or equal to 1, α i , β j , and π k ij are strictly smaller than π k +1 ij . Let ˜ π ∈ R n × m ,

Then φ (˜ π ) < φ ( π k ) , this contradicts to π k +1 is the optimal solution to problem (11). Thus, π k +1 ∈ A .

For the proof of Theorem 3.3, we first prove the sufficient decrease property of BPALM, i.e., there exist a constant κ 1 > 0 and an index k 1 ≥ 0 such that for k ≥ k 1 ,

And then we prove the subsequence convergence result.

Proof. It is worth noting that f ( π ) is a quadratic function, i.e., f ( π ) = 〈 L ( D, ¯ D ) ⊗ π, π 〉 , then f ( π ) is gradient Lipschitz continuous with the constant max i,j ( ∑ k,l L ( D, ¯ D ) 2 i,j,k,l ) 1 / 2 . To simplify the

( ♢ ) is because as x log x is σ -strongly convex, we have

By letting κ 1 = σ 2 min (( 1 t k -L f σ ) , 1 ¯ r ) > 0 , we get

Summing up (12) from k = 0 to + ∞ , we obtain

As F is coercive and {( π k , α k , β k )} is a bounded sequence, it follows that the left-hand side is bounded. This implies

and

Let l ( x ) = ∑ i x i log x i . Recall the optimality condition of BPALM, we have

Let ( π ∞ , α ∞ , β ∞ ) be a limit point of the sequence { ( π k , α k , β k ) } k ≥ 0 . Then, there exists a sequence { n k } k ≥ 0 such that { ( π n k , α n k , β n k ) } k ≥ 0 converges to ( π ∞ , α ∞ , β ∞ ) . Since we assume that h is twice continuous differentiable and α k and β k are in a compact set, then ∇ 2 l ( α k ) and ∇ 2 l ( β k ) are bounded. Therefore, lim k →∞ ∇ 2 l ( α k +1 )( α k +1 -α k ) = 0 and lim k →∞ ∇ 2 l ( β k +1 )( β k +1 -β k ) = 0 . Replacing the k by n k in (13), (14), and (15), taking limits on both sides as k →∞ , we obtain that

Thus ( π ∞ , α ∞ , β ∞ ) belongs to X .

C.4 Discussion of Computational Complexity of PGW, UGW, and RGW

We consider the measure min 0 ≤ k ≤ K ( ∥ π k +1 -π k ∥ F + ∥ α k +1 -α k ∥ 2 + ∥ β k +1 -β k ∥ 2 ) as the stationary measure, and it is observed that the convergence rate of our algorithm is O ( 1 √ K ) . Similarly, the Frank-Wolfe algorithm for PGW also exhibits a convergence rate of O ( 1 √ K ) . The literature on UGWdoes not provide a discussion on the convergence rate of alternate Sinkhorn minimization for UGW. In each iteration of PGW, UGW, and RGW, the computation of L ( D, ¯ D ) ⊗ π k is a crucial step. According to [31], the complexity of this computation is O ( n 2 m + m 2 n ) . Additionally, PGW involves utilizing the network simplex algorithm to solve a linear programming problem as a subroutine, which has a complexity of O (( n 2 m + m 2 n ) log 2 ( n + m )) . On the other hand, both UGW and RGW utilize the sinkhorn algorithm to solve an entropic unbalanced optimal transport problem. The complexity of the sinkhorn algorithm for unbalanced OT is O (( n 2 + m 2 ) / ( ε log( ε ))) for computing an ε -approximation.

D Additional Experiment Results

D.1 Additional Experiment Results on Subgraph Alignment

Source codes of all baselines used in this paper:

FW [17]: https://github.com/PythonOT/POT

SpecGW [15]: https://github.com/trneedham/Spectral-Gromov-Wasserstein

eBPG [17]: https://github.com/PythonOT/POT

BAPG [24]: https://github.com/squareRoot3/Gromov-Wasserstein-for-Graph

UGW[35]: https://github.com/thibsej/unbalanced_gromov_wasserstein

PGW [10, 17]: https://github.com/PythonOT/POT

srGW: [42]: https://github.com/cedricvincentcuaz/srGW

RGWD: [25]: https://github.com/cxxszz/rgdl

Results in Figure 4 The data utilized to create Figure 4 is provided in Table 2 and Table 3.

Selection of Stepsize t k , c k , and r k In the subgraph alignment task, we have used constant values for the stepsizes t k , c k and r k . We have conducted a sensitivity analysis for these parameters, and the details are summarized in Tables 4 and 5. Specifically, Table 4 reveals that RGW achieves its highest accuracy with t in the range of 0.01 to 0.05, allowing us to select t = 0 . 01 as the default. Table 5 further indicates that accuracy is not significantly affected by variations in c , leading us to set c = 0 . 1 as the default.

Experiment Results of Normalized Degree as Marginal Distribution In addition to employing the uniform distribution as node distribution, we also explore the use of normalized degrees as node distribution. The results presented in Table 6 confirm that RGW surpasses other methods in terms of accuracy when utilizing normalized degrees as marginal distributions.

Experiment Results of Adding Noise to Query Graph We conducted an experiment by adding 10% pseudo edges to the subgraphs, and the results can be found in Table 7. These findings

demonstrate that RGW significantly outperforms other methods on the Enzymes and Proteins datasets.

D.2 Additional Experiment Results on Partial Shape Correspondence

Convergence of BPALM The convergence results for the proposed BPALM using various step sizes are presented in Figure 7 for the toy example discussed in Section 4.1.

Additional Experiment Result on TOSCA Dataset Additional experiment results on TOSCA Dataset are shown in Figure 8, 9, and 10.

0

100

200

300

400

500

Iteration

5

4

3

2

1

0

log10(

F

(

k

,

k

,

k

)

F

(

*

,

*

,

*

))

t

=0.01

t

=0.05

t

=0.1

t

=0.5

t

=1.0

0

100

200

300

400

500

Iteration

5

4

3

2

1

0

log10(

F

(

k

,

k

,

k

)

F

(

*

,

*

,

*

))

c

=0.01

c

=0.05

c

=0.1

c

=0.5

c

=1.0

(a) Shape Geometry

(e) Robust GW

(d) Partial GW

(c) Partial Functional Map

50

100

150

200

250

100

200

300

400

500

50

50

50

100

100

100

150

150

150

200

200

200

250

250

250

100

200

300

400

500

100

200

300

400

500

100

200

300

400

500

(a) Shape Geometry

(e) Robust GW

(d) Partial GW

(c) Partial Functional Map

20

40

60

80

100

120

140

100

200

300

400

500

20

20

20

40

40

40

60

60

60

80

80

80

100

100

100

120

120

120

140

140

140

100

200

300

400

500

100

200

300

400

500

100

200

300

400

500

(a) Shape Geometry

(e) Robust GW

(d) Partial GW

(c) Partial Functional Map

50

100

150

200

100

200

300

400

500

50

50

50

100

100

100

150

150

150

200

200

200

100

200

300

400

500

100

200

300

400

500

100

200

300

400

500

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Robust Gromov-Wasserstein (RGW) distance to tackle the high sensitivity to outliers of common GW distances. The key idea is to relax both the marginal distribution and the transport plan based on the KL divergence. An algorithm based on BPALM is further proposed with guaranteed convergence. Theoretical analysis demonstrates the robustness to Huber-$\\epsilon$ contamination, and empirical results demonstrate the effectiveness in partial correspondence with the presence of outliers.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow, with theoretically and empirically sounded analysis.\n- The paper tackles the crucial robustness problem in GW distances, which could potentially benefit many OT-related research.\n- Experiment results demonstrate the effectiveness of RGW on the partial correspondence task, a task that is hard to addressed by most OT methods.\n\nWEAKNESSES:\n- A section on related works is **strongly encouraged**. It would be beneficial to discuss the relation/superiority of the proposed RGW compared with other relaxed GW methods, e.g., UGW and srGW.\n- Some modifications in the experiment part are encouraged for better presentation. See details in the “Questions” section.\n\nQUESTIONS:\n- RGW imposes an extra relaxation on the marginal distribution, i.e., $KL(\\alpha|\\mu)<\\rho$. Comparing with UGW (UGW is equivalent to $GW_{0,0}$), can you illustrate the benefit/intuition of imposing this extra relaxation?\n- For Theorem 2.3, it is possible to have a similar result when both marginals $\\mu, \\nu$ are contaminated?\n- Line 161-162: the statement “By appropriately selecting …  closely approximates the true GW distance (without outliers).” is inaccurate. Setting the second term in Theorem 2.3 only indicates achieving a tighter upper bound but does not necessarily indicates a better approximation.\n- For eq. (4-6), why the proximal operator is $KL(\\pi|\\pi^k)$ in Eq.(4) but $KL(\\alpha^k|\\alpha)$ in Eq.(5)? Since KL divergence is not symmetric, the authors should justify their choices here.\n- Experiments:\n  - It is encouraged to show an empirical evaluation on the convergence of the proposed BPALM.\n  - A key property of RGW is being robust to outlier. In the subgraph alignment part, will RGW achieve more significant improvements when the query graph is not an exact subgraph, but rather a subgraph with noises (e.g., pseudo nodes/edges) added or deleted?\n  - For table 1, why different metrics are used for different datasets?\n  - For figure 5, It would be good to show the results with $\\rho=0$. i.e., no relaxation on the marginal, to justify the effect of marginal relaxation.\n- Typo: the abbreviation for fused Gromov-Wassertein should be FGW instead of FW.\n\nLIMITATIONS:\nThe authors properly addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

comments: SUMMARY:
The authors propose a robust version of the Gromov Wasserstein (RGW) distance and theoretically bound their RGW distance applied to noisy distances in terms of the GW distance between the clean distributions and an error term depending on the choice of parameters in RGW and the distance between the clean distribution and the outlier distribution. The authors develop an algorithm for computing the RGW distance using Sinkhorn’s algorithm and Newton’s method.

STRENGTHS:
The theory provides guidance on how to choose parameters in order to get a good approximation (at least an upper bound) of the RGW distance in terms of the GW between clean distributions.

WEAKNESSES:
To get the upper bound (mentioned above) one needs to know the GW distance between the clean distributions.

QUESTIONS:
(1) How can one estimate the GW distance between clean distribution?
(2) Can Theorem 2.3 be extended to where \nu also has an outlier distribution?
(3) An obvious question, but can one hope to have a lower bound on the RGW distance? Numerically, I would have thought this would be fairly simple to investigate (for example in the context of the examples in Section 4.1) by computing GW of the clean distributions and then varying \rho in the RGW to see how close the upper bound in Theorem 2.3 comes to RGW.

LIMITATIONS:
The authors close the paper by discussing the heavy computational cost of the method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new formulation of unbalanced Gromov-Wasserstein (GW), called Robust Gromov-Wasserstein (RGW). Instead of controlling the deviations of the OT marginals from the prior marginal distributions $\\mu$ and $\\nu$ directly via quadratic phi-divergences like UGW, RGW proposes to use linear phi-divergences (KL) operating on the intermediate learned marginals $\\alpha$ and $\\beta$ which are constrained to be relatively close to $\\mu$ and $\\nu$ w.r.t the KL. It is shown that this new formulation is theoretically better at handling outliers than the UGW under the assumption of Huber's contamination model.\n\nThey then suggest using a Bregman proximal alternating linearisation minimisation (BPALM) algorithm to solve this problem, which is shown to converge asymptotically to a critical point. Finally, the authors show empirically on a synthetic dataset that their method is more robust to outliers, and that it leads to better performance than many competitors on subgraph alignments and partial shape correspondences.\n\nSTRENGTHS:\n-\tOverall I find the paper well-written and I didn’t find any mistake in the proofs.\n-\tRGW is novel in the Gromov-Wasserstein literature, even if inspired from prior work on unbalanced linear OT as mentioned by authors. The theoretical results on robustness are compeling. The solver proposed by authors seem clearly competitive compared to the one for UGW, while its use is also theoretically supported.\n-\tRGW shows better results than competitors on partial shape matching of both synthetic and real-world data.\n-\tRGW also shows better results than competitors on subgraph alignment problems on many real-world graph datasets. Authors empirical study considers subgraphs of various proportions sampled from given input (full) graphs. RGW shows consistent improvements compared to competitors along these various proportions. Moreover authors show that RGW performances are rather consistent within considered sets of validated hyperparameters.\n\nWEAKNESSES:\n- 1. Positioning w.r.t the unbalanced OT literature: \n     - a) *[addressed by authors]* The extension to GW of the work on Wasserstein in [Balaji & al, 2020] should be more explicit. As the latter does not consider regularizing their loss function, you should justify why you do so in your GW setting.\n\n     - b) *[addressed by authors]* The recent paper of [A] provides in a sense a primal point of view of robust GW, It would be good to discuss it and to benchmark it in your subgraph alignment benchmark.\n\n- 2. Robustness guarantees :\n     - a) *[addressed by authors]* Missing proof: Could you provide the proof for UGW in Remark 2.4 ?\n \n     - b) *[addressed by authors]* Tightness: Could you empirically analyze the tightness of both bounds for instance while e.g validating algorithm initializations to get better estimates of various GW terms ? \n\n     - c) *[addressed by authors]* Positioning with recent analysis: Theorem 1 in [Tran & al, 2023] also provides an affine relation of the form: noisy UCOOT <= a * clean UCOOT + b. I believe that this theorem can be easily extended to UGW so could you discuss the relation of this theorem to your bounds ?\n \n- 3. Benchmark: \n     - a) *[addressed by authors]* RGW depends on a lot of hyperparameters. Authors show that their performances were rather consistent for the hyperparameters tau and rho which are involved in the RGW program however it seems that the sensitivity analysis always includes an intensive validation of the BPALM step sizes hyperparameters.  i) Could you clarify if shown results in Section 4.2 always includes this validation of BPALM hyperparameters ? ii) Could you provide a sensitivity analysis to these hyperparameters ?\n\n     - b) *[partially addressed by authors using normalized degrees]* Sensitivity to input graph distribution:  In the context of graph representation learning, an open question relates to a good choice of input graph distribution for tasks like graph partitioning [Xu & al, 2019a ; Chowdhury & al, 2021; Vincent-Cuaz & al , 2022]. Uniform distribution is the common simple choice which seems to be a good enough choice, but power-law distributions based on normalized degrees clearly allow better partitioning performances. Could you integrate these kind of considerations in your benchmark for subgraph alignment ? That would also provide insights on the robustness of RGW to these hyperparameters inherent to all transport problems.\n\n\n[A] Liu, Weijie, et al. \"Robust Graph Dictionary Learning.\" The Eleventh International Conference on Learning Representations. 2022.\n\nNB: the citation model seems to be the wrong one for Neurips, currently names and dates instead of simple numbers.\n\n*Update after rebuttal : The authors have addressed most of my concerns through their rebuttals and discussions. Considering that the authors have undertaken to amend the paper and supplementary material accordingly, I increase my grade from 5 (borderline accept) to 7 (accept).*\n\nQUESTIONS:\nI refer the authors to the weaknesses' list above for suggestions and questions that I encourage them to discuss. A last suggestion:\n- I believe both discussions and proofs for the Newton algorithm could be simplified simply observing that $p$ is strictly convex on the positive orthant.\n\nLIMITATIONS:\nI think discussing the weaknesses I've mentioned and answering my questions detailed above would help identify and work around the other limitations of their current work. If the authors manage to address these, I will gladly increase my rating. \nThis work has no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new variant for unbalanced Gromov-Wasserstein with Kullback-Leibler (KL) divergence for marginal relaxation (Sejourne et al., 2021) by leveraging the outlier-robust approach (Mukherjee et al., 2021). The authors propose an algorithm approach by using Bregman proximal alternating linearization. The authors illustrate the advantages of the proposed method on several experiments.\n\nSTRENGTHS:\n+ The authors propose a new variant for Unbalanced GW with KL divergence for marginal relaxation (Sejourne et al., 2021) by extending the outlier-robust approach (Mukherjee et al., 2021).\n+ The authors propose to use the Bregman Proximal Alternating Linearized Method to optimize the proposed outlier-robust GW.\n+ The authors empirically demonstrate the advantages of the proposed approach.\n\nWEAKNESSES:\n+ The authors leverage the outlier-robust approach (Mukherjee et al., 2021) for unbalanced GW. The proposed method is essentially a new variant of unbalanced GW, but somewhat expected results.\n+ Some claims and experimental results are needed to elaborate with more details (see the Questions part)\n\nQUESTIONS:\nThe ideas of the proposed method are clear. It is easy to follow the presentation. However, some parts are needed to elaborate with more details to clarify the claims and the contributions.\n+ The authors should discuss the relation between the proposed outlier-robust GW with the approach in (Sejourne et al., 2021) for unbalanced GW and (Mukherjee et al., 2021) for outlier-robust approach.\n+ As in Definition 2.2, the authors propose to use KL-divergence instead of quadratic KL-divergence (as in Sejourne et al., 2022) due to non-convexity. However, the problem (1) is non-convex. It is not clear the advantages of using KL-divergence over quadratic KL-divergence. Could the authors elaborate more details on this point? Additionally, as  results in Theorem 3.3, could the authors comment the gain of such choice?\n+ The quadratic KL-divergence helps to maintain the homogeneity of GW while KL-divergence will result in non-homogeneous unbalanced GW (especially when input measures have small or large total mass). Could the authors comment on the choice of KL-divergence over the quadratic KL-divergence?\n+ For Theorem 2.3, is the bound tight? Additionally, Theorem 2.3 only address one of the input measures is corrupted, but not two input measures may be corrupted as in problem (1). Is there any limitation for this result? It is better if the authors extend this result for the case as considered in problem (1).\n+ As in line 161-162, Theorem 2.3 gives the upper bound only, it is not clear why “robust GW value that closely approximates the true GW distance (without outliers)”.\n+ It seems that the Proposition 3.2 is a standard optimization result about the unique solution for the root of a monotonic function?\n+ For results in Figures 2 and 3, how to choose the hyperparameters? Do the hyperparameters affect the results?\n+ Could the authors clarify the stopping condition of algorithms for the reported time? (since the problem is non-convex, are any algorithms sensitive to the initialization?)\n+ As in line 285-288, could the authors elaborate how to choose the best result for those hyperparameters?\n+ For results in Figure 5, it is surprising that the hyperparameters have negligible effects on accuracy when one maintains some fixed ratio? However, it is not clear how one can choose a suitable ratio? I suggest that it seems better to extend the range of $\\rho$ and $\\tau$ for results in Figure 5. Could the authors explain results in Figure 5 with more details? \n\nMinor point:\n+ It is better to add the standard deviation for results in Table 1, Figure 4?\n\n---\n\nThank you for the rebuttal.\n\nLIMITATIONS:\nThe authors have discussed limitations of their work. However, there is no discussion about the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}