Exploiting Edge Features in Graph-based Learning with Fused Network Gromov-Wasserstein Distance
Authors: Junjie Yang
Matthieu Labeau
Florence d'Alché-Buc
Venue: Accepted by TMLR ()

Abstract: Pairwise comparison of graphs is key to many applications in Machine Learning ranging from clustering, kernel-based classification/regression and more recently supervised graph prediction. Distances between graphs usually rely on informative representations of these structured objects such as bag of substructures or other graph embeddings. A recently popular solution consists in representing graphs as metric measure spaces, allowing to successfully leverage Optimal Transport, which provides meaningful distances allowing to compare them, namely the Gromov-Wasserstein distance and its variant the Fused Gromov-Wasserstein that applies on node attributed graphs. However, this family of distances overlooks edge attributes, which are essential for many structured objects. In this work, we introduce an extension of the Fused Gromov-Wasserstein distance for comparing graphs whose both nodes and edges have features. We propose novel algorithms for distance and barycenter computation. We present a range of studies that illustrate the properties of the proposed distance and empirically demonstrate its effectiveness in supervised graph prediction tasks.

Published in Transactions on Machine Learning Research (07/2024)
Exploiting Edge Features in Graph-based Learning with Fused Network Gromov-Wasserstein Distance
Junjie Yang LTCI, Télécom Poris Paris, France
junfie. minffe.gnng@tellcommpprii
Matthicu Labeau LTCI, Télcco Paris IP Paris, France
muttheev.labamm&@tccccoo--op
Florence d' d'Alché-Bue LTCI, Télccom Paris IP Paris, France
forence.dalchene Babe
Reviewed on OpenReview: https: /ttp:r///ppeneevteeee net/ forum? id- SuCNt
Abstract
Pairwise comparison of graphs is key to many applications in Machine Learning ranging from clustering, kernel-based classification /regressioonn and more recently supervised graph prediction. Distances between graphs usually rely on informative representations of these structured objects such as bag of substructures or other graph embeddings. recently popu- lar solution consists in representing graphs metric measure spaces, allowing to successfully leverage Optimal Transport, which provides meaningful distances allowing compare them, namely the Soomoo-Wassersterr distance and its variant the Fused Gromov-Wassersile Wasserstein that applies on node attributed graphs. However, this family of distances overlooks edge attributes, which are essential for many structured objects. In this work, we introduce Wasserstein distance for comparing graphs whose both
an extension of the Fused nodes and edges have features. We propose novel algorithms for distance and barycenter computation. We present range of studies that illustrate the properties of the proposed distance and empirically demonstrate its effectiveness in supervised graph prediction tasks.
1 Introduction
Optimal Transport (OT) (Villani 2009) has witnessed growing attention in Machine Learning wherein it has become a key tool to compare probability distributions (Shen et. al. 2018). In particular, optimal transport distances exhibit properties, such as their ability to take into account, via ground metrics, the geometry of the sample space and their differentiability, that make them especially suitable as loss functions (see for instance Wasserstein Auto-encoders Tolstikhin et al. 2018) or Wasserstein Generative Adversar- ial Neural Networks (Arjovsky al. (017)):
Particularly of interest here are novel insights that recent benefiting a series of tasks, e.g. graph classification (Vayer dictionary learning (Vincent-Cuaz et al. 2021) or supervised Based on the representation of graph as metric measure space, where the nodes of the graph are considered as the support of the probability measure, OT provides a natural way to compule a meaningful distance between graphs, such as the Gromov- Wassoe-seissnnniinn (GW) distance [Mémoli 2011 Sturm 2012). The asymmetric structure directed graphs has led to the general- ization of GW via the Network Cromov-Wassorstoil (NGW) distance (Chowdhury & Mémoli] 2019) while Gromov-Wassersteh
works have brought on graph-based learning,
et al. 2019), graph matching (Xu et al.
2019
graph prediction (Brogat-Motte et al. 2022)
the Fused Gromov- Wasserstein (FGW) distance (Vayer al.) 2019) was proposed to extend GW node- labeled graphs. Recently, Barbe et al. (0021) proposed the Diffused Gromov Wasserstein (DFGW) distance

---

Published in Transactions on Machine Learning Research (07 (07/2024)
GW
FGW
NGW
(NG
FbGW(b)
FNGW(c)
Figure 1: Different types of graphs represented by the GW-based distances
to smooth the node features along the graph structure through a heat diffusion operator which uses the Laplacian kernel. Overall, this new class of distances enjoys useful metric and geodesic properties, allowing for example to derive the minimum path between two structured objects (Sturm 2012 Vayer et al. 2020
On the other hand, many machine learning applications require dealing with graphs with complex edge features, such as abstract meaning representation (AMR) (Banarescu et al. 2013) in natural language pro- cessing, scene graphs (J. Johnson et al., 2015) in computer vision, or molecules chemistry (Irwin ot al. 2012 Dührkop et al., 2015). For instance, when considering molecule identification tasks, leveraging the distance as a loss does not allow reach the same level of performance than kernel that considers more complex molecule representations. Dynamic graphs may also require more sophisticated modcling, as they temporal odge features which continuously evolve over time (Kazemi et al. 2020). There have been several allempts in the literature to oblaín meaningful graph representations by including edge features: the two dominant solutions are to use graph kernels and Graph Neural Networks
Fused
(GNNs). Among graph kernels, the Neighborhood Subgraph Pairwise Distance Kernel (NSPDK) (Costa &
Grave 2010 takes edge labels into consideration to build graph-invariant encodings of subgraphs while more
involved graph kernels leverage bags of subgraphs (Grenier et al. 2015). The GNNs belonging to the Message Passing Neural Networks (MPNNs) framework (Simonovsky & Komodakis 2017 Gilmer et al.) 2017 Fey
et al. 2018 Corso et al., 2020) may incorporate directly edge features via their aggregation procedure. Some
iirentioonbbsed variants (Veličković et al. 2018 Shi et al. 2021 Brody et al. 2022 leverage edge features
to compute the attention weights.
Building on recent computational and theoretical results on GW distances, this paper proposes to represent the edge features of a graph by equipping the original measure space with an additional binary function whose codomain falls in a metric space. This gencralization allows us to ficxibly include edge features into the computation of novel fused GW distance while keeping its desirable lopological properties. As consequence, barycenters of edge and node featured graphs can also be computed. These new tools especially unlocks more accurate graph modeling for wide array of tasks. In this work, we first propose to check learning where, for example, the distance is
the relevance of the novel FNGW distance in distance- used to define a kernel in the input space to solve graph classífication tasks. We then target supervised graph prediction applications where the FNGW distance used as a loss function and the predictive model provides prediction under the form of FNGW barycenter.
Summary of contributions:
We propose new OT-based distance (FNGW), which is generalization of both the Network Gromov- Wasserstein Distance and the Fused Gromov- Wasserstein Distance to edge-featured graphs.
We derive an algorithm for the computation of the FNGW distance in the discrete case, along with procedures for barycenter computation, unlocking supervised edge-featured graph prediction.
We present extensive experiments on node and edge featured graph classification tasks that showcase the relevance of the FNGW distance as a means for input graph comparison.
We tackle supervised prediction of graphs where the loss function is the FNGW distance. On this task, we consider two real-world problems with and without candidate test sets and extend a method basod on graph barycenters, by scaling large datasets. We observe a significant increase in performance compared to the of FGW distance.
2

---

Published in Transactions on Machine Learning Research (07 (07/2024)
For two probability measures μ E Prob(Xo), Prob(X1) where Xo and X₁ are both polish spaces, we note [T v) the set of all couplings of and v, iee, the set of probability measures
Notations:
the product space Xo x X1 satisfying (A), = μ²(o)) and T(Xo, A1) = v(A1) for all Ao E B(Xo), A E [ where (.) denotes the Borel o-algebra. Σn = [h E (R+)" Σi= h; = 1 is the simplex histogram with n bins. In the case where both y and v are discrete, i.e. we can write p= === a:627 and v = دド= byday with a E En and b E Em; we note I[ b) the set of matrices л E R= satisfying E, Ti. = by and Σ, Tij = a despite the abuse of the notation. Here 8 denotes the Dirac measure. We use # to denote the pushforward operator on measures. We note Xn the n-mode tensor-matrix product. Given a tensor X E R4x12=xXxx and & matrix A E IR)xl, X Xn A gives a lensor of shape I x ... x In-1 x J x la+1 X ... x IN where (X Xn A) (,-1,,,,n+1," ,iN) = I ((i, in..... .ix A(j,)( in). We note INxM tensor of shape N x N x M with INxM (m1, R2, m) = Sning where 8 is the Kronecker delta function. At denotes the Penrose pseudoinverse of a matrix A.
2 Fused Network Gromov-Wasserstein Distance and Barycenter
In this section, we first give the discrete definition of the Fused Network Gromov- Wassersttnn Distance, designed for the representation of edge-featured graphs. We present our method for computing the distance, along with its constraints. We also introduce a general form of our distance. We end with an algorithm for computing FNGW-based barycenter. Proofs propositions and theorems are given in Appendix A
2.1 Fused Network Gromov-Wasserstei Distance and Computation
Along the paper, we use the following definition for describing node and edge featured graph.
Definition 2.1 (Node and Edge Featured Graph). A node and edge featured graph is quadruple of the form (F, E, p) where F' E Imm is tuple of points valued in metric space (V, dv), A E RR is a real-valued matrix, E E Qmmx is a tuple of points valued in metric space (S,, da) and p E Em is simplex histogram. We denote G as a set of such quadruples.
We illustrate this definition with the following example.
Example 2.2 (Node and Edge Featured Graph). Consider the graph (a) in Figure a an illustration. Here, the space V = yellow) can be chosen to be the node-color space, while Ω {solid, dashed, non-edge) can be designated as the edge-type space. The elements in both spaces are encoded using one-hot encoding scheme equipped with Euclidean distance dy and dg. Consequently, we can represent the graph with:
11 0
0,,0,11
[0,1,0]
[1,0,0]
0 1 0 p= 0000
1/3
1
E=
1,,1,0
[0,0,
[0.0.1 1
A =
1/3
[1,0,0]
00,0,1
[0,0, 1]
1/3
where F(1) E R² represents the color type of the node i, E(i,)) J) E R3 denotes the edge type between nodes and j, A represents the shortest path matrix of the graph and p represents the uniform weights on the nodes. Graph (c) in Figure ! is another example where edge features are vector-valued, giving Ω = IR, where the first dimension indicates the length of the edge and the second dimension indicates its width.
Now, relying on the Definition 2.1 we introduce the Fused Network Gromov-Wassereee Wasserstein (FNGW) distance dedicated to the comparison of node and edge featured graphs D
Definition 2.3 (FNGW Distance, Discrete Case). Given (F, A, E, p) of size m, Ō= (F) A, p) of size m corresponding to two tuples of G, and tradc-off parameters (a. () E [0, 13, the Fused Network Gromov- Wasserstein distance between them for (p. g) E [1, oc] is wrillen as
FNGW (NGW.,.g..(9..)
Ea,3. A, F., (F, A, min rEll(p.P)
1 A very recent paper (Kawano et al. 2024 introduced similar notion under the name Multidimensiooall Fused Gromov- Wasserstein discrepancy.
3

---

Published in Transactions on Machine Learning Research (07 (024)
with
Σ k.j.k.
adn ((i,)), 3)) A(j.1)1
+ (1 - a B)dv (F(1), (()) TK.
(2)
Example 2.4 (FNGW Distance). By using the graph representation procedure described in Example 2.2 the FNGW distance between the graph (a) and the graph (b) illustrated in Figure is 0.296 when a 3 B = 3: p= and q = 2, while the FGW distance between them O.
Computing the FNGW Distance: Let us first examine the computational complexity of our distance in this discrete case. We define the 4-dimensional tensors J(A, and I( F) as follows:
J. A) = [A(i,)) A( (E, = da (Eli, k), k), E(,
(3)
and the cost matrix ((F, F):
(F,(F,F)= du (F(I,
(4)
Choosing p= 1, we can rewrite
Ba ((0---)))M(E))) + @ o
(5)
Here, @ operator gives a matrix of the form (J E Ji. Following Peyré et al. (016)) the
term [AA A) & π can be computed efficiently when q 2. The computation of LE,, E) X π similarly non-trivial, requiring O(m2m²T) operations, where T is the number of operations necessary to compute the distance dg (E(i,)) k), E(j, I))- However, by choosing again an appropriate metric space (9, dg) and q, its complexity can be reduced via the application of the tensor-matrix multiplication trick from Proposition 1 of Peyré el. al. 20066)
Proposition 2.5. When Ω = RT with its associated metric da(a, b) = ||a - bllRT and = 2, the term L(F, F) & π becomes
( + +..) - 2 Σ E[(TTE11
(6)
R - Rixx is cxpressed by h( h(F),, = f1 and the matrix (i, (.i. for any i,j.t. 4. can hence be computed with the complezity 0mm2m +mm²T).
where g RmxmxT Rmxm is expressed as g(E)..) (l,, h
Hence, with focus on computational efficiency, we have elected to operate within the metric space Ω defined in Proposition [.5 with p 1,4 2, for the applications to graphs targeted in this work. In practice, calculating the FNGW distance amounts to solving non-convex constrained quadratic optimization problem. Following Vayer et al. (2019), we use the Conditional Gradient Descent (CGD) (i.c, the Frank-Wold Wolle algorithmn.. The complete a algorithm is given in Algorithml [ to the transport plan n (k-1) having the following form:
with the gradient with respect
G = (1 a a (F) 23J @ (k-1))
2a, E) & (k-1)
General form: We now introduce the general definition of the FNGW distance, which extends both the NGW-distance (Chowdhury & Mémoli 2019 and the FGW-distance (Vayer et al., 2020). We propose experiments illustrating this generalization in Appendix D.3 and Appendix D.4
Definition 2.6 (FNGW Distance, General Form). Let G be the set of tuples of the form (X, Vx, xx, where X polish space, Φx - V is bounded continuous measurable function from a metric space (V, de) 6x X x X IR bounded continuous measurable function, wx X x X + is
4

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm Computation of the FNGW Distance by CGD
input: g= E, p): 9= (,,,A,E,,, and trade-off parameters (a; 3) init: x (0) = pp' RRmx.. x
for == l,..,, K do
Calculate gradient: V= ((F, (F, (F,A,)) n(hh n(1-1))
Solve the optimization problem with an OT solver: x(k-1) E arg aig (G,
Update the optimal plan: =(k) (1 (1- (x)) + ( (k-11 with 7(h) E (0, 1) given by line-search
algorithm (See details in Appendix B). end for
Calculate the distance: FNGW, a. ((0 (a,s([F,A,,)) (F, C. ((K)
output: FNGW (g, 9) and x(K)
bounded continuous measurable function from X to a metric space (Ω, dn) and px is a fully supported Borel probability measure. Given two tuples 9x = (X, xxx gr = (Y, th,, wy, from g and trade-off parameters (a, E [0, 1]2, the Fused Network Cromov-Wasserstedn Distance between 9x and gy is defined for any (p. 4) E [1, oo] as follows:
min FNGW, (.x. ren(xx.a))
(8)
with
C..8apllx.r = Uxwr/
xxx 111-001-- a- B)dv (txx (xxx(x),
(xda(wx(xx wr 3lxxx(x, -pr (t :1)))
(9)
The particular form of the FNGW distance in Def. 2.3 aligns with the scenario in which we assume that is a finite set of size m and μx = m P,On.- As a consequence, F is the result of evaluation of thx on every point of X, while A and E are outcomes of evaluating fx and wx on every pair of points in X2, Note that yp are not necessarily symmetric functions, which corresponds to the setting of the NGW (Chowdhury & Mémoli, 2019), as opposed to the GW distance. We do not impose that constraint on the newly introduced functions either. In the context of edge-featured graphs, v serves as the node labeling function, 9x represents the function that calculates the shortest path between two nodes, and wx acts as the edge labeling function.
The following theorem states the existence of the optimal coupling, so that the FNGW distance is well defined.
Theorem 2.7 (Optimal Coupling). Given 9x (X, XXx gy (Y,b,, py, wy,,y) py), for any (p.g) E [c] and (a, [0, 1]2, there erists an optimal coupling l¹ E T(xx, which satisfies (9x = Ea (xx
The following thcorem states the metric properties of FNGW, which allows application a wide range of machine learning algorithms that only require pairwise comparisons.
Thcorem 2.8 (Metric Propertics). The FNGW W distance satisfies the following properties: for all gx
(X, vx,, Sxx, wxw ,xx),, gY = (Y, bryy:yr, and gz = (Z, uzz p2, wZ, from G:
(Positivity)
20
(Symmetry) FNGW INGWa.9...pox..
FNGW a. (gy (gy.9x)
(Equality) FNGW (.GWa.Daapp9.... 0. Moreover, FNGWa.Ba.lee gr) = 0 if and only there is Borel probability space (2) with measurable maps f Z + X and g Z + Y such that
f# = px 1#11 = pAy
(10)
[|(1 - - 3)dx (vx o f. o 9)" wx w + 3) px lxyll =0
(11)
5

---

Published in Transactions on Machine Learning Research (07/2024)
where f wx: Z Z - Ω is the pullbock weight function defined by (2,2') wx(((())) fx 7. x 7. - R is given by (z, 2') (x(((=)) and g is defined similarly.
(Relazed Triangle
Inequality)
FNGW FNGWo.B..y9X. x.
2a 2FNGW a, Bepxx.x
FNGW N.GWa,,....999922.
define a weak isomorphism between the objects of g, while the last property states a relaxed triangle inequality with a factor of 29-1, Consequently, when q= 1, FNGW is a proper metric over 9 endowed with such weak isomorphism as an oquivalence relation.
Equations 10
It should be noted that although FNGW theoretically satisfies the metric properties defined above, its practical guarantees depend on the specific computation algorithm used.
2.2 Fused Network Gromov-Wasserstein Barycenter and Computation
The notion of (weighted) barycenter can be encountered in many approaches in data science. Considering graph data, some recent works have successfully exploited GW-based barycenters in graph clustering (Peyré et al.
2016 Vayer et al. 2019) or in supervised graph prediction (Brogal-Motte et al. 2022). Similarly, our FNGW-based barycenter can be directly applied to the clustering of labeled graphs; we provide an example in Appendix D.4 Additionally, Graph Dictionary Learming (GDL) is feasible with the FNGW barycenter,
where each graph is represented as the barycenter the graphs the dictionary. We develop this algorithm in Appendix B.2 and present an example in Appendix [D.5] In what follows, we give a formal definition of such a barycenter, based on our proposed FNGW distance, and describe an algorithm to compute it.
Definition 2.9 (FNGW Barycenter). Given set { with 9s (F,, A,, E., P;) RT RRm.x:: [m,xxxXTT Em, and a set of weights {Ar)=z such that E, 1, the FNGW Barycenter for pre-defined histogram p 2m is defined follows:
B({A: (9:t..p)
arg min FERmXX .ARRR "t.":/.................... BERmxmX7
AFFNGW a.
(12)
The above optimization problem be reformulated as:
min (. ArE
(F,, A,, m.),
(13)
To obtain the FNGW barycenter, we employ the Block Coordinate Descent (BCD) algorithm, which means that carry out the minimization in Equation 13 iteratively with respect to (m.h. F, A and E. The minimization with respect to E has closed form, which is given in the following proposition:
Proposition 2.10. Optimizing Equation 13 with respect to tensor E has closed-form solution:
E
Σ Ae X2 Ti) X1
(14)
ImxT X2 T
The complete optimization algorithm is detailed in Algorithm 2
We can notice that the optimization procedure preserves an interesting property for the tensor E of the resulting barycenter:
Proposition 2.11. If the set of tensors [Eits satisfies the condition: Wj,,i, E.G.,,t = a E R, then the barycenter E given by Algorithm g also verify the same property.
Remark 2.12. When the set of edge features for the graphs {ohi lies in a simplex space, the above proposition gives us the guarantee that the edge features of their barycenter will also a simplex. For example, when the edge labels of the graphs are represented using onc-hot encoding, the resulting barycenter can be discretized into a true graph by applying a simple argmar operation on the edge features, due to their simplex nature. This can be seen generalization of the thresholding operation the adjacency matrix.

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm 2 Computation of FNGW Barycenter with BCD
imput: (geh, fixed histogram p, trade-off parameter (a. (3)
init: Randomly initialize F F() and A
for == ,,...,K K do
Calculate { with Alg,
T (R) arg min pEUPp)) Ea,o A E(k-1)),
1),, (F,, E,), T
Update E: ((()
Ei
de E X2 mi (k))
X1 x1
(R)
Proposition 2.10
ImxT 2pp Update A: A 1
pp NTT
A,T;; (R)T
> Proposition
Peyré et al.
(0116)
Update F: F(k) = Ei A,diag( )
(R) F;
Equation 8 in Cuturi & Doucet (2014)
end for
output: The barycenter (F(K), , (K) E(K)
orgin
NB0 mode
Venra rode
UHIN RIgA
Figure 2: FNGW barycenter (rightmost) of the graphs obtained by perturbing a random molecule (leftmost)..
(a) Random circle samples
(b) Circle baryeenter
Figure 3: Illustration graph barycenters with the FNGW distance. The color of the node indicates the scalar value of the node feature.
The proofs of Proposition 2.10 and 2.11
be found in Appendix
Example 2.13 (Barycenter Computation). To test the proposed barycenter computation algorithm, we perturb random molocule using the following operations: node addition, node deletion, edge addition, and edge deletion. The resulting graphs are then used to compute the barycenter, successfully retrieving the original molecule before the perturbation, as shown in Figure 2 The FNGW distance between the barycenter (before discretization) and the original graph is 0.0079, with = 31 B = 11 1, and = 2. Another example barycenter computation is shown in Figure We randomly create circle graphs, for which the number of nodes randomly drawn between 10 and 20. Node features are scalars following Sine function variation with additive Gaussian Noise (O, 0.3). There exists two directed edges between each pair of adjacent nodes, the ascending one being colored in blue and the descending one in green. Supplementary edges generated with probability 0.3 between every pair of nodes separated by another node. The toy graphs are therefore node-labeled, edge-labeled, and directed. The samples are shown in Figure Ba To compute the FNGW barycenter, we take A as the adjacency matrix, and encode the presence of colors in the one-hot tensor E. We choose the number of nodes of the barycenter to be 15, and the resulting graphs are shown Figure 3b) We observe that both node and edge features are well preserved in the predicted barycenter, showing the offectiveness of Algorithm
In the following section, we explore the application of the FNGW barycenter to supervised graph prediction.

---

Published in Transactions on Machine Learning Research (07/2024)
3 Supervised Graph Prediction with the FNGW Barycenter
Supervised Graph Prediction task consists in learning to predict an output graph g from given input in input space X. An appealing and original application of Optimal Transport for graphs consists in using a Gromov-Wasserstee distance as a loss in this supervised task (Brogat-Motte et al. 2022).
Let us define g, the set of labeled graphs with maximum number of nodes mmax.
mg S maxx Ae {0, 10m, F = (Fi) F g (F
= (Bg) T Xma p = p= 1m
(15)
where F C R and T C RD are finite node feature and edge feature spaces, and mmax is the upper graph size. Denote Im the relaxed version of set g:
9m (F,, A [0, 1] 1mmxm F (F,) E CCnv(FF)m
E = (B,) E Conv(T) p = m mm
(16)
where Conv(.) denotes the convex hull of the set.
We consider a set of training pairs consisting of inputs and graphs to be predicted {(E,, H- drawn from a fixed but unknown distribution on X' x G. We are interested in the relaxed supervised graph prediction problem, i.e. finding a estimator f: - Gm of the minimizer f of the expected risk R(f) = E, aNG G)] where the FNGWa,a-distan solution this problem, estimator based on surrogale least square regression (Ciliberto et. al.] 2020) proposed by BrogattMoote al. 2022) that expresses barycenter of the output training data weighted by function of the input x;
dcfinition extended to 9m G. We choose, as
f(x) arg min gEG...
(()):ENGWo.fl (g
(17)
with {(x) (K + +XI)= kg,, where K E Ruxn is the Gram matrix of the positive definite kernel & defined on the input space X such that K. I,); K2 = (k(2, (1)..,))....,ke k(x, z,,)) E R", and 0 is a ridge regularization parameter. Note that for a given æ, I E G is obtained by discrctization of f(x).
The proposed estimator relies on the Implicit Loss Embedding (ILE) condition Ciliberto et al 2020), which expresses that an ILE loss can be written as an inner product of feature maps in some well-chosen Hilbert space. The ILE framework justifies the relevance of solving surrogate regression problem in the so-called output feature space and ensures theoretical guarantees for this class of estimators. The next proposition guarantees that the estimator proposed in Equation 17 fits ILE condition.
Proposition 3.1. The FNGW loss admits Implicit Loss Embedding (ILE).
Informally, this implies that our estimator universally consistent and its learning rate is of order n m114 with additional assumptions. The proof is given in Appendix A.6
Sketched ILE for big data regime. To avoid computational issues when scaling-up the number of samples, we propose a sketched version of the previous estimator. Sketching (Woodruff al. 2014) applied kernel approximation (Rudi et al., 2015 Avron et al. 2017 El Ahmad et al. (023)) leverages random projections of the Gram matrix, allowing to work with low-rank matrices and thus reducing compute time. Here, it gives {(x) = KST (SK2 + nASKST)t of where S E Raxx with < is a sketching random matrix.
In order to conduct the prediction process described by Equation 17 we distinguish two possible situations. When a limited set of candidates from a set G. provided, one can resort to computing the score for each candidate. Otherwise, we resort the method described in Alg. 2 We will showease the two situations respectively with the experiments presented in Section 4.2 and Section 4.3
8

---

Published in Transactions on Machine Learning Research (07/ (07/2024)
4 Numerical Experiments
In this section, we conduct series of studies around the proposed distance; then, we assess its relevance in supervisod graph prodiction problems, including the Fingerprint to Molecule task and Metabolite Identifica- tion task. A Python implementation of the algorithms presented for these tasks can be found on Githut We leave additional experiments on clustering for Appendix D
4.1 Study of the FNGW Distance
We begin our investigation by studying the potential benefit brought by our FNGW distance to graph-level supervised classification tasks. Then, we use these tasks to investigate how well the associated loss converges, and the impact of its hyperparameters on performance.
4.1.1 feteure Graph Classification
We represent input graphs to be classified by kernels based on the FNGW-distano;; hence, we choose to use a variety of datasets where both node and edge features are present: Cunciform (Kriege et al.) 2018)
MUTAG (Debnath et al.] 1991; Kriege & Mutzel 2012). PTC-MR (Helma et al.) 2001; Kriege & Mutzel 2012), BZR-MD, CD, DHFR-MD and ER-MD (Sutherland al., 2003 Kriege & Mutzel 2012). All
these datasets were collected by Kersting et al. (2016). Our classifier is a Support Vector Machine (SVM) for
which the kernel matrix K is computed with Kij exp(->PNGW(o,,.) 9,). The kernel defined by FNGW
is indefinite. While there exist in the literature SVM algorithms dedicated to indefinite kernels (sce for instance (Ong et al. 2004 Luss & D' aspremont 2007)), ), in these experiments, we rely on the classic SVM implementation. To compute the FNGW distance, we consider F as the matrix of node features, and we set A to be the matrix of shortest path distance and F. as the tensor of edge features where the non-odge position is attributed to random vector of the same dimension. Details about the node and edge features present in each dataset, as well as details about graph representation for the FNGW compulation are given in Appendix C.1
We choose to use both sarnel-based methods and graph neural networks (GNNs) for our basclines. We include a large set of kernels in our experiments: the Shortest Path Kernel (SPK) Borgwardt & Kriegel 2005). the
Random Walk Kernel (RWK) (Gärtner et al. 2003), the Weisfeler Lehman Kernel (WLK) (Shervashidze
ct al. 2011), 1, the Graphlet Sampling Kernel (GSK) (Shervashidze et al. 2009), the Neighborhood Subgraph Pairwise Distance Kernel (NSPDK) (Costa & Gravel 2010), the Hopper Kernel (HOPPERK) (Feragen
et al. 2013), the Edge Histogram Kernel (FHK), the Vertex-edge Histogram Kernel (VEHK) (Sugiyama & |orgwardt] 2015) and the Propagation Kernel (PROPAK) (Neumann et al. 2016). For the GNNs, we consider the Principal Neighbourhood Aggregation (PNA) (Corso et al. 2020 and the Graph Attention Networks (GAT) (Veličkovć et al. 2018). Lastly, we also include in our experiments a kernel-based classifier induced by the FGW distance, computed with the same matrices F and A. It should be noted that similarly to this last baseline, models don't necessarily use all available features.
We perform nested cross-vali with 50 iterations of the outer CV loop and report our graph classification rosults in Table 1 In Appendix C.1 we provide detailed information on hyperparameter scarch for all methods conducted during the inner CV loop. Our FNGW-based classifier outperforms consistently other graph kernel methods and GaN-based classifiers on most of the datasets. Furthermore, FNGW offers performances similar to FGW on PTC-MR, and significantly improves them on all the others datasets, except for MUTAG. We conjecture that the mutagenic effect on a bacterium (MUTAG) or carcinogenicity on rodents (PTC-MR) of molecules may not be sensitive to their chemical bond type, which is the information encoded in the edge features. But when combined with information about the distance between atoms, these features seem useful for distinguishing between active and inactive molecules (BZR-MD, COX2-MD, DHFR-MD, ER-MD).
httpe://sshuu.cc//mauchhhh/////t con.chhhnhieehh//fgg
9

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: Graph classification performance on real datasets. The best results are highlighted in Bold. The last seven rows show the results for methods leveraging the edge features.
Methods
Cuneiform
MUTAG
PTC-MR
BZR-MD
COX2-MD
DHFR-MD
ER-MD
RWK
73.93 7.73
88.42 7.59
57.43 7.61
71.81
8.48
62.65 = 7.93
67.20 5.67
70.22 6.32
SPK
74.00 7.39
84.00 8.80
57.37 8.16
71.10 6.71
64.90 1 8.43
65.85 5.87
69.78 6.14
GSK
15.63 = 6.84
83.89 = 8.04
51.8
48.13 6.73
44.90 = 6.94
17.45
59.16 6.44
WLK-VH
73.70 1 7.34
88,00 1 7.95
61.71
63.10 6.992
56.71 7.82
66.65 1.5.
66.98 6.199
HOPPERK
74.00 + 7.39
83.89 9.50
54.29
73.81 7.00
61.10 = 7.86
67.60 5.88
70.84
PROPAK
73.26 + 7.49
76.84 9.59
58.57
73.10
58.58 8.45
66.85 6.14
66.44
FGW
86.59 + 7.51
8.34
63.77 9.22
73.61 6.60
64.90 7.43
62.70 + 5.95
72.09 7.14
NSPDK
77.48 = 8.91
88.63 + 5.99
8.33
73.10 = 7.03
60.00 = 8.70
67.30 = 6.46
63.16 7.41
EHK
5.63 3.65
67.
9.76
55.43 1.73
66.97 1 7.34
60.26 7.11
67.35 1 6.35
70.09 7.20
VEHK
74.00 7.39
77.26 9.32
58.97 8.14
72.52 + 7.67
80.06 + 7.09
66.80 = 7.65
75.24 6.93
WLK-VEH
73.48 7.46
82.53 8.90
62.23 7.07
64.00 7.25
58.65 8.48
66.65 5.54
71.20 5.83
PNA
84.15
83.79 8.01
59.94 7.33
71.14 9.60
7.94
6.30
77.24 6.51
GAT
37.19
11.90
75.26 10.38
57.94 = 7.94
70.20 7.46
70.13 = 9.93
69.10 5.80
72.40 1
FNGW
89.41
6.06
83.05 - 8.95
63.94 7.54
75.68 7.01
68.90 - 8.33
73.45 7.28
79.78 5.85
BZR-MD
0.725
0,90
0,700
0.85
Fhow
0.675
0.650
0.4
0,0
0.625
0.4
3-
0.2
6.600
0.2
0,80
6.0
0.0
0.6
0.2
0.4
1.0
0.8
0.4
0.6
0.8
0.8
Figure 4: (Left) Convergence rate of FNGW with the CGD algorithm. (Middle) Classification accuracy on MUTAG w.r.t different values of and B. (Right) Classification accuracy on BZR-MD w.r.t different values of and 3. The optimal values of a and 3 vary across different datasets.
4.1.2 Convergence of the FNGW Distance
The left panel of Figure | shows the convergence the FNGW distance using the conditional gradient descent (CGD) in Algorithm [ on the MUTAG and BZR-MD datasets. Each computation step involves randomly selecting two graphs, with this procedure being repeated 10 times. We can see that during this computation, the FNGW converges in fewer than 10 steps of CGD, on both datasets.
4.1.3 Impact of the Parameters Presented in the FNGW
We explore the influence of parameters and & on the FNGW distance classification performance. The heat maps illustrating test set accuracies for the MUTAG and BZR-MD datasets are presented respectively in the central and right panels of Figure 4 These visuals show that for a, which governs the weighting of edge features, a small value (even approaching zero) yields commendable accuracy on MUTAG. Conversely, it larger a that scems to give optimal results for BZR-MD. These observations reinforce our previous hypothesis: it does not seem useful exploit edge features on MUTAG, which could explain why FNGW lags behind FGW for that dataset.
10

---

Published in Transactions on Machine Learning Research (07 (224)
4.2 Supervised Graph Prediction: Fingerprint to Molecule
In this experiment, we delve into to a novel graph prediction problem. The aim is to predict a molecule from its fingerprint representation: hence, we will use here our FNGW distance as a loss, through Supervised Graph Prediction estimators, as defined in Scction 3 It should be noted that this is a very difficult end-to-end task as it implies to predict new molecules rather than selecting them from a candidate set.
Dataset. To conduct our experiment, we have chosen the widely used QM9 molecule dataset (Ruddigkeit et al. 2012; Ramakrishnan et al., 2014). QM9 is collection of small organic molecules, totaling around 130,000 in number. We use subset of 2,000 molecules as test set. The molecules are kekulized 3 by the chemical software RDKit R accompanied by the removal of hydrogen atoms. After the pre-processing, each molecule contains up to 9 atoms of Carbon, Nitrogen, Oxygen, or Fluorine. There is three types of edges: single, double, and triple bonds. We adopt a fingerprint representation of the molecules as input features; these fingerprints are generated using the Morgan algorithm provided by RDKit, with a specified radius of and a size of 2048. For each molocule, we represent by an atom matrix E Rmx4 and a bond tensor E Rmxmx3 where m is the number of the aloms. Both the alom features and bond features are encoded by one-hot encoding. To apply the FNGW distance on the molecules of QM9, use the uniform measure over the atoms and the parameter / is set to O. We name the resulting dataset Fin2Mol.
Graph Edit Distance and Correlation with FNGW. We utilize Graph Edit Distance (GED) as a metric to assess the effectiveness of graph prediction, given its widespread use in measuring the similarity between pairs of graphs. Given two graphs 9I and g2, the graph edit distance between them is defined by:
GFD(g1,92) = (o ox (g1))99)))
min
c(oo)
(18)
where Ogg1, 92) is the set of all edit paths allowing to transform 91 into graph isomorphic to 92 and c(o) 2 0 is the cost of each elementary edit operation o which is one of vertex substitution, edge substitution, vertex deletion, edge deletion, vertex insertion, and edge insertion. We use the GED implementation provided by the package NetworkX (Hagberg et al. 2008) and the cost of each elementary edit operation is set to one.
In Figure 5 the left panel shows the correlation between GED and FNGW across various values of a. Specifically, we randomly select 1000 pairs of molecules from the QM9 dataset and compute both the GED and the FNGW distance for each pair. Subsequently, we determine the Pearson and Spearman correlation coefficients between the GED and the FNGW distance. The results reveal a strong correlation between the FNGW and the GED, particularly for commonly chosen values of a. This correlation demonstrates well the interest of employing FNGW as loss function for graph prediction: FNGW effectively serves as viable proxy or approximation for GED, which is prohibitively expensive to compute in practice, and is not differentiable.
Choice of the Parameter a. While a, as an hyper-parameter of the FNGW loss, can be selected via cross-vali validatiii,,,,,, our preceding observation prompts us to propose an alternative approach. Indeed, we can use correlation between the GED and the FNGW values resulting from a as signal to select the latter. Subsequently, the a value yielding the strongest correlation with the GED is selected to define the training loss in the graph prediction model. The underlying philosophy of this strategy is straightforward: by ensuring consistence between the training loss and the evaluation metric, we anticipate achieving good performance.
Experimental Settings. We build five training-test splits using different seeds. Each split has 131,885 training samples and 2,000 Lest samples. We choose to compare our ILE-FNGW with ILE-FGW [Brogal- Motte et al. 2022). Additionally, we adapt the NNBary estimator proposed by Brogat- Motte et al. (2022) with FNGW loss to study its effect. NNBary expresscs the predicted graph as the barycenter of several graph templates, which are jointly learned with the coefficients of the templates parameterized by neural network. At prediction time, we provide the true mumber of atoms to the estimators in order to compute the barycenter. For the ILE estimator, we choose the kernel A used to compute the coefficient {(x) to be Gaussian; given the large quantity of data, we use SubSample sketching (Rudi et al. 2015) for the input kernel approximation. For the NNBary estimator, a MLP of one hidden layer is used to encode input features
Kekullize: converts aromatic rings to their Kekule form. httpa:/ http://wwu.rddct wtt///cs//tttttttttttrooo// BaceekiittBooooeeeeee Book.html
4DDKit: Open-source cheminformatics.

---

Published in Transactions on Machine Learning Research (07 /024)
1.4
Pearson 0.61, Spearman 0.63
ED, Pawry
2-
1.0
Pearson 0.73, Spearman
Spcaman
ED, Pewry
-0.8
FNOW distance
2- 0.9
Edit distance
0.6
0.4
3
0.2
0.0
2-
0.0
0.2
0.4
0.6
0.8
1.0
FNGW
Figure 5: (Left) A strong correlation exsits between the FNGW and the GED for commonly chosen values of a. (Right) The performances of ILE-FNGW and the correlation cocfficients between GED and FNGW with varions values of a. The optimal a values for correlation generally lead to favorable performance.
Graph edit distances of different methods on the Fin2Mol test set. The symbol & indicates the result where a chosen based on the correlation coofficient, while other cases, is selected through cross-validation. Best results in Bold.
Table 2:
GED w/o edge feature + GED w/ edge feature
NNBary-FGW NNBary-FNGW
5.000 = 0.140
5.311 0.090
5.756 + 0.073
Sketched ILE-FGW
3.037 + 0.111
Sketched ILE-FNGW
1.449 + 0.034
1.534 + 0.029 1.809 = 0.065
Sketched ILE-FNGW
1.739 = 0.068
True
ILE-NGW
ILE-ENGW
ILE-FGW
5
ILE-FNGW
Graph Edit distance
4-
3
2-
1
Figure 6: (Teft) Quantitative results with respect the number of training points used to compute the pre- diction. Fewer training points are needed for the barycenter with ILE-FNGW while it provides better performance than ILE-PGW. (Right) Qualitative comparison of the predicted QM9 molecules.
into templates weights. The FNGW parameter a, the ridge regularization parameter X, the parameter of the Gaussian kernel 7, the sketching size, the number of points used to compute the barycenter and the dimension of the NNBary MLP are chosen through cross-validation over subset size 500. We provide an additional result on ILE with FNGW, where chosen based on the correlation coefficient with GED, described above. More details about the cross-validation given in Appendix C.2
12

---

Published in Transactions on Machine Learning Research (07/2024)
Table 3: Top-k accuracies on the metabolite identification test set, Best results are in Bold.
Top-1 f
Top-10
Top-20 t
WL kernel
9.8%
29.1%
37.4%
IOKR Fingerprint w/ linear kernel
28.6%
54.5%
59.9%
1OKR Fingerprint w/ gaussian kernel
41.0%
62.0%
67.8%
ILE-FGW diffuus
28.1%
53.6%
59.9%
ILE-ENGW diffuse + Bond stereo
27.7%
55.2%
60.9%
IL.-FNNW diffuse Bond type
34.6%
55.1%
60.0%
ILE-FNGW diffuse Mix
36.2%
58.2%
61.9%
Experimental Results. Our results are presented in Table 2 They clearly indicate that FNGW, as loss function, outperforms FGW with ILE and achieves comparable performance to FGW with NNBary. The results for NNBary can be attributed to the model's limited expressibility. For ILE, we envision two different explanations, which may overlap: firstly, from a lcarning standpoint, it is conceivable that FNGW induces a more meaningful implicit graph embedding than FGW, thereby facilitating the resolution of the surrogate regression problem. Secondly, from an optimization viewpoint, the incorporation of bond type constraints within FNGW aids in computing the barycenters, consoquently resulting in more realistic molccules. Addi- tionally, FNGW's ability to predict the bond types of molecules enables us to evaluate with version of the CED that takes edge features into account. The final row of Table 2 highlights that selecting a using the best Prarson/Spearman cross-validation. Additionally, as depicted in the right panel of Figure the optimal a values for correlation generally lead to favorable performance of predictions. Finally, the left panel of Figure G shows the influence of varying number of training points used in the barycenter computation on the performance of ILE It is evident that increasing the size of training data generally enhances model performance. However, beyond threshold of 15 training points, the observed improvement becomes less significant. A visual compari- son showcasing predicted graphs generated by various methods is presented in the right panel of Figure 0 The first two examples demonstrate how the FNGW method enhances predictions related to atom types and molecule structure, while the final example validates its efficacy concerning the optimizing during the barycenter computation. See Appendix D.6 for more examples of prediction.
correlation coefficient yields comparable performance to what is achieved through
4.3 Supervised Graph Prediction: Metabolite Identification
In this last experiment, our task is again to predict a molecule; however, this application differs from the first one, as a candidate set Ge is given at inference time.
the Metabolite Identification dataset processed by Dührkop et al. (2013). The learning algorithm is expected to predict the metabolites given a tandem mass spectra. For each spectra, set of possible metabolites of the same chemical formula is provided. The candidate sets were built with molecular structures from PubChem. The dataset is split into a training set of size n = 3000 and a test set of size n4eat = 1022. More details about the dataset can be found Appendix C.3.
Datasct. As in Brouard et al. (2016a): Brogat-Motte et al. (0022)
Experimental Settings. In order to represent the metabolites, we encode their adjacency into A and the atom types one-hot vectors into F. Following Brogat-Motte et al. (2022). use the matrix diffused by the normalized Laplacian of the adjacency matrix: Fai -LappA)F F. To obtain E, use three configurations: the chemical bond type, the chemical bond stercochemistry (which are embedded through one-hot encodings) and the concalenation of both (Mir). The input representation is obtained through a probability product kernel (Heinonen et al. 2012) on the input mass spectra, which has been shown effective on this problem (Brouard et al.] 2016a). Due to the computational cost, during prediction, only the training samples with the greatest weights {(r); are taken into account, rather than all the samples, described in Equation 17 Details about the hyperparameter choices given in Appendix C.3
13

---

Published in Transactions on Machine Learning Research (07/2024)
Experimental Results. We measure the performance of various models Metabolite Identification via Top-k accuracy with & E (1. 10, 20). The results are presented in Table 3 Results for the WT. kernel and LOKR (Brouard et al. 2016b) are taken from Brogat-Motte et al. (2022). We observe that with mixed edge features, FNGW significantly outperforms FGW. Our method approaches the performance of fingerprint with a Gaussian kernel, which uses an expert-derived molecular representation. This results further confirms that using a more complete representation of the output space, able exploit more structure information, is crucial for graph prediction.
5 Conclusion
We propose a novel Optimal Transport distance for pairwise graph comparison in the presence of edge features, unlocking many applications where this information is available and relevant. This distance built on the Network Gromov-Wasee Wasscrstein and Fused Gromov- Wassov-Wassrrtin distances and inherits similar geometric properties; to apply it on graph data, we devise algorithms to compute both the novel distance and the associated barycenter in the discrete case. Comprehensive experimentation on our proposed distance gives a tool for graph representation, and empirical evaluation
deeper understanding of the behavior of FNGW real-world datasets shows that the use of this distance as a loss function yields significant. improvements in terms of performance in supervised graph prediction tasks. Future works will be dedicated to scale-up both the distance computation and the barycenter computation algorithms to target very large graph datasets. We also plan to investigate propagating edge information to nodes and reverse.
Acknowledgments
The authors are grateful to Luc Brogat-Motte for the insightful discussions at the beginning of the project They also thank reviewers for their comments and suggestions. The first author supported by PhD grant provided by Hi! PARIS Center.
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Networks. In Pro- ceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214 223. PMLR, August 2017.
Haim Avron, Kenneth Clarkson, and David P Woodruff. Faster kernel ridge regression using sketching and preconditioning. SIAM Journal on Motriz Anolysis and Applications, 38(4):1116 1111-118111 2017.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. Abstract Meaning Representation for Sembanking, In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pp. 178-186. Association for Computational Linguisties, August 2013.
Amélie Barbe, Marc Sebban, Paulo Gonçalves, Pierre Borgnat, and Rémi Gribonval. Graph Diffusion Wasserstein Distances. In Machine Learning and Knowledge Discovery in Databases, pp. 577-592. Springer International Publishing, 2021.
Karsten M. Borgwardt and Hans-Pete Peter Kriegel. Shortest- Path Kernels on Graphs. In Proceedings of the 5th JEEE International Conference on Data Mining, pp. 74-81. IEEE Computer Society, November 2005.
Shaked Brody, Uri Alon, and Eran Yahav. How Attentive are Graph Attention Networks? In International Conference on Learning Representations, 2022.
Luc Brogat-Motte, Rémi Flamary, Celine Brouard, Juho Rousu, and Florence d' d'Alché Buc. Learning to Predict Graphs with Fused Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2321- 2335. PMLR, July 2022.
Barycenters. In Proceedings of the 39th International
14

---

Published in Transactions on Machine Learning Research (07 (224))
Céline Brouard, Huibin Shen, Kai Dührkop, Florence d' d'Alché Buc, Sebastian Böcker, and Juho Rousu. Fast Metabolite Identification with Input Output Kernel Regression. Bioinformalics, 32(12):128 136, 2016a.
Céline Brouard, Marie Szafranski, and Florence d' Alché Buc. Input Output Kernel Regression: Supervised and Semi-Supervised Structured Output Prediction with Operator- Valued Kernels. Journal of Machine Learning Research, 17(176):1 48, 2016b.
Samir Chowdhury and Facundo Mémoli. The Gromov Wasserstein Distance Between Networks and Stable Network Invariants. Information and Inference: A Journal of the IMA, 8(4):757-787, 2019.
Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A General Framework for Consistent Structured Prediction with Implicit Loss Embeddings. Journal of Machine Learning Research, 21(98):1 67, 2020.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal Neighbourhood Aggregation for Graph Nets. In Advances in Neural Information Processing Systems, volume 33, pp. 13260-13271. Curran Associates, Inc., 2020.
Fabrizio Costa and Kurt De Grave. Fast Neighborhood Subgraph Pairwise Distance Kernel. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pp. 255 262. Omnipress, 2010.
Marco Cuturi and Arnaud Doucet. Fast Computation of Wasserstein Barycenters. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 685 693. PMLR, June 2014.
Asim Kumar Debnath, Rosa L. Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity Relationship of Mutagenic Aromatic and Heteroaromatic Nitro Compounds. Correlation with Molecular Orbital Energies and Hydrophobicity. Journal of medicinal chemistry, 34(2):786 797, 1991.
Kai Dührkop, Huibin Shen, Marvin Meusel, Juho Rousu, and Sebastian Böcker. Searching Molecular Struc- ture Databases with Tandem Mass Spectra using CSI: FingerID. Proceedings of the National Academy of Sciences, 112(41):12580-12555 2015.
Tamim El Ahmad, Pierre Laforgue, and Florence d' Alchh Buc. Fast Kernel Methods for Generic Lipschitz Losses via. \p\-Sparsified Sketchos. Transactions on Machine Learning Research, 2023.
Aasa Feragen, Niklas Kasenburg, Jens Petersen, Marleen de Bruijne, and Karsten Borgwardt. Scalable Kernels for Graphs with Contimous Attributes. In Proceedings of the 26th International Conference on Neural Information, Processing Syslems, volume of NIPS'13, pp. 216 224. Curran Associates Inc., 2013.
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. SplineCNN: Fast Geometric Deep Learning With Contimuous B-Spline Kernels. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. Tn Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1263 1272. PMLR, August 2017.
Pierre-Anthowd Grenier, Luc Brun, and Didier Villemin. From Bags to Graphs of Stereo Subgraphs in Order to Predict Molecule's Properties. In Graph-Based Representations in Pattern Recognition: 10th IAPR-TCA International Workshop, pp. 305 304- Springer, 2015.
Thomas Gärtner, Peter Flach, and Stefan Wrobel. On Graph Kernels: Hardness Results and Efficient Alternatives. In Learning Theory and Kernel Machines, pp. 129-143. Springer Berlin Heidelberg, 2003.
Aric A. Hagberg, Daniel A. Schult, and Picter J. Swart. Exploring Network Structure, Dynamics, and Function using NetworkX. In Gaäl Varoquaux, Travis Vaught, and Jarrod Millman (eds.), Proceedings of the 7th Python in Science Conference, pp. 11 15, Pasadena, CA USA, 2008
15

---

Published in Transactions on Machine Learning Research (07/2024)
Markus Heinonen, Huibin Shen, Nicola Zamboni, and Juho Rousu. Metabolite Identification and Molecular Fingerprint Prediction through Machine Learning. Bioinformaties, 28(11):2333-234 September 2012.
Christoph Helma, Ross D. King, Stefan Kramer, and Ashwin Srinivasan. The Predictive Toxicology Challenge 2000-2001 Bioinformatics, 17(1):107-108, 2001.
John J. Irwin, Teague Sterling, Michael M. Mysinger, Erin S. Bolstad, and Ryan G. Coleman. ZINC: A Free Tool to Discover Chemistry for Biology. Journal of Chemical Information and Modeling, 52(7):1757-1768,
July 2012.
J. Johnson, R. Krishna, M. Stark, L. Ii, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei. Image Retrioval using Scene Graphs. 2015 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 3668 3678, June 2015.
Keisuke Kawano, Saloshi Koide, Hiroaki Shiokawa, and Toshiyuki Amagasa. Multi-dimennsioall fused gro- mov wasserstein discrepancy for edge-attributed graphs. IEICE TRANSACTIONS on Information and Systems, 107(5))683-6%) 66))) 2024.
Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation Learning for Dynamic Graphs: A Survey. Journal of Machine Learning Research, 21(70):1-74 73, 2020.
Kristian Kersting, Nils M. Kriege, Christopher Morris, Petra Mutzel, and Marion Neumann. Benchmark Data Sets for Graph Kernels, 2016. URL
cs. tu de
Nils Kriege and Petra Mutzel. Subgraph Matching Kernels for Attributed Graphs. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, pp. 291-298. Omnipress, 2012.
Nils M. Kriege, Matthias Fey, Denis Fisseler, Petra Mutzel, and Frank Weichert. Recognizing Cunciform Signs Using Graph Based Methods. In Proceedings of The International Workshop on Cost-Sensitive Learning, volume 88 of Proceedings of Machine Learning Research, pp. 31-44. PMLR, May 2018.
Ronny Luss and Alexandre D' aspremont. Support Vector Machine Classification with Indefinite Kernels. In Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
Facundo Mémoli. Gromov Wasserstein Distances and the Metric Approach to Object Matching. Foundalions of Computational Mathematics, 11(4):417-48) 2011.
Marion Neumann, Roman Garnett, Christian Bauckhage, and Kristian Kersting. Propagation Kernels: Efficient Graph Kernels from Propagated Information. Machine Learning, 102(2):209 240-2459,,,4 2016.
Cheng Soon Ong, Xavier Mary, Stéphane Canu, and Alexander J. Smola. Learning with Non-positive Kernels. In Proceedings of the Twenty-Fine First International Conference on Machine Learning, pp. 81. Association for Computing Machinery, 2004.
Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov- Averaging of Kernel and Distance Matrices. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Pro- ceedings of Machine Learning Research, pp. 2664-2677 PMLR, June 2016.
Raghmnathan Ramakrishnan, Paylo O Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chem- istry structures and properties of 134 kilo molecules. Scientific Data, 2014.
Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jean-Louis Reymond. Enumeration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDBB-17. Journal of Chemical Information and Modeling, 52(11):2864-2875 November 2012.
Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is More: Nyström Computational Reg- ularization. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein Distance Guided Representation Learning for Domain Adaptation. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 32, April 2018.
Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Efficient Graphlet Kernels for Large Graph Comparison. In Proceedings of the Twelth International Conference Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 488-495. PMLR, April 2009.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeller-Lchman Graph Kernels. Journal of Machine Learning Research, 12((7))2539-25)) 2561, 2011.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1548 1554. International Joint Conferences on Artificial Intelligence Organization, August 2021.
Martin Simonovsky and Nikos Komodakis. Dynamic Edge-Coonition Filters in Convolutional Neural Networks Graphs. In Proceedings of the TEEE Conference on Computer Vision and Paltern Recognétion (CVPR), July 2017.
Karl-Theodor Sturm. The Space of Spaces: Curvature Bounds and Gradient Flows on the Space of Metric Measure Spaces. arXiv preprint ar Xiv: 120.0044, 2012.
Mahito Sugiyama and Karsten Borgwardt. Halting in Random Walk Kernels. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Jeffrey J Sutherland, Lee A O'brien, and Donald F Weaver. Spline-fitting with a Genetic Algorithm:: A Method for Developing Classification Structure-aclivity Relationships. Journal of chemicaal information and computer sciences, 43(6):1906-2 1915, 2003.
Nyy Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein Auto-Encoders. In International Conference on Learning Representations, 2018.
Titouan Vayer, Laetitia Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Optimal Transport for structured data with application graphs. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6275-6284. PMLR, June 2019.
Titouan Vayer, Lactitia Chapel, Remi Flamary, Romain Tavenard, and Nicolas Courty. Fused Gromov- Wasserstein Distance for Structured Objects. Algorithms, 13(9):212, 2020.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. Tn International Conference on Learning Representations, 2018.
Cédric Villani. Optimal Transport: Old and New, volume 338. Springer, 2009.
Cédric Vincent-Cuaz, Titouan Vayer, Rémi Flamary, Marco Corneli, and Nicolas Courty. Online Graph Dictionary Learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10564-10574. PMLR, July 2021.
David P Woodruff et al. Sketching as tool for numerical linear algebra. Foundations and Trends0 in Theoretical Computer Science, 10(1-2):1-157, 2014.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Cromov-Wassersteen Learning for Graph Matching and Node Embedding. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6932 6941. PMLR, June 2019.
Hongtengl Xu. Gromov- Wasserstein Factorization Models for Graph Clustering, In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp.
April 2020.
17

---

Published in Transactions on Machine Learning Research (07/2024)
The supplementary material is structured as follows: In Appendix A present the proofs and detailed computations related to our theoretical derivations. Appendix B complements the main part of the paper by providing additional details on our algorithms as well as a method for dictionary learning, Appendix C gives additional information about the data and settings of the experiments presented in the main part of the paper. Finally, Appendix D includes additional experiments.
A Technical Proofs
In Scctions A.1 and A.2 we present the proofs of the thooretical propertics of the FNGW distance that appear as natural extensions of the properlies of the NGW distance Chowdhury & Mémoli (2019) and the FGW distance Vayer et al. (2020). We therefore leverage results from both Chowdhury & Mémoli (2019)
and Vayer ct al. (2020), and additionally the metric propertics of space for our proofs of Theorem 2.7
and 2.8 In Sections A.3 A.4 and A..5 we provide proofs and calculation of the expressions we need when
computing the distance itself and the barycenter. In Section A.6 we demonstrate the ILE property for the FNGW distance.
For the sake of completeness, recall first the definitions of the NGW and the FGW distance.
Definition A.1 (Network Gromov- Wasserstein Distance (Chowdhury Mémoli 2019)). Let g be the set of tuple of the form (x, /x) where X is a polish space, 9x X X - R is bounded continuous measurable function and μх is fully supported Borel probability measure. Given two tuples gx (X, pxx, gy (Y, ny) from G, the Network Gromov-Wasse Wasserstein Distance between gx and gr is defined for any [L, po] as follows:
NGW,,ex.9w.
min Ep( (eT()xxp)))
(19)
with
Ex USrV Jaw (xx(xxx)) (pr (ddp udd
(20)
It should be noted that the FNGW distance extends NWG into fused case and generalize the codomain space of yp from R to metric space. In the discrete case, when applied to graphs, we leverage this property to take into account edge labels as feature vectors.
Definition A.2 (Fused Gromov- Wasserstein Distance (Vayer et al. 2020 ) Let be the set of tuple of the form (X, 1x, (x, where X is polish space, tx X V is bounded continuous measurable function from X to a metric space (V, dy) probability measure. Given two tuples 9x = (X, thx (x). gY = fyyy,yy from 9 and trade-off parameter a E [0, 1], the Fused Network Gromov- Wasscrstoin Distance between 9x and gy defined for any (p. 4) E [1, oo] as follows:
X x X + R4 is metric, and μх is fully supported Borel
FGW PGW..p(0x..r)) = min Ex)
(21)
=Ell(x.ar)
with
(xx 11a ad (1x (vx)
+alpx(1,x)) - sy (y / sddpeed
(22)
Compared with the FGW distance, FNGW fuses another more general function: wx X X Ω while releasing the symmetric constraint of &p and allowing for feature vectors edge labels.
18

---

Published in Transactions on Machine Learning Research (07 2024)
A.1 Proof of Theorem 2.7 Existence of FNGW Distance
The proof takes mainly advantage of the Weierstrass theorem, which will use the following lemmas:
Lemma A.3 (Compactness Couplings; Lemma 10, Chowdhury & Mémoll 2019 ). Let X, Y two Polish spaces and let px Prob(X), jay Prob(Y). Then (Yx) compact Prob X x Y).
Lemma A.4 (Contimuity of the Functional μ + g. (9x sy: p)). For (p.0) 9) E [1, let 9x (X, pxx, wx, wx,xx)) px), gr thr, sy,, uV, ,y,, both from 9, then the functional
I(x,, + RR +00
p p+ E0,
is lower semicontinuous on 11(μx, (y) for the weak convergence of measures.
Proof. We define the functional f: X x X x + R4 U +0 with
f((2. (z'.8') [(1 a B)de (vix (x), by
1/1a +81px(0, - -pr 1991
(23)
then, f is lower semicontinuous due to the continuity of (dy, da) and (ex, bY. wy, 9x, pY). Using Lemma 3 from Vayer et al. (2020) by considering W = =X x Y, which is a polish space, we can conclude p + E. (Ux, (ux,,9'' SY, ,Y,4)) is lower semicontinuuus on [I) for the weak convergence of measures.
We can now prove Theorem 2.7 which mainly takes advantage of the Weierstrass theorem.
.y Proof. Since l1 uy) C Prob(X Y) is compact and the functional μ - Eap gY p)) is lower semicontinous on I(ux.py) we can conclude the functional achieves its infirum for some μ by applying directly the Weierstrass theorem.
A.2 Proof of Theorem 2.8 Metric Properties of FNGW Distance.
We divide Theorem
2.8 into four lemmas, and we suppose gx = (xxxx = (Y, w/, wywyyyywwy,rr. ,y) and Oz = (Z, 42, are from G.
Lemma A.5 (Positivity). ZNGWa.9.ap00s 20
Proof. FNGW w.x. since dy and dg are both metrics and (a, 3) [0, 1|..
Lemma A.6 (Symmetry).
gr) = FNGW a. (g .x))
Proof. For any u E Il(xx,Hy), let n² = Tap u be the push forward of a via a Borel map defined as follows
T:X Yxx
(I, v) + (v.z)
19

---

Published in Transactions on Machine Learning Research (07/2024)
Then we have, (by the property of the push forward)
pa. ,x,
- -(/rrwy - a (v), (vx(x))) adolww (v.
(v' :
+ - -xx
-V (1-c0 a1-a- a1 3)du (v> (0) (xx()))) addalwy (v: s/)xwxx (x, 2'))
+ + px -1 w)dpa' $
-oorrr a1-aa a- - (xx by (v)) + anda(ux(s, (x)
+ +81ox(0,,')) - py //'0/ dulr, d)p( s/1) $
The first equality given by property of the push forward (J fdT f foTdp) the second equality is given by the symmetry of du and da- As consequence, have PNGWo.8...r ,gx)))))))))))))))) FNGW A.p gr):
Lemma A.7 (Equality). FNGW 8. = O. Morcover, FNGW 0.3.9. gy) 0 if and only there is Borel probability space (Z, with measurable maps f 7 - X and g 7 - Y such that
f#Hz = p.
(24)
9#MZ = jay
(25)
[11 a - B)dy (vxx o f, "g) + adalf wx 9 wy + 31 px 9 pr 10 =0
(26)
where f#ux Z x Z + Ω is the pullback weight function defined by (2,2)) + wx(f(2)) $#xx
Z x Z + IR is given by (z, x') + px (x(),,,, ((x')) and g# is defined similarly.
Proof. The proof analogous to the proof of Theorem 18 in Chowdhury & Mćmoli 2019 the case where p E +x). For the backward direction, let us assume that there exists a Borel probability
We first deal with
space (Z, and measurable maps f : Z - X, 9 Z Y verifying Equation 24 26
We consider
n= = (f. First of all, it easy to prove that n² E & (ux,
AA B(X), n² (A X Y) uz(([.g) | Y) [zz (AA)) =x
BB E B(),, (X B) (z(([]g)) [X B)) = = 11 ([B] (B
Then we have
FNGW wWW,,,,0000x9
$( 1/ (xxxxy (1- a - -d)du (y), +
+ xlixx (1, - (v. 1/ )19" dur (z. ydd vz))
=11(1 3)dv (vix S, thy g) + ada wx, n wr # se (
which is equal to 0. Conversely, let or E my) be the optimal coupling satisfying ENGWa.9.4. (gx.gr) = O. So we have ((11---))de (1xx
= 0.
To prove the existence of the desired probability space and measurable maps, we define
Z = X Y, == μz f = projx == projy
20

---

Published in Transactions on Machine Learning Research (07/2024)
where projx: Z - X and projy Z + Y are projection maps. It can be shown that
AA E BX),, (ANz(A) - (A =x AB E (()) (B)) 'x x B) - (B
and
[I(1 a -Bdo (txx o fy o 9) adall wx n wr)" =1 xx 9 lr
=| Bddy (xxx + adalwx, wyy p 1 =0
The proof for the case p 2o is analogous.
For the specific case 9x gY, we consider (2, = (X, mx) and identity maps (f.g), then Equation 24 26 are well verified, so we have FNGW (00 = 0. The proof is thus concluded.
Lemma A.8 (Relaxed Triangle Inequality).
( FNGW 3. (9x, 02) S 2 o. 3.r + FNGW FNGW........ p9
Proof. Let Pab be the optimal coupling between (gx.9% . and pho be the optimal coupling between (0Y:92) By the Gluing Lemma, there exists a probability mesure j E Prob(X, Y, Z) with marginals Pab on (X x Y) and Mso on (Y x Z). Let Pae be the marginal jı on (X Z). Due to the fact that pag is not necessary a optimal coupling between (gx.g2) we have
FNGW BNGWa.B..p(9x.50 9z)
<| B) (5xx, dhz))² + adalux wz) Blyx -00-==211111 xr 11 @mac)
<1 - a B) (do (vx, thy + do (vy,, v2)) (dalwx, wr) + do wz))
+ -1pxx - spr + kp -21)" (((
<1( -a a 8)2- (dv (vxx, + do (vr, + a2a-1 (dolwxx " + deelwr, wz)
+.329-1 [|px pr + lpp -22 (
-2- I|| ((1 - a - 3)do (wxx wyy adg ( wy) + B|px (prl")
+ ((1 - a (
(+
ww + B/ - (22 (1(())))
$29-1 ( - a 3)
+
)d + 3 xx - 40 (
+ [1 - a (v)
Bl p2 (l
=24 (II(1 - a 3)d
adg + 3 px - yy |
+ [1 - a - 33)d (Wy 3
wdn " Blyp p2 9 (1)
=2911 (FNGW....... (gx 9y) ENG (gr gz)))
The second inequality is the result of the triangle inequality of the inner metrics, the third inequality is due (a" + b4). The fourth incquality is the consoquence of the
to the fact that Vq 2 1, Va, 2 0, (a + b)d < 20-1 Minkowski's inequality of the norm L @ ft). Finally, it proves the relaxed triangle inequality with a factor of 29-1, When g 1, the triangle inequality is well satisfied.
21

---

Published in Transactions on Machine Learning Research (07/2024)
A.3 Proof of Proposition 2.5
Computation Complexity Reduction for FNGW
Proof. Given tensor E, we define the matrix E[t] by Ei k) E(i,k, for any k,t. t. By the definition of tensor-matrix multiplication @ we have
(LE, B) B) =
E [E(i,)) (),,,))lee
E kE(i, E.G,,
ΣE (E(i, +G.40) t)) The
ΣΣ RJ
(E) + EIE - 2E (((())) TH. (27)
We note that the inner & and 1, in the last equation above, is the the one that is computed in the GW distance, considering that E[t] and E|t] are the similarity matrices. Taking advantage of Prop.1 in Peyré et al. 2016) the above equation becomes:
([(E,E)8T)
Σ Eltl pl + +1, Elt - 2E E(t]"
(((())p + 11 "1mp"NEE -
-2 > E[t]
where g R Rm is defined by g(E) (EE and h R - Rūxi is defined by h(E),,) [[E(i,))
A.4 Proof of Proposition 2.10: Justification of the Barycenter Algorithm
Proof. Using Equation and Equation
we can write
arg min EER
Σ>: Ai Σ pI m. + 1mp; TE[[t]T 2E[t]T Ex
arg min BERmxmXT
arg min BER
ΣΣ
X (E[t)'p1,, + +mmp E. E, nT)
Now let us write the first-order optimality condition. have:
is a minimum of the previous expression, we
V E ΣΣ Σ d [E[t)ppTT +Imps' Fg F.(11 (,]]mm)))))))))m))))
|E-E-
which reads
Vt. Σ > (VER (Elt)pp1)) + 1..pi
1..,TT E 2E(t]m E[(t] .T)))))))))) (1)))
We notice that for each we have the same optimization problem as the one described by Equation 12 of Peyré et al. (2016). Taking advantage of Proposition 13 of Peyré et al. (2016) and with the notations of the tensor operations, we obtain the desired solution.
22

---

Published in Transactions on Machine Learning Research (07/2024)
A.5 Proof of Proposition 2.11 Property of the FNGW Barycenter
Proof. By Equation 14 we have the expression for each element of the barycenter tensor (we omit here the iteration index for the sake of clarity):
E(s,r,t)= = P.P. ΣEΣ T. (ss,))
Summing it up along the third dimension, we have
Σ B((nn
Σ
Ai Σ (B( E E.G.L,0,
P.P.
1
A5 Σ T((8, Σ
Tee (r(r, E E.G.4.)
P.P,
1
Σ X
Σ TT(8,)) E Te(r.))
P.Pr
1
Σ X.P.P,
a P,P,
A.6 Proof of Proposition 3.1 Statistical Guarantees for Supervised Graph Prediction Estimator
Before going through the proof, let us recall the definition of ILE property.
Definition A.9 (ILE, Ciliberto et al. (0200)) ). A contimous map f: x - R is said to admit an Implicit Loss Embedding (ILE) if there exists a separable Hilbert Space H and two measurable bounded maps 3i Z + H and y + H. such that for any 2 Z and y E y we have
((2,8) (+(=), (()))
(28)
and (xp())) 1.
Then we recall the FNGW ddistancess definition when extended to Gm x G. Given Jm = EF,E, A. E 9m and g = (F, E.E.A.p) e G,
FNGW. a,9((9m:))
min Σ rEll(p.))
(o))F(()))) BU.DIE + +BA((T,)) A(j,1)?
1.
+ (1-a-))]1(() - THITIS
(29)
Proof. Using Theorem 12 from CCiliberto et al. 2020), we are going to show that FNGW a, satisfics the ILE property by proving that i) Gm is compact, ii) G is finite (trivial with the definition) and iii) the function INGWa,9(;s g) is contimuous.
First of all, we can see that 9m is compact, since [0, 1 1m Conv(F)m and Conv (T) mmmmm are compact (F and are finite and thus compact). Secondly, is finite by definition (Equation 15 Now, we will prove the continuity of ANGWa,a(ssh for any g E G. Denote dgm (dF, dA, 9m. For given g E 9,
23

---

Published in Transactions on Machine Learning Research (07 2022)
have, for any (gms dgm):
FNGW INGWa.s(Om + :gm:g)) - FNGW a. (Bm:
a (2(dE(i, k), E(i, B(, +
< sup
Σ mell(p.P) k.jkkl
3 (2dA(i, k) x (A(i,k) A(j.0) dA((EA))))
(1 a- a- (2(dF(1), F(1) F())RS +IdF(i))i))) TEIITES
$ Σ a (2]1F k) [((i,) F.G,))RRT 1Ka (I,
k..kk
B (2)dA(i, x A(i AG. + dA(i,A)3
(1 a1
a 3) (2))d) [F)) FG)lus + (1F(
(30)
The first inequality is due to the fact that vf.g [1 p) - RR min, f(x) min, g(T)) S sup, lf(x) g()))
The second is direct application of the Cauchy-Sehwarz inequality and the fact that Vi, Fij S 1. Since E, F and A are all bounded, there exists M E IR such that for any i, k,t:
M [[ E(j. Rlr
(31)
M 2 [A k) A(j,1)
(32)
M 2 [F(() F()
(33)
Then we have
FNGW D (gm dgm. 9) FNGW a, (gm.
Σ
(2) M + || k) (2))) + (2]dA(i, dd(i,A)))
kJ.ky!
+ (1-a- (2]ldF()) - [[dF (2=))
(34)
Now when dg |Rmx Rmx x RXxmx 0, we have |mxs + 0, |dA Rmxm + 0, and ||dE| Rmxmxr O, we can easily deduce from Equation 34 that
FNGW (gm + g) - FNGW (NGGW,,(0m..t)) idgml11+0
0
(35)
Hence, we have that for any g E 9, FNGW (...()))))) is continuous on RRmx x Raxx x |RixmxX" and thus on 9m.
Since FNGWa.. a.3 admits an ILE, Theorems 8 and 9 from Ciliberto al. (2020) can be instantiated on the FNGW .-GWa,s-bbsed estimator in Equation 17 which gives the following results.
Theorem A.10 (Universal Consistency). Let k be a bounded universal reproducing kernel. For any N and any distribution p X' 9, let fo X Gm be the estimator defined in Equation 17 trained [1x 91 samples independently drawn from p and with X 1 then
lim R(fx)= = R(F) with probability 1
(36)
with R()) = F,,FFF,,,,W.. (J(X), G)] and fr denotes the bayes estimator.
Theorem A.11 (Excess-risk risk Bounds). Let H be the Hilbert space associated with the loss FNGW o. 9m X9- R4 the ILE definition. Let k X x + R be continuous reproducing kernel on X with associated RKHS F such that x² == suprex k(2, x) < t00. Let p be distribution on X G and suppose that the solution h" of the surrogate regression problem belongs to the considered hypothesis space H 8 F.
24

---

Published in Transactions on Machine Learning Research (07/2024)
Let 8 E (0, and no sufficiently large such that no 1/2 9m.2 log 8. Then, for any n no; the estimator fn defined in Equation 17 trained on points independently sampled from and with X = n 1122 is such that, with probability at least 8
no
R(()) - { cclog(400) n1/4
(37)
with constant independent of and 8.
A.7 Proof of Invariance of FNGW to Node Permutation
Proposition A.12 (Iuvariance of FNGW to Node Permutation). Given graphs 91 = (1,, E1,P)) 92
(F2, A2, E2, P2), and a new graph 91 (F1, E, obtained by applying a node permutation a on 91. then we have
(...l. 92) FNGW (00
(38)
Proof. Given two graphs gn = (Fi, A1, Pt) and g2 = (F2, A2, Foz, P2), the FNGW distance between them is given by
(ΣΣX 1= 31
P adn + -AAA:A46.6.-----A------------000 (P) (())) T
min rell(p)PP))
(39)
Suppose that `1 (Fi, A,, p1) is isomorphic to In with a permutation application a [1,m] - [1.m1]. so the FNGW distance between the new permuted graph Ø1 and 92 is
min (ΣΣΣ) adn HAAA://AA--AA-AAAAAAAAA-A------- "
fx.
#ETT(P, P2) k=
(40)
Since F (i, F
10), a (k))
- (1) (k)), (i) (i *x.1 ab) and
i no (i).s due to the permutation, Equation 40 can be rewritten by
MI mi
min PP
Σ
add (())=
(0 A2(j.1)1
k=1 JM
TT-1(0) ( $
a11-a-3yde
(41)
Denoting = (i) and = (K) and set S (1) [L,m]] we obtain thus
ΣΣΣ SES LES
P adn B...))).....)))) -0 (F)) (s)))) Tai
min #ETT(P)-PP)
(42)
Since the permutation is bijective from [1,m.] to [1, m.], have S [1, which leads to
ma
m
min nETT(P, PP))
ΣΣΣ
(Fi(s, Te,iTes
(43)
It's clear that Equation 43 describes the same minimization as the one of Equation 39 the proof is thus
concluded.
B Additional Details on Algorithms
In this section, present additional details our algorithms.
25

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm 3 Line-Search in Conditional Gradient Descent.
Input: a, b
if > then 7(h) = min(1, max(0, -2))
else
if + b then ((k)
clsc
((k) 0
end if
end if
Output: (0)
B.1 Line Search in Algorithm 1
Denoting A = ((k-1) - n (k-1)) A = A'p + +m T and A2 g(E)p1) + (m)))((E) then we have
((k) n (1) + (k) A and
(T()(
(1 a1-a-10 F) +J 8) x + aL(E, E) & (k), (())
(1 ()M(F, + B(A, (A) a(A2 2 > Fr(t)r(A)) B(t) ), x()
11 11-a B)M(F, + 3(A1 - 2A(T(k-1) + +2(1)))(T)
+ +a(A2 2 Σ El + +
x (k-11) + 70(0
(44)
Then we can rewrite E (T((1) as (k) + (k) + with
T
-2
F[AAET - -2AAATT,
(45)
1=1
= (1 BFM(FF F) aa(A2 - 2
)(1 + 3(AI (K-1) (1-1) A
t=1
-2a
El 23A44 (K-11)
23A4A
(46)
Hence, the line-search presented in Algorithm above.
! can be performed with Algorithm 3 Juiin and defined
B.2 Graph Dictionary Learning with the FNGW Barycenter
We now describe a learning task that exploits the notion of FNGW-barycenter. Dictionary learning based on graphs is an extension of factorization methods to graphs and was popularized by the seminal works of Xul (2020) and Vincent-Cuaz et al. (2021). In our sctting, given a set of labeled graphs (9, 1 with 9: (F Er, P,) we want to learn dictionary of atoms {g.15- so that each graph of the training set can be reconstructed FNGW-barycenter of the dictionary atoms.
26

---

Published in Transactions on Machine Learning Research (07 (024)
Algorithm 4 Unmixing Problem Solver
input: Graph 9, Dictionary {g.]s-
init: Initialize uniformly w.
repeat
With fixed, compute the barycenter with Algorithm 2 obtaining B(w, (g.) and save S independent optimal transport plans т. between 9 and J.-
Compute the optimal transport plan æ between and g.
Compute the optimal w by minimizing Equation 48 with fixed transport plans л and { using the CG algorithm.
until Convergence of the reconstruction loss. output: w, π and {T
Assuming that the probability distributions for the atoms {pa}-- are fixed beforehand, our dictionary learning problem writes as:
min (w.).- 1............ (E.,A,
FNGW a,9 (9.: B(ws, (8.1 pi)) 10.1
Σ w
(47)
1=1
where the weights (w:); describe the embeddings of our graphs the dictionary and a regularization parameter which controls their sparsity.
Remark B.1. should be noted that this problem setting corresponds the dictionary learning in Xu (2000) rather than the one presented in Vinneet-CCur et al. (2021). ), where graph is projected linear reprosentation of atoms. Here, representing a graph as FNGW-baryconte of the atoms allows in particular to use aloms comprising various number of nodes.
We propose to solve the optimization problem of Equation 17 with a stochastic iterative procodure, which for each sampled batches of data alternatively updates the embeddings {w;}: and atoms {(,,,,,,F.))3 Finding the embedding {wi} for a fixed dictionary requires an intermediate procedure called unmining, which our case a bi-level optimization problem, since the reconstructed graph is the solution of a barynttryee problem. Assuming small change in w; will not affect the solution of the barycenter problem, we propose therefore to solve the latter first, and find the optimal w, with the fixed OT plans of the barycenter.
Let us formalize the unmixing procedure, which can be described solving the following problem:
min FNGW ( 6. wEEs
(9, B(w, (o)) Al|w>|1
(48)
Our detailed algorithm is provided in Algorithm
This procedure is applied to the graphs in (othi, giving corresponding {w:}; and transport plans, which B
allows us to update atoms {(7,)) by gradient descent. Given a batch of graph samples { we define the following functional, representing the batch loss:
B Ea, ((Eb, Ab, Fb), (Es, Ao. Fb) mb)
A. E.) =
(49)
27

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm 5 Stochastic Graph Dictionary Learning
imput: Graph dataset {9: (0
init: Randomly initialize the atoms ((P,, A,. Fs)
for h = ,,... do Sample a mini-batch of graphs from dataset: {(E,, Ag, Fb. Ph)
for b= I..... B do
With {(Es: A, -- fixed, solve the unmixing
problem:
Wb: Wb. (Tb,
A 1 (Eb, Ab, Fb, Po), {(E,, As, F)))
end for for s= S do
With fixed wt. (To la: compute the gradients of the mini-batch loss with respect to {E,, Ag. F.).
Update (Es, As, F using gradients from Equation 53 to Equation 55
end for end for Output: The atoms of the dictionary ((E,, A,,
with (Es, As, Fb) the reconstructed graph of the batch sample do from the results the unmixing problem and my the optimal transport plan between them. The reconstructed graph is then expressed as:
Es
1
Σ Was(CC Te.b) X1 Tob
(50)
IMXT X2 pp
As ppl
1
Σ
W. We, ,,
(51)
Fs Σ
W. w.
TT.
(52)
where T& denotes the optimal transport plan between the reconstructed graph 96 and atom J.. Then the estimated gradients is written as:
2
V
L
Σ
W 8.b
-Diii
Fr
Diag(
] Pb
(53)
B
Pb
2
V
7,
L
B
W
Wabbks s.b
A
PoP!
Th As
T
(54)
b=1
2
V
Ws.b
Es
(Es X2 mh) x1 mb
x2 Tap x F..
(55)
B
IMxT X2 P&Pp
b=1
The complete algorithm for dictionary learning is summarized by Algorithm 5
C Additional Details on Experiments
This section is dedicated to additional details about the experiments carried out in Section
C.1
featuree Graph Classification
Datasets. We consider 7 graph datasets for classification: Cuneiform Kriege et al. 2018
MUTAG Deb-
nath et al. 1991: Kriege & Mutzel 2012), PTC-MR (Helma et al. 2001 Kricge & Mutzel 2012 BZR-MD,
COX2-MD, DHFR-MD and ER-MD (Sutherland et al.) 2003 Kriege & Mutzel 2012). While Cuneiform is one of the earliest systems of writing realized by wedge-shaped marks on clay tablets, the rest of the datasets are of the nature of small molocules. Table describes the details of features occurring
these datasets. All the datasets can be downloaded from https://1si1-ww cs. tu-dortmund. de/staff/ morris//gaapkerrelllatasssss
28

---

Published in Transactions on Machine Learning Research (07/2024)
Table The details the features concerning the datasets used in graph classification.
Features
Cunciform
MUTAG
PTC-MR
BZR-MD
COX2-MD
DHFR-MD
ER-MD
Node label (Discrete)
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Node attribute (Real-valned)
Yes
No
No
No
No
No
No
Edge label (Discrete)
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Edge attribute (Real-valued)
Yes
No
No
Yes
Yes
Yes
Yes
Table 5: The types of the features treated by the methods used in graph classification.
Restures
RWK
SPK
GK
WLK-VH
RОPPERR
PROPAK
FGW
NSPDK
ERK
VEHK
WLK-VEH
PNA
GAT
FNGW
Node lobel (Decrete)
Yes
Yes
Node attribure (Real-nined)
No
No
No
No
No
No
Yos
No
No.
No.
No
Yes
Yos
Yes
Edge label [Dircrese]
No
No
Yos
Yes
Yes
Edge sttribute valued)
No
No
No
No
No
No
No
No
No
No
No
Yes
Yes
Ye:
Experimental Settings. For datasets whose node features are discrete, in Vayer al. (2020), we use Weisneiler-Lehnnnn (WL) labeling to transform them into vectors of dimension K, being the number of iterations in WL labeling. We use dataset-specifie random vector feature for empty edges. To compute the FNGW distance, regarding the distance dv, we use the Hamming distance when WL labeling is used, or the e--orrm otherwise. The distance da between edge features is the e in all cases.
For all methods, we conduct nested cross-validation with 50 iterations of the outer CV loop and 10-fold inner CV. In cach outer CV loop, the dataset split into a training set and test set, with 10% of the data held out for the latter. The splitting is kept consistent across all methods for fair comparison. We report the mean and standard deviation of the test accuracy.
For the classifiers based on FNGW or FGW, the cocfficient 7 presented in the Gram matrix computation is cross-validated within {2' i e [-10, 10]} while the number of iteration K in WL labeling is searched in [0, 4] for MUTAG and PTC-MR or [0,3] for the other datasets (0 means no WT. labeling is used) except Cuneiform. For FNGW, we cross-validate values the trade-off parameter a via a logspace search in [0, 0.5] and the same strategy is applied 8, while a total of 15 values are drawn from logspaces [0, 0.5] and [0.5, cross-validate a for FGW, For the kernel-based methods, decay factor À of RWK cross-vas pross-validated from {10' E [-6,- -2],, the number of iterations of WLK is cross-validated in [1, 10] while the of PROPAK is chosen from {1, 3, 5, 8, 10, 15, 20). For GK, the CV range of the graphlet size is {3, 4,55 and the one of the precision level and the confidence 8 is {0.1, 0.05). For NSPDK, the maximum considered radius r between vertices cross-validated within [[0, 5], and neighborhood depth d is chosen from {3, 4, 5, 6, 8, 10, 12, 14, 20). Finally, the regularization cocfficient C of the SVM rrossvalidated within {10' | E [ [-7. 7, 7 for all the methods except for the case MUTAG-FNGW where the value 107 is not included.
Runtime Comparison. We compare the Gram matrix computation time of the different graph kernels, FGW and FNGW used in the experiments. The results are presented in Table
C.2 Fingerprint to Molecule.
Experimental Settings. Table 7 outlines the details of the parrm-erreeeter searching range for the cross- validation for both FGW and FNGW loss. The constrained exploration of the NNBary range reflects the substantial computational resources demanded by this method.
C.3 Metabolite Identification.
Dataset. To evaluate the performance for metabolite identification from tandem mass spectra, we use the data extracted and processed in Dührkop al. (2015), which is set of 4,022 small compounds from the public GNPS Public Spectral Libraries sets built. with molecular structures from PubChem. The dataset. be downloaded from https:
uesd. ProotoosAFe/lbbrariess
The candidate
29

---

Published in Transactions on Machine Learning Research (07 /024)
Table 6: Runtime Gram matrix computation for graph classification on MUTAG, PTC-MR, and BZR-MD
datasets. 10 times computation are conducted for each method. The last five rows show the results for methods leveraging the edge features.
Methods
MUTAG
PTC-MR
BZR-MD
RWK
38.756 + 0.126
64.095 + 0.473
114.663 + 0.646
SPK
0.283 0.019
0.431 + 0.019
1.423 + 0.078
GK
42.201 + 0.317
78.518 + 0.124
83.448 0.122
WLK-
0.072 = 0.006
0.089 = 0.011
0.385 = 0.016
HOPPERK
11.491 = 0.048
46.148 = 0.549
14.411 0.070
PROPAK
0.344 0.033
0.796 = 0.026
0.633 = 0.016
FGW
66.120 + 0.202
167.880 + 0.209
99.201 0.152
NSPDK
1.736 0.019
2.242 + 0.040
34.017 0.259
EHK
0.002 0.000
0.003 + 0.000
0.022 + 0.004
VEHK
0.005 0.000
0.008 0.001
0.069 = 0.019
WLK-VEH
0.076 0.002
0.120 0.009
0.731 0.010
FNGW
240.508 1 4.958
565.021188.266
$ 27.510
Table 7: Hyper-paramene searching ranges during the cross-validation on Fing2Mol dataset.
saper-parameters
NNBary
ILE
of the FGW /FNGW
{0.01, 0.5, 0.9) {10, 15}
{0.01, 0.1, 0.5, 0.9, 0.99]
Number of graphs for barycenter
{15, 20, 25}
? of the Gaussian input kernel
{10-6 10-5 10-1 10 {10-10, 10 10 1006,, 1,00-,10-- 1,-4, 10-4,10-3 10-3)
10-2, 20--,,00000111111 10-1)
Ridge regularization parameter
Sketching size
(1000,5000)
Hidden dimension of MLP
{128}
//zenooo . org/recorr/8004411 1tt.././//..00211.....1..22 which is released under Creative Commons Attribution 4.0 International license.
Experimental Settings. We choose the ridge regularization parameter (A 10-4) and the diffusion rate = (.6) following Brotat-Motte et al. (2022). We use separate validation set (1/5 of the training set) to select the hyperparameters a and / of FNGW loss. and B are chosen from f0.1, 0.33, 0.5, 0.67] with the constraint a 3 < 1. For the experiment with FGW, we keep the same hyper-parameter as in Brogat-Motte
ct al. 2022)
D Additional Experiments
In this section, we present additional experiments showing the application of our proposed FNGW distance.
D.1 Runtime of FNGW Computation with Respect to Graph Size
To give an idea of the scales involved, the run-time with respect to the number of nodes (the graphs are randomll created with both node feature and edge feature of dimension 2; the graphs to be compared are of the same size) is shown in Table The maximum number CGD steps is set to 100 for both distances. The experiments are conducted on an Intel(R) Xeon(R) CPU E5-2650 v2. While experiments this paper are made on small graphs, where the difference in computation time between FGW and FNGW reasonable, applying FNGW efficiently large graphs will require further investigation.
30

---

Published in Transactions on Machine Learning Research (07/2024)
Table 8: Runtime (s) of computation of the FNGW w.r.t graph size.
# nodes
FNGW
FGW
25
0.0294 + 0.0089
0.0039 + 0.0003
50
0.0949 + 0.0290
0.0047 + 0.0009
100
1.4640 0.5598
0.0137 + 0.0005
200
10.2419 + 2.8562
0.0435 + 0.0063
Table 9: Computation runtime (s) comparison of FNGW and GED on QM9 molecules.
FNGW (a = 0.33, 3 = 0.33)
GED w/ Edge feature
0.63 + 0.01
276.84 + 0.56
D.2 Runtime Comparison Between FNGW and GED
We randomly select 100 pairs of molecules from the QM9 dataset and compute the distance between each pair using FNGW or GED. This procedure is repeated 10 times, and the total time required is reported in Table We utilize the NetworkX implementation of GED.
D.3 Real Migration Network Clustering
In this experiment, we use the global bilateral migration networks relcased by the World Bank and processed by Chowdhury & Mémoli (2019) to test our distance. Differently than Chowdhury & Mémoll (2019), the capacity of the FNGW distance model higher dimensional edge features allows us to take account both networks (corresponding to the male migration and the female migration, separated) conjointly, by combining their edge features. The resulting dataset consists of 5 graphs, each containing 225 nodes representing countries or administrative regions. The feature of each edge (i j) is two-dimensional vector, where each indicates the number of respectively male and female migrating from region i to region j. Since the networks are complete and do not possess any node features, the tradc-off parameter a of the FNGW distance set to 1. The weights of the nodes the network are uniformly distributed. Figure shows the dissimilarity matrix of the migration networks and its associated single-linkage clustering dendrogram.
Single-linkage Clustering Dendrogram
Dissimilerity metrix
1960
2.0
1.0
1.0
3990
0.4
2000
0.5
0.0
1980
1960
1990
2000
year
Figure 7: Results of migration networks
31

---

Published in Transactions on Machine Learning Research (07 (274))
D.4 Synthetic Labeled Graph Clustering
In this synthetic task, generate 3 groups of graphs, following the Stochastic Block Model (SBM) with the number of blocks in each graph chosen from {1, For each cluster, we generate 15 graphs whose numbers nodes are randomly chosen from {20, 30, 40). For each graph, the feature of the nodes from block is sampled from ((i,1). For the edges, we consider the following labels: {black, blue, green). Edges between nodes the same block are labeled as black with probability 1. For every pair nodes (p: q) with p from block i and node g from block i + 1, there exists an edge p + q labeled as green, and there exists edge 9 p labeled as blue with the same probability. There is other possible edges. In total, the dataset contains 45 graphs. Several synthetic graph samples are shown in Figure g We apply the k-means algorithm in order to perform graph clustering, considering the FNGW barycenter the centroid of each cluster and the FNGW distance as the cluster assignment metric. Cluster centroids are randomly initialized with 20 nodes; their evolution as found by k-means is presented in Figure 9 We can observe that the resulting centroids recover not only the node features but also the characteristic of the edge labels of the different groups in the synthetic dataset.
Figure 8: Samples graphs for clustering with 20 nodes
Figure Evolution of centroids of each cluster found by k-means algorithm.
32

---

Published in Transactions on Machine Learning Research (07 (07/2024)
D.5 Representation Learning through Dictionary Learning
We structure this toy problem around 3 graph templates, that we generate following the Stochastic Block Model (SBM) with 1, 2 and 3 blocks. Each template consists of 20 nodes. For each, node features are scalars representing the block they belong to. Within block, all nodes are connected with black odges. Pairs of nodes from adjacent blocks are connected with probability 0.3 by edges that are colored in blue when ascending, green when descending. The Templates are shown Figure [0a] We repeat the following procedure create a synthetic dataset for dictionary learning: given vector R which components are sampled from (00, 1), compute simplex barycenter of the templates defined above with Algorithm 2 The number of nodes is set to be 20 for all barycenters. To control the quality of the generated barycenters, we set maximum barycenter loss value of 0.3. The goal of the experiment is to retrieve graphs presenting the same characteristics with the 3 original templates through dictionary learning, even with a varying number of nodes, Setting the numbers of nodes of the atoms to 5, and respectively, we perform dictionary learning on the dataset. The learned atoms are shown in Figure 10b and clearly recover the properties of the synthetic templates.
softmax(v) and generate new graph by computing the
(a) Synthetic templates
(b) Learned atoms
Figure 10: Dictionary learning on synthotic datasct.
D.6 More Prediction Examples Fin2Mol Dataset.
Figure
shows more predictions of the different methods Fin2Mol dataset.
33

---

Published in Transactions on Machine Learning Research (07 2024)
GED wia EF 8.0
GEO WFo - 6.0
GED w/o EF 13.0
GED w/o EF 0.0
GED w/o EF 7.0
GED w/o EF - 4.0
GED w/o EF - 2.0
GED w/c EF 1.0
GED 5.0
weo
GED w/n 2.0
w/o 1.0
w/a w/
w/O
GED
(a) NNB
(b) NNBary-FNGW
(c) ILE-FGW
(d) ILGW
(e) True target
Figure 11: More predicted molecules on the Fin2Mol dataset.
34


Reviews:
[{"review_id": "MJawl08cRr", "reviewer": "Reviewer_ZFM9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n- This paper addresses the challenge of effectively incorporating edge features within the framework of OT-based distance for graphs. This method is considered to be widely applicable, and also can be a good literature for further development of this line of research. \n- The proposal has been generalized to bounded continuous measurable functions on Polish spaces, and its properties have been carefully analyzed, constituting a good theoretical contribution.\n- In experiments, the authors have compared not only graph neural networks but also a important family of graph kernels and the graph edit distance, representing a valuable empirical contribution.\n\n### Weaknesses\n\nThere are several concern regarding the presentation and quality of the paper.\n- Although we can guess from Example 2.2, the procedure for converting graphs into the quadruples used in Definition 2.1 is never provided in this paper. Since this process is nontrivial and the outcomes may largely depend on the way of conversion, it should be carefully explained (maybe at the beginning of Section 2).\n- Since Example 2.2 is an example of Definition 2.1, please also present the actual FNGW value based on this example. Otherwise, this example should be placed before Definition 2.1.\n- How were the hyperparameters of each graph kernel and graph neural network tuned in the experiments?\n- The vertex-edge histogram kernel should be used instead of the edge histogram kernel (EHK) because EHK does not utilize label information, leading to an unfair comparison with other methods that utilize vertex label information.\n- I believe edge label information is ignored in the WL kernel, although it is feasible to incorporate it by extending the WL, for instance, by integrating neighboring edge labels along with vertex labels. While such a straightforward extension may not yield optimal results, it is worthwhile to compare to establish a baseline WL that incorporates both vertex and edge features.\n- Please include a runtime comparison, particularly with other methods in Table 1, and compare with Graph Edit Distance (GED) in Figure 4. Although the proposal might not be the fastest, showing its runtime alongside other methods adds significant value for readers of TMLR.\n\nMinor points:\n- In \"Notations\", $\\Sigma\\_n$ should be not like $\\\\{A, B\\\\}$ but $\\\\{A \\mid B\\\\}$ or $\\\\{A : B\\\\}$ to distinguish enumeration and predicates.\n- In Equation (1), $\\\\{F, A, E\\\\}$ should be $(F, A, B)$ as it is not a set.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In measuring the distance between graphs, this paper proposes incorporating not only node features but also edge features of graphs using the Gromov-Wasserstein distance, which is applicable to downstream graph machine learning tasks. The proposal extends the fused and network Gromov-Wasserstein distance, called FNGW, and its theoretical properties as a metric have been analyzed. Furthermore, its effectiveness in graph classification and prediction tasks has been evaluated on real-world graph datasets, compared to other established methods such as graph kernels and graph neural networks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns outlined in the Weaknesses section above.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gcLb7yvTVo", "reviewer": "Reviewer_WuDS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The premise is that graphs with edge attributes constitute an important application case. The examples given are: 1) abstract meaning representation in NLP, 2) scene graphs in computer vision, 3) cheminformatics and 4) temporal graphs where the edge features evolve over time. However the authors offer an empirical evaluation of their approach only in the cheminformatics domain. \n2. The introduction of the approach is overly formal, hard to follow and lacks a natural language guiding: section 2 needs to preface each theorem and each formal entity with a description in plain English (what are the concepts used in Eq. 9? what are \\phi_X \\psi_X \\omega_X ?)  , what is theorem 2.5 and 2.6 saying in English? What is the interesting property for the barycentre enunciated in Proposition 2.9 in English?\n3. The formulation of the approach makes use of the matrices E and A that code the graph structure in a non invariant way, i.e. not invariant to an isomorphic permutation of the node ordering. \n4. Clarifications. The empirical evaluation needs to clarify if real valued edge attributes are taken into consideration, in this case please be clear on: 1) datasets properties, i.e. do graphs have only discrete edge labels or also real valued edge attributes? ad 2) methods capabilities, i.e. can the method  process discrete edge labels?  can it process real valued edge attributes?  \nPlease include a takeaway message in the caption of the figures, for example in Figure 2, what does the reader need to understand from the two rightmost plots? that there is an optimal value for alpha and beta and that it is robust/not-robust?\n5. The barycentre computation is the second major contribution of the paper and deserves: 1) a more detailed justification of why it is an important operation and how it can be used in applications and 2) a more intuitive and convincing empirical demonstration.  \n6. The edit distance correlation measure is a simple to understand quality score. However the score is not compared to other approaches.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new distance for graphs based on optimal transport. They extend the Fused Gromov-Wasserstein Distance to the case of graphs with edge labels. They additionally provide an algorithm to compute the barycentre of a set of graphs based on the proposed distance notion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be of interest to see how the proposed approach performs on the other application domains: NLP, CV and temporal graphs. \n2. Preface each theorem and symbol with an explanation of its meaning in English. \n3. An explanation of how the distance computation is not affected by isomorphic permutations would be beneficial (or showing what is the effect of such permutations on the distance).\n4. Please add a clarification on the treatment of real valued edge attributes. Please add take away message in captions. \n5. It should be possible to conceive a clear artificial case where the result of the barycentre computation is known. For example one could iteratively perturb a graph (e.g. using operators such as node addition/removal or edge addition/removal) to generate a set of graphs such that the original graph would be the average/barycentre and compare the computed barycentre graph to the source graph. \n6. It should be of interest to compare the correlation score of the proposed approach to the other similarity measures used in 4.1.1. \n\nCritical adjustments are: 2 and 5", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7etTnRvz5X", "reviewer": "Reviewer_EgyT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S: The problem setting should be important for the community, and the experimental results indicate high prediction performance. \n\nW: Some technical detail should have been clarified in more detail as I mentioned below.\n\nW: I don't fully understand the novelty of the paper. The edge attribute simply introduced through the distance of the attribute, and the optimization applies the Frank-Wolfe algorithm as is. Supervised graph prediction is directly taken from Ciliberto et al., 2020, and Brogat-Motte et al., 2022. In other words, technical difficulty newly solved by this paper is not clear (Note that I'm aware of that technical significance is not necessarily required in TMLR).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension of fused network Gromov-Wasserstein distance so that edge attributes can be incorporated into the distance. The cost matrix of the transportation is defined by a linear combination of edge attribute distance, difference in the shortest path matrix, and the distance of node attributes. Further, an application to the barycenter problem and its supervised prediction are also discussed. The barycenter optimization is formulated as a block coordinate descent procedure, and the framework of the supervised graph prediction is also discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Recently, the fused Gromov Wasserstein with edge-attribute has been consider by the following paper. Although it is quite recent, relation should be clarified in detail for scientific fairness. \n  - K. Kawano, S. Koide, H. Shiokawa, T. Amagasa, Multi-dimensional Fused Gromov Wasserstein Discrepancy for Edge-Attributed Graphs, IEICE TRANSACTIONS on Information and Systems, 2024\n\n\n- Theorem 2.6 should be explained in more detail. In particular, what the last two properties indicate? The last one is similar to the triangular inequality, but slightly different. In the end, does FNGW satisfy the axiom of metric?\n \n\n- The third property of theorem 2.6 guarantees FNGW(g_X,g_X) = 0. However, because of the non-convexity of the FNGW optimization problem, it is not clear whether this condition can be satisfied in practice. This seems also same for The second and last properties. I hope it clarified.\n\n- Although in the experiments, exp(-gamma FNGW(g_i,g_i)) is used, is it guaranteed to be positive semi-definite? (I guess it isn't). It should be clarified that how it is handled.\n\n- GED in (16) does not depend on edge attributed? If so, how does the edge attribute contribute to the prediction?\n\n- In Appendix D.1 the authors mentioned 'the difference in computation time between FGW and FNGW is negligible'. However, I do not agree with this claim because the computational time of FNGW and FGW has 10 times difference even in #nodes=25. Since kernel method is used in the experiments, at least, O(train_size^2) times distance calculations are requires by which I guess difference is not negligible.\n\n- The authors mentioned that hyper-parameters are selected by CV. Does that mean the nested CV is performed? (hyper-parameters are selected by the inner CV, and table 1 is performance on the outer CV?).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript considers the problem of supervised learning on graph-valued data. Here a common approach is to first define a metric between graphs, and then proceed with learning procedures that can leverage this notion of distance (or similarity) between the training inputs. A recent innovation in this space is to use ideas from optimal transport, such as the Wasserstein distance and extensions thereof, to formalize and compute this notion of distance between graphs. However, existing tools are not natively equipped to handle the case where both nodes and edges in the graphs have attributes. The authors address this problem by defining a new graph metric that can handle this case and demonstrate the power of their proposed methodology in a series of well-designed empirical studies on real-world datasets.

The reviewers were universally positive in their initial assessment of this work, and this sentiment was only strengthened throughout the author-reviewer discussion period. Ultimately the reviewers agreed that the paper satisfies both TMLR acceptance criteria and were unified in their recommendation of acceptance. I agree with their assessment.

I encourage the authors to reflect on the reviewer feedback and author-reviewer discussion in preparing the camera-ready version of their manuscript.
Decision: Accept (poster)