Semi-relaxed Gromov-Wasserstein divergence and applications on graphs

1 INTRODUCTION . One of the main challenges in machine learning ( ML ) is to design efficient algorithms that are able to learn from structured data ( Battaglia et al. , 2018 ) . Learning from datasets containing such nonvectorial objects is a difficult task that involves many areas of data analysis such as signal processing ( Shuman et al. , 2013 ) , Bayesian and kernel methods on graphs ( Ng et al. , 2018 ; Kriege et al. , 2020 ) or more recently geometric deep learning ( Bronstein et al. , 2017 ; 2021 ) and graph neural networks ( Wu et al. , 2020 ) . In terms of applications , building algorithms that go beyond Euclidean data has led to many progresses , e.g . in image analysis ( Harchaoui & Bach , 2007 ) , brain connectivity ( Ktena et al. , 2017 ) , social networks analysis ( Yanardag & Vishwanathan , 2015 ) or protein structure prediction ( Jumper et al. , 2021 ) . Learning from graph data is ubiquitous in a number of ML tasks . A first one is to learn graph representations that can encode the graph structure ( a.k.a . graph representation learning ) . In this domain , advances on graph neural networks led to state-of-the-art end-to-end embeddings , although requiring a sufficiently large amount of labeled data ( Ying et al. , 2018 ; Morris et al. , 2019 ; Gao & Ji , 2019 ; Wu et al. , 2020 ) . Another task is to find a meaningful notion of similarity/distance between graphs . A way to address this problem is to leverage geometric or signal properties through the use of graph kernels ( Kriege et al. , 2020 ) or other embeddings accounting for graph isomorphisms ( Zambon et al. , 2020 ) . Finally , it is often of interest either to establish meaningful structural correspondences between the nodes of different graphs , also known as graph matching ( Zhou & De la Torre , 2012 ; Maron & Lipman , 2018 ; Bernard et al. , 2018 ; Yan et al. , 2016 ) or to find a representative partition of the nodes of a graph , which we refer to as graph partitioning ( Chen et al. , 2014 ; Nazi et al. , 2019 ; Kawamoto et al. , 2018 ; Bianchi et al. , 2020 ) . Optimal Transport for structured data . Based on the theory of Optimal Transport ( OT , Peyré & Cuturi , 2019 ) , a novel approach to graph modeling has recently emerged from a series of works . Informally , the goal of OT is to match two probability distributions under the constraint of mass conservation and in order to minimize a given matching cost . OT originally tackles the problem of comparing probability distributions whose supports lie on the same metric space , by means of the so-called Wasserstein distance . Extensions to graph data analysis were introduced by either embedding the graphs in a space endowed with Wasserstein geometry ( Nikolentzos et al. , 2017 ; Togninalli et al. , 2019 ; Petric Maretic et al. , 2019 ) or relying on the Gromov-Wasserstein ( GW ) distance ( Mémoli , 2011 ; Sturm , 2012 ) . The latter approach is a variant of the classical OT in which one aims at comparing probability distributions whose supports lie on different metric spaces by finding a matching of these distributions being as close as possible to an isometry . By expressing graphs as probability measures over spaces specific to their topology , where the structure of a graph is represented through a pairwise distance/similarity matrix between its nodes , the GW distance computes both a soft assignment matrix between the nodes of the two graphs and a notion of distance between them ( see the left part of Figure 1 ) . These properties have proven to be useful for a wide range of tasks such as graph matching and partitioning ( Xu et al. , 2019a ; Chowdhury & Needham , 2021 ; Vayer et al. , 2019 ) , estimation of nonparametric graph models ( graphons , Diaconis & Janson , 2007 ; Xu et al. , 2021 ) or for graph dictionary learning ( Vincent-Cuaz et al. , 2021 ; Xu , 2020 ) . GW was also extended to directed graphs in Chowdhury & Mémoli ( 2019 ) and to labeled graphs via the Fused Gromov-Wasserstein ( FGW ) distance in Vayer et al . ( 2020 ) . Despite those recent successes , applications of GW to graph modeling still have several limitations . First , finding the GW distance remains challenging , as it boils down to solving a difficult nonconvex quadratic program ( Peyré et al. , 2016 ; Solomon et al. , 2016 ; Vayer et al. , 2019 ) , which , in practice , limits the size of the graphs that can be processed . A second limit naturally emerges when a probability mass function is introduced over the set of the graph nodes . The mass associated with a node refers to its relative importance and , without prior knowledge , each node is either assumed to share the same probability mass or to have one proportional to its degree . In this paper , we somehow argue that these choices can be suboptimal in several cases , and should be relaxed , with the additional benefit of lowering the computational complexity . As an illustration , consider Figure 1 : on the left image , the GW matching is given between two graphs , with respectively two and three clusters , associated with uniform weights on the nodes . By relaxing the weight constraints over the second ( middle image ) or first graph ( right image ) we obtain different matchings , that can better preserve the structure of the source graph by reweighing the target nodes and thus selecting a meaningful subgraph . Contributions . We introduce a new optimal transport based divergence between graphs derived from the GW distance . We call it the semi-relaxed Gromov-Wasserstein ( srGW ) divergence . After discussing its properties and motivating its use in ML applications , we propose an efficient solver for the corresponding optimization problem . Our solver better fits to modern parallel programming than exact solvers for GW do . We empirically demonstrate the relevance of our divergence for graph partitioning , Dictionary Learning ( DL ) , clustering of graphs and graph completion tasks . With srGW , we recover SOTA performances at a significantly lower computational cost compared to methods based on pure GW . 2 MODELING GRAPHS WITH THE GROMOV-WASSERSTEIN DIVERGENCE . In this section we introduce more formally the GW distance and discuss two of its applications on graphs , namely graph partitioning and unsupervised graph representation learning . In the following the probability simplex with N-bins is denoted as ΣN : = { h ∈ RN+ | ∑ i hi = 1 } . GW as graphs similarity . In the OT context , a graph G with n nodes can be modeled as a couple ( C , h ) where C ∈ Rn×n is a matrix encoding the connectivity between nodes and h ∈ Σn is a histogram , referred here as distribution , modeling the relative importance of the nodes within the graph . The matrix C can be arbitrarily chosen as the graph adjacency matrix , or any other matrix describing the relationships between nodes in the topology of the graph ( e.g . adjacency , shortest-path , Laplacian ) . The distribution is often considered as uniform ( h = 1n1n ) but can also convey prior knowledge e.g . using the normalized degree distribution ( Xu et al. , 2019a ) . Consider now two graphs G = ( C , h ) and G = ( C , h ) , respectively with n and m nodes , potentially different ( n 6= m ) . The GW distance between G and G is defined as the result of the following optimization problem : GW22 ( C , h , C , h ) = min T1m=h T > 1n=h ∑ ijkl ∣∣Cij − Ckl∣∣2 TikTjl with T ∈ Rn×m+ ( 1 ) The optimal coupling T ? acts as a probabilistic matching of nodes which tends to associate pairs of nodes that have similar pairwise relations in C and C respectively , while preserving masses h and h through its marginals constraints . GW defines a distance on the space of metric measure spaces ( mm-spaces ) invariant to measure preserving isometries ( Mémoli , 2011 ; Sturm , 2012 ) . In the case of graphs , which corresponds to discrete mm-spaces , such invariant is a permutation of the nodes that preserves the structures and the weights of the graphs ( Vayer et al. , 2020 ) . First applications of GW to ML on graphics problems ( Peyré et al. , 2016 ; Solomon et al. , 2016 ) motivated further connections with graph partitioning . In the GW sense , this paradigm is illustrated by finding an ideally partitioned graph G = ( D , h ) of m clustrers whose structure is a diagonal matrix D ∈ Rm×m , representing the cluster ’ s connections , and its distribution h estimates the proportion of the nodes in each cluster ( Xu et al. , 2019a ) . The OT plan between a graph G = ( C , h ) represented by its adjacency matrix C to this ideal graph can be used to recover G ’ s clusters , if their number and proportions match the number and the weights of G ’ s nodes . Xu et al . ( 2019a ) suggested to empirically refine both graph distributions , by choosing h based on power-law transformations of the degree distribution of G and to deduce h from h by linear interpolation . Chowdhury & Needham ( 2021 ) proposed to use heat kernels from the Laplacian of G instead of its adjacency binary representation , and proved that the resulting GW partitioning is closely related to the well-known spectral clustering ( Fiedler , 1973 ) . The GW distance has been extended to graphs with node attributes ( typically Rd vectors ) thanks to the Fused Gromov-Wasserstein distance ( FGW ) ( Vayer et al. , 2019 ) . In this context , an attributed graph G with n nodes is a tuple G = ( C , F , h ) where F ∈ Rn×d is its matrix of node features . FGW between two attributed graphs aims at finding an OT by minimizing a weighted sum of a GW cost on structures and a Wasserstein cost on features ( balanced by a parameter α ∈ [ 0 , 1 ] ) . Most of the applications of GW can be naturally extended with FGW to attributed graphs . GW for Unsupervised Graph Representation Learning . More recently , GW has been used as a data fitting term for unsupervised graph representation learning by means of Dictionary Learning ( DL ) ( Mairal et al. , 2009 ; Schmitz et al. , 2018 ) . While DL methods mainly focus on vectorial data ( Ng et al. , 2002 ; Candès et al. , 2011 ; Bobadilla et al. , 2013 ) , DL applied to graphs datasets consists in factorizing them as composition of graph primitives ( or atoms ) encoded as { ( Ck , hk ) } k∈ [ [ K ] ] . The first approach proposed by ( Xu , 2020 ) consists in a non-linear DL based on entropic GW barycenters . On the other hand , Vincent-Cuaz et al . ( 2021 ) proposed a linear DL method by modeling graphs as a linear combination of graph atoms thus reducing the computational cost . In both cases , the embedding problem that consists in the projection of any graph on the learned graph subspace requires solving a computationally intensive optimization problem .

Summary: The paper proposed a relaxed version of the Gromov-Wasserstein (GW) distance between two probability distributions on (graph) structured data. The main idea of new divergence is to relax the target distribution via an optimization setting. Based on the proposed divergence, the paper also introduced new formulations for existing problems of learning with graphs such as graph clustering (partitioning), graph dictionary learning, clustering of the graph, and graph completion. The experimental results for those applications on graph data have shown the efficiency of the proposed divergence.