Deep Graph Matching Consensus

1 INTRODUCTION . Graph matching refers to the problem of establishing meaningful structural correspondences of nodes between two or more graphs by taking both node similarities and pairwise edge similarities into account ( Wang et al. , 2019b ) . Since graphs are natural representations for encoding relational data , the problem of graph matching lies at the heart of many real-world applications . For example , comparing molecules in cheminformatics ( Kriege et al. , 2019b ) , matching protein networks in bioinformatics ( Sharan & Ideker , 2006 ; Singh et al. , 2008 ) , linking user accounts in social network analysis ( Zhang & Philip , 2015 ) , and tracking objects , matching 2D/3D shapes or recognizing actions in computer vision ( Vento & Foggia , 2012 ) can be formulated as a graph matching problem . The problem of graph matching has been heavily investigated in theory ( Grohe et al. , 2018 ) and practice ( Conte et al. , 2004 ) , usually by relating it to domain-agnostic distances such as the graph edit distance ( Stauffer et al. , 2017 ) and the maximum common subgraph problem ( Bunke & Shearer , 1998 ) , or by formulating it as a quadratic assignment problem ( Yan et al. , 2016 ) . Since all three approaches are NP-hard , solving them to optimality may not be tractable for large-scale , real-world instances . Moreover , these purely combinatorial approaches do not adapt to the given data distribution and often do not consider continuous node embeddings which can provide crucial information about node semantics . Recently , various neural architectures have been proposed to tackle the task of graph matching ( Zanfir & Sminchisescu , 2018 ; Wang et al. , 2019b ; Zhang & Lee , 2019 ; Xu et al. , 2019d ; b ; Derr et al. , 2019 ; Zhang et al. , 2019a ; Heimann et al. , 2018 ) or graph similarity ( Bai et al. , 2018 ; 2019 ; Li et al. , 2019 ) in a data-dependent fashion . However , these approaches are either only capable of computing similarity scores between whole graphs ( Bai et al. , 2018 ; 2019 ; Li et al. , 2019 ) , rely on an inefficient global matching procedure ( Zanfir & Sminchisescu , 2018 ; Wang et al. , 2019b ; Xu et al. , 2019d ; Li et al. , 2019 ) , or do not generalize to unseen graphs ( Xu et al. , 2019b ; Derr et al. , 2019 ; Zhang et al. , 2019a ) . Moreover , they might be prone to match neighborhoods between graphs 3Correspondence to matthias.fey @ udo.edu 4Work done during an internship at NNAISENSE inconsistently by only taking localized embeddings into account ( Zanfir & Sminchisescu , 2018 ; Wang et al. , 2019b ; Zhang & Lee , 2019 ; Xu et al. , 2019d ; Derr et al. , 2019 ; Heimann et al. , 2018 ) . Here , we propose a fully-differentiable graph matching procedure which aims to reach a data-driven neighborhood consensus between matched node pairs without the need to solve any optimization problem during inference . In addition , our approach is purely local , i.e. , it operates on fixed-size neighborhoods around nodes , and is sparsity-aware , i.e. , it takes the sparsity of the underlying structures into account . Hence , our approach scales well to large input domains , and can be trained in an end-to-end fashion to adapt to a given data distribution . Finally , our approach improves upon the state-of-the-art on several real-world applications from the fields of computer vision and entity alignment on knowledge graphs . 2 PROBLEM DEFINITION . A graph G = ( V , A , X , E ) consists of a finite set of nodes V = { 1 , 2 , . . . } , an adjacency matrix A ∈ { 0 , 1 } |V|×|V| , a node feature matrix X ∈ R|V|×· , and an optional ( sparse ) edge feature matrix E ∈ R|V|×|V|×· . For a subset of nodes S ⊆ V , G [ S ] = ( S , AS , S , XS , : , ES , S , : ) denotes the subgraph of G induced by S. We refer toNT ( i ) = { j ∈ V : d ( i , j ) ≤ T } as the T -hop neighborhood around node i ∈ V , where d : V × V → N denotes the shortest-path distance in G. A node coloring is a function V → Σ with arbitrary codomain Σ . The problem of graph matching refers to establishing node correspondences between two graphs . Formally , we are given two graphs , a source graph Gs = ( Vs , As , Xs , Es ) and a target graph Gt = ( Vt , At , Xt , Et ) , w.l.o.g . |Vs| ≤ |Vt| , and are interested in finding a correspondence matrix S ∈ { 0 , 1 } |Vs|×|Vt| which minimizes an objective subject to the one-to-one mapping constraints∑ j∈Vt Si , j = 1 ∀i ∈ Vs and ∑ i∈Vs Si , j ≤ 1 ∀j ∈ Vt. As a result , S infers an injective mapping π : Vs → Vt which maps each node in Gs to a node in Gt . Typically , graph matching is formulated as an edge-preserving , quadratic assignment problem ( Anstreicher , 2003 ; Gold & Rangarajan , 1996 ; Caetano et al. , 2009 ; Cho et al. , 2013 ) , i.e. , argmax S ∑ i , i′∈Vs j , j′∈Vt A ( s ) i , i′A ( t ) j , j′Si , jSi′ , j′ ( 1 ) subject to the one-to-one mapping constraints mentioned above . This formulation is based on the intuition of finding correspondences based on neighborhood consensus ( Rocco et al. , 2018 ) , which shall prevent adjacent nodes in the source graph from being mapped to different regions in the target graph . Formally , a neighborhood consensus is reached if for all node pairs ( i , j ) ∈ Vs × Vt with Si , j = 1 , it holds that for every node i′ ∈ N1 ( i ) there exists a node j′ ∈ N1 ( j ) such that Si′ , j′ = 1 . In this work , we consider the problem of supervised and semi-supervised matching of graphs while employing the intuition of neighborhood consensus as an inductive bias into our model . In the supervised setting , we are given pair-wise ground-truth correspondences for a set of graphs and want our model to generalize to unseen graph pairs . In the semi-supervised setting , source and target graphs are fixed , and ground-truth correspondences are only given for a small subset of nodes . However , we are allowed to make use of the complete graph structures . 3 METHODOLOGY . In the following , we describe our proposed end-to-end , deep graph matching architecture in detail . See Figure 1 for a high-level illustration . The method consists of two stages : a local feature matching procedure followed by an iterative refinement strategy using synchronous message passing networks . The aim of the feature matching step , see Section 3.1 , is to compute initial correspondence scores based on the similarity of local node embeddings . The second step is an iterative refinement strategy , see Sections 3.2 and 3.3 , which aims to reach neighborhood consensus for correspondences using a differentiable validator for graph isomorphism . Finally , in Section 3.4 , we show how to scale our method to large , real-world inputs . 3.1 LOCAL FEATURE MATCHING . We model our local feature matching procedure in close analogy to related approaches ( Bai et al. , 2018 ; 2019 ; Wang et al. , 2019b ; Zhang & Lee , 2019 ; Wang & Solomon , 2019 ) by computing similarities between nodes in the source graph Gs and the target graph Gt based on node embeddings . That is , given latent node embeddings Hs = Ψθ1 ( Xs , As , Es ) ∈ R|Vs|×· and Ht = Ψθ1 ( Xt , At , Et ) ∈ R|Vt|×· computed by a shared neural network Ψθ1 for source graph Gs and target graph Gt , respectively , we obtain initial soft correspondences as S ( 0 ) = sinkhorn ( Ŝ ( 0 ) ) ∈ [ 0 , 1 ] |Vs|×|Vt| with Ŝ ( 0 ) = HsH > t ∈ R|Vs|×|Vt| . Here , sinkhorn normalization is applied to obtain rectangular doubly-stochastic correspondence matrices that fulfill the constraints ∑ j∈Vt Si , j = 1 ∀i ∈ Vs and ∑ i∈Vs Si , j ≤ 1 ∀j ∈ Vt ( Sinkhorn & Knopp , 1967 ; Adams & Zemel , 2011 ; Cour et al. , 2006 ) . We interpret the i-th row vector S ( 0 ) i , : ∈ [ 0 , 1 ] |Vt| as a discrete distribution over potential correspondences in Gt for each node i ∈ Vs. We train Ψθ1 in a dicriminative , supervised fashion against ground truth correspondences πgt ( · ) by minimizing the negative log-likelihood of correct correspondence scores L ( initial ) = −∑i∈Vs log ( S ( 0 ) i , πgt ( i ) ) . We implement Ψθ1 as a Graph Neural Network ( GNN ) to obtain localized , permutation equivariant vectorial node representations ( Bronstein et al. , 2017 ; Hamilton et al. , 2017 ; Battaglia et al. , 2018 ; Goyal & Ferrara , 2018 ) . Formally , a GNN follows a neural message passing scheme ( Gilmer et al. , 2017 ) and updates its node features ~h ( t−1 ) i in layer t by aggregating localized information via ~a ( t ) i = AGGREGATE ( t ) ( { ( ~h ( t−1 ) j , ~ej , i ) : j ∈ N1 ( i ) } ) , ~h ( t ) i = UPDATE ( t ) ( ~h ( t−1 ) i , ~a ( t ) i ) ( 2 ) where ~h ( 0 ) i = ~xi ∈ X and { { . . . } } denotes a multiset . The recent work in the fields of geometric deep learning and relational representation learning provides a large number of operators to choose from ( Kipf & Welling , 2017 ; Gilmer et al. , 2017 ; Veličković et al. , 2018 ; Schlichtkrull et al. , 2018 ; Xu et al. , 2019c ) , which allows for precise control of the properties of extracted features . 3.2 SYNCHRONOUS MESSAGE PASSING FOR NEIGHBORHOOD CONSENSUS . Due to the purely local nature of the used node embeddings , our feature matching procedure is prone to finding false correspondences which are locally similar to the correct one . Formally , those cases pose a violation of the neighborhood consensus criteria employed in Equation ( 1 ) . Since finding a global optimum is NP-hard , we aim to detect violations of the criteria in local neighborhoods and resolve them in an iterative fashion . We utilize graph neural networks to detect these violations in a neighborhood consensus step and iteratively refine correspondences S ( l ) , l ∈ { 0 , . . . , L } , starting from S ( 0 ) . Key to the proposed algorithm is the following observation : The soft correspondence matrix S ∈ [ 0 , 1 ] |Vs|×|Vt| is a map from the node function space L ( Gs ) = L ( R|Vs| ) to the node function space L ( Gt ) = L ( R|Vt| ) . Therefore , we can use S to pass node functions ~xs ∈ L ( Gs ) , ~xt ∈ L ( Gt ) along the soft correspondences by ~x ′t = S > ~xs and ~x ′s = S~xt ( 3 ) to obtain functions ~x ′t ∈ L ( Gt ) , ~x ′s ∈ L ( Gs ) in the other domain , respectively . Then , our consensus method works as follows : Using S ( l ) , we first map node indicator functions , given as an injective node coloring Vs → { 0 , 1 } |Vs| in the form of an identity matrix I|Vs| , from Gs to Gt . Then , we distribute this coloring in corresponding neighborhoods by performing synchronous message passing on both graphs via a shared graph neural network Ψθ2 , i.e. , Os = Ψθ2 ( I|Vs| , As , Es ) and Ot = Ψθ2 ( S > ( l ) I|Vs| , At , Et ) . ( 4 ) We can compare the results of both GNNs to recover a vector ~di , j = ~o ( s ) i −~o ( t ) j which measures the neighborhood consensus between node pairs ( i , j ) ∈ Vs ×Vt . This measure can be used to perform trainable updates of the correspondence scores S ( l+1 ) i , j = sinkhorn ( Ŝ ( l+1 ) ) i , j with Ŝ ( l+1 ) i , j = Ŝ ( l ) i , j + Φθ3 ( ~dj , i ) ( 5 ) based on an MLP Φθ3 . The process can be applied L times to iteratively improve the consensus in neighborhoods . The final objective L = L ( initial ) + L ( refined ) with L ( refined ) = −∑i∈Vs log ( S ( L ) i , πgt ( i ) ) combines both the feature matching error and neighborhood consensus error . This objective is fullydifferentiable and can hence be optimized in an end-to-end fashion using stochastic gradient descent . Overall , the consensus stage distributes global node colorings to resolve ambiguities and false matchings made in the first stage of our architecture by only using purely local operators . Since an initial matching is needed to test for neighborhood consensus , this task can not be fulfilled by Ψθ1 alone , which stresses the importance of our two-stage approach . The following two theorems show that ~di , j is a good measure of how well local neighborhoods around i and j are matched by the soft correspondence between Gs and Gt . The proofs can be found in Appendix B and C , respectively . Theorem 1 . Let Gs and Gt be two isomorphic graphs and let Ψθ2 be a permutation equivariant GNN , i.e. , P > Ψθ2 ( X , A ) = Ψθ2 ( P > X , P > AP ) for any permutation matrix P ∈ { 0 , 1 } |V|×|V| . If S ∈ { 0 , 1 } |Vs|×|Vt| encodes an isomorphism between Gs and Gt , then ~di , π ( i ) = ~0 for all i ∈ Vs. Theorem 2 . Let Gs and Gt be two graphs and let Ψθ2 be a permutation equivariant and T -layered GNN for which both AGGREGATE ( t ) and UPDATE ( t ) are injective for all t ∈ { 1 , . . . , T } . If ~di , j = ~0 , then the resulting submatrix SNT ( i ) , NT ( j ) ∈ [ 0 , 1 ] |NT ( i ) |×|NT ( j ) | is a permutation matrix describing an isomorphism between the T -hop subgraph Gs [ NT ( i ) ] around i ∈ Vs and the T -hop subgraph Gt [ NT ( j ) ] around j ∈ Vt . Moreover , if ~di , argmaxSi , : = ~0 for all i ∈ Vs , then S denotes a full isomorphism between Gs and Gt . Hence , a GNN Ψθ2 that satisfies both criteria in Theorem 1 and 2 provides equal node embeddings ~o ( s ) i and ~o ( t ) j if and only if nodes in a local neighborhood are correctly matched to each other . A value ~di , j 6= ~0 indicates the existence of inconsistent matchings in the local neighborhoods around i and j , and can hence be used to refine the correspondence score Ŝi , j . Note that both requirements , permutation equivariance and injectivity , are easily fulfilled : ( 1 ) All common graph neural network architectures following the message passing scheme of Equation ( 2 ) are equivariant due to the use of permutation invariant neighborhood aggregators . ( 2 ) Injectivity of graph neural networks is a heavily discussed topic in recent literature . It can be fulfilled by using a GNN that is as powerful as the Weisfeiler & Lehman ( 1968 ) ( WL ) heuristic in distinguishing graph structures , e.g. , by using sum aggregation in combination with MLPs on the multiset of neighboring node features , cf . ( Xu et al. , 2019c ; Morris et al. , 2019 ) .

Summary: This paper proposes a two-stage GNN-based architecture to establish correspondences between two graphs. The first step is to learn node embeddings using a GNN to obtain soft node correspondences between two graphs. The second step is to iteratively refine them using the constraints of matching consensus in local neighborhoods between graphs. The overall refining process resembles the classic graph matching algorithm of graduated assignment (Gold & Rangarajan, 1996), but generalizes it using deep neural representation. Experiments show that the proposed algorithm performs well on real-world tasks of image matching and knowledge graph entity alignment.