Learning deep graph matching with channel-independent embedding and Hungarian attention

1 INTRODUCTION . Without loss of generality , we consider the bijection problem for graph matching : given graph G1 and G2 of equal size n , graph matching seeks to find the one-vs-one node correspondence1 : max x x > Kx s.t . Px = 1 ( 1 ) where x = vec ( X ) ∈ { 0 , 1 } n2 which is the column-wise vectorized form of the permutation matrix X that encodes the node-to-node correspondence between two graphs , and K ∈ Rn 2×n2 + is the so-called affinity matrix2 , respectively . Note P is a selection matrix encoding the one-to-one correspondence constraint . This problem is called Lawler ’ s QAP ( Lawler , 1963 ) and has attracted enormous attention for its generally NP-complete ( Hartmanis , 1982 ) challenge , as well as a wide spectrum of applications in computer vision , graphics , machine learning and operational research etc . In particular , Koopmans-Beckmann ’ s QAP ( Loiola et al. , 2007 ) with objective tr ( X > F1XF2 ) is a special case of Eq . ( 1 ) , which can be converted to Lawler ’ s QAP by K = F2⊗F1 and Fi refers to the weighted adjacency matrix . A series of solvers haven been developed to solve graph matching problem ( Leordeanu & Hebert , 2005 ; Cho et al. , 2010 ; Bernard et al. , 2018 ; Yan et al. , 2015 ; Yu et al. , 2018 ) . All these methods are based on deterministic optimization , which are conditioned with pre-defined affinity matrix and no learning paradigm is involved . This fact greatly limits the performance and broad application w.r.t . different problem settings considering its NP-hard nature . Recently , the seminal work namely deep graph matching ( DGM ) ( Zanfir & Sminchisescu , 2018 ) is proposed to exploit the high capacity of deep networks for graph matching , which achieves stateof-the-art performance . This is in contrast to some early works which incorporate learning strategy 1We assume graphs are of equal size for narrative simplicity . One can easily handle unbalanced graph size by adding dummy nodes as a common protocol in graph matching literature ( Cho et al. , 2010 ) . 2Aia : jb typically encodes the affinity between pair ( i , j ) and ( a , b ) where node i , j ∈ G1 and a , b ∈ G2 . separately in local stages ( Caetano et al. , 2009 ; Cho et al. , 2013 ) . On the other hand , Graph Convolutional Networks ( GCN ) ( Kipf & Welling , 2017 ) brings about new capability on tasks over graph-like data , as it naturally integrates the intrinsic graph structure in a general updating rule : H ( l+1 ) = σ ( ÂH ( l ) W ( l ) ) ( 2 ) where Â is the normalized connectivity matrix . H ( l ) and W ( l ) are the features and weights at layer l , respectively . Node embedding is updated by aggregation from 1-neighboring nodes , which is akin to the convolution operator in CNN . By taking advantages of both DGM and GCN , Wang et al . ( 2019 ) and Zhang & Lee ( 2019 ) incorporate permutation loss instead of displacement loss in ( Zanfir & Sminchisescu , 2018 ) , with notable improvement across both synthetic and real data . Note that Eq . ( 1 ) involves both node and edge information , which exactly correspond to the diagonal and off-diagonal elements in K , respectively . Edges can carry informative multi-dimensional attributes ( namely weights ) which are fundamental to graph matching . However existing embedding based graph matching methods ( Wang et al. , 2019 ; Xu et al. , 2019 ) are focused on the explicit modeling of node level features , whereby the edges are only used as topological node connection for message passing in GCN . Besides , edge attributes are neither well modeled in the embedding-free model ( Zanfir & Sminchisescu , 2018 ) since the edge information is derived from the concatenation of node features . To our best knowledge , there is no deep graph matching method explicitly incorporating edge attributes . In contrast , edge attributes e.g . length and orientation are widely used in traditional graph matching models ( Cho et al. , 2010 ; Yan et al. , 2015 ; Yu et al. , 2018 ) for constructing the affinity matrix K. Such a gap shall be filled in the deep graph matching pipeline . Another important consideration refers to the design of loss function . There are mainly two forms in existing deep graph matching works : i ) displacement loss ( Zanfir & Sminchisescu , 2018 ) similar to the use in optical flow estimation ( Ren et al. , 2017 ) ; ii ) the so-called permutation loss ( Wang et al. , 2019 ) involving iterative Sinkhorn procedure followed by a cross-entropy loss . Results in ( Wang et al. , 2019 ) show the latter is an effective improvement against the former regression based loss . However , we argue that the continuous Sinkhorn procedure ( in training stage ) is yet an unnatural approximation to Hungarian sampling ( in testing stage ) for discretization . If the network is equipped with a continuous loss function ( e.g . cross-entropy ) , we argue that the training process will make a great “ meaningless effort ” to enforce some network output digits of the final matching matrix into binary and neglect the resting digits which might have notable impact on accuracy . This paper strikes an endeavor on the above two gaps and makes the following main contributions : i ) We propose a new approach for edge embedding via channel-wise operation , namely channelindependent embedding ( CIE ) . The hope is to effectively explore the edge attribute and simulate the multi-head strategy in attention models ( Veličković et al. , 2018 ) by decoupling the calculations parallel and orthogonal to channel direction . In fact , edge attribute information has not been considered in existing embedding based graph matching methods ( Wang et al. , 2019 ; Xu et al. , 2019 ) . ii ) We devise a new mechanism to adjust the loss function based on the Hungarian method which is widely used for linear assignment problem , as termed by Hungarian attention . It resorts to dynamically generating sparse matching mask according to Hungarian sampling during training , rather than approximating Hungarian sampling with a differentiable function . As such , the Hungarian attention introduces higher smoothness against traditional loss functions to ease the training . iii ) The empirical results on three public benchmarks shows that the two proposed techniques are orthogonal and beneficial to existing techniques . Specifically , on the one hand , our CIE module can effectively boost the accuracy by exploring the edge attributes which otherwise are not considered in state-of-the-art deep graph matching methods ; on the other hand , our Hungarian attention mechanism also shows generality and it is complementary to existing graph matching loss . 2 RELATED WORKS . Graph embedding . To handle graph-like data , early works adopt recursive neural networks ( RNNs ) treating input as directed acyclic graphs ( Sperduti & Starita , 1997 ; Frasconi et al. , 1998 ) . Gori et al . ( 2005 ) ; Scarselli et al . ( 2008 ) generalized early models to graph neural networks ( GNNs ) so as to be directly applied on cyclic , directed or undirected graphs . Li et al . ( 2016 ) further improved this line of model by replacing standard RNNs with gated recurrent units ( GRUs ) ( Cho et al. , 2013 ) . Inspired by the great success of convolutional neural networks ( CNNs ) ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ) , researchers have made tremendous effort on applying convolution operator to graphs ( Bruna et al. , 2014 ; Kipf & Welling , 2017 ; Gong & Cheng , 2019 ) . Bruna et al . ( 2014 ) defined a convolution operator in Fourier domain which is obtained by performing eigen-decomposition on graph Laplacian . However , such convolution will affect the whole spatial domain once taking inverse Fourier transformation . This method was improved by Chebyshev expansion to approximate filters ( Defferrard et al. , 2016 ) . Kipf & Welling ( 2017 ) propose a graph convolutional operator over 1-neighbor nodes derived from graph spectral theory , which is invariant to node permutation and achieved significant performance on semi-supervised learning tasks . There are series of works following GCN , such as GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Veličković et al. , 2018 ) and MPNN ( Gilmer et al. , 2017 ) . Refer to ( Cai et al. , 2018 ) for a more comprehensive survey . While the aforementioned models are focused on learning node state/embedding , a parallel line of work seek to learn edge embedding by taking into account the information carried on edges ( Li et al. , 2016 ; Gilmer et al. , 2017 ; Gong & Cheng , 2019 ) . Edges are intrinsic portion of graphs , and thus edge embedding can be essential to reveal the relation among nodes . Gilmer et al . ( 2017 ) introduce a general embedding network incorporating edge information and node-edge information merging , and a serious of works fall into this framework e.g . Gated GNN ( Li et al. , 2016 ) , Tensor GNN ( Schütt et al. , 2017 ) and EGNN ( Gong & Cheng , 2019 ) . An improved version is devised in Chen et al . ( 2019 ) by interpreting this framework as maximizing mutual information across layers . Loss for combinatorial learning . For the relatively easy linear assignment problem , it has been known that Sinkhorn algorithm ( Sinkhorn , 1964 ) is the approximate and differentiable version of Hungarian algorithm ( Mena et al. , 2017 ) . The Sinkhorn Network ( Adams & Zemel , 2011 ) is developed given known assignment cost , whereby doubly-stochastic regulation is performed on input non-negative square matrix . Patrini et al . ( 2018 ) devise the Sinkhorn AutoEncoder to minimize Wasserstein distance , and Emami & Ranka ( 2018 ) propose to learning a linear assignment solver via reinforcement learning . For permutation prediction , DeepPermNet ( Santa Cruz et al. , 2018 ) adopts the Sinkhorn layer on top of a deep convolutional network . However this method can not be directly applied for graph matching as it is not invariant to input permutations which is conditioned on a predefined node permutation as reference . In particular , existing supervised methods on combinatorial learning are generally cross-entropy-based . Pointer Net ( Vinyals et al. , 2015 ) incorporates cross-entropy loss on learning heuristics for combinatorial problems . Milan et al . ( 2017 ) propose an objective-based loss , where the gradients are only updated if the objective improves after update . Learning for graph matching . The early effort ( Caetano et al. , 2009 ) aims to incorporate learning to graph matching . The key is to learn a more effective affinity function with given correspondence as supervision . While the ability by only learning affinity is limited , Cho et al . ( 2013 ) propose a matching function learning paradigm using histogram-based attributes with Structured-SVM ( Tsochantaridis et al. , 2005 ) . A recent work ( Zanfir & Sminchisescu , 2018 ) is a breakthrough to introduce deep learning paradigm into graph matching task , which utilizes a neural network to learn the affinity function . The learning procedure is explicitly derived from the factorization of affinity matrix ( Zhou & De la Torre , 2012 ) , which makes the interpretation of the network behavior possible . However , the displacement loss in ( Zanfir & Sminchisescu , 2018 ) measures the pixel-wise translation which is similar to optical-flow ( Dosovitskiy et al. , 2015 ) , being essentially a regression task instead of combinaotiral optimization . Seeing this limitation , Wang et al . ( 2019 ) employ elementwise binary cross-entropy , termed as permutation loss . This loss has proved capable of capturing the combinatorial nature rather than pixel offset , and achieves improvement over displacement loss . Node embedding is also used in ( Wang et al. , 2019 ) to explore the structure information .

Summary: The authors proposed a new way to train graph siamese networks for the graph matching problem. The overall framework of this paper is somehow similar to [1] and [2], except for the final Hungarian attention module, which is the key contribution of this paper. In the current settings, the authors proved that using their Hungarian attention module, the performance can be improved. However, it would be better if the Hungarian attention can be applied to DGCNN in [1] and CMPNN in [2]. It would be good if the author can do an extra experiment to apply the Hungarian attention module to these two modules. Also, the authors may want to add some discussion about these two papers in the related works section (both papers do have published their codes).