Learning Latent Topology for Graph Matching

1 INTRODUCTION . Being a long standing NP-hard problem ( Loiola et al. , 2007 ) , graph matching ( GM ) has received persistent attention from the machine learning and optimization communities for many years . Concretely , for two graphs with n nodes for each , graph matching seeks to solve1 : max z z > Mz s.t . Z ∈ { 0 , 1 } n×n , Hz = 1 ( 1 ) where the affinity matrix M ∈ Rn 2×n2 + encodes node ( diagonal elements ) and edge ( off-diagonal ) affinities/similarities and z is the column-wise vectorization form of the permutation matrix Z. H is a selection matrix ensuring each row and column of Z summing to 1 . 1 is a column vector filled with 1 . Eq . ( 1 ) is the so-called quadratic assignment problem ( QAP ) ( Cho et al. , 2010 ) . Maximizing Eq . ( 1 ) amounts to maximizing the sum of the similarity induced by matching vector Z . While Eq . ( 1 ) does not encode the topology of graphs , Zhou & Torre ( 2016 ) further propose to factorize M to explicitly incorporate topology matrix , where a connectivity matrix A ∈ { 0 , 1 } n×n is used to indicate the topology of a single graph ( Aij = 1 if there exists an edge between nodes i and j ; Aij = 0 otherwise ) . To ease the computation , Eq . ( 1 ) is typically relaxed by letting z ∈ [ 0 , 1 ] n 2 and keeping other parts of Eq . ( 1 ) intact . Traditional solvers to such relaxed problem generally fall into the categories of iterative update ( Cho et al. , 2010 ; Jiang et al. , 2017 ) or numerical continuation ( Zhou & Torre , 2016 ; Yu et al. , 2018 ) , where the solvers are developed under two key assumptions : 1 ) Affinity M is pre-computed with some non-negative metrics , e.g . Gaussian kernel , L2-distance or Manhattan distance ; 2 ) Graph topology is pre-defined as input either in dense ( Schellewald & Schnörr , 2005 ) or sparse ( Zhou & Torre , 2016 ) fashion . There have been several successful attempts towards adjusting the first assumption by leveraging the power of deep networks to learn more effective graph representation for GM ( Wang et al. , 2019a ; Yu et al. , 2020 ; Fey et al. , 2020 ) . However , to our best knowledge , there is little previous work questioning and addressing the problem regarding the second assumption in the context of learning-based graph matching2 . For example , existing 1Without loss of generality , we discuss graph matching under the setting of equal number of nodes without outliers . The unequal case can be readily handled by introducing extra constraints or dummy nodes . Bipartite matching and graph isomorphism are subsets of this quadratic formulation ( Loiola et al. , 2007 ) . 2There are some loosely related works ( Du et al. , 2019 ; 2020 ) on network alignment and link prediction without learning , which will be discussed in detail in the related works . standard pipeline of keypoint matching in computer vision will construct initial topology by Delaunay triangulation or k-nearest neighbors . Then this topology will be freezed throughout the subsequent learning and matching procedures . In this sense , the construction of graph topology is peeled from matching task as a pre-processing stage . More examples can be found beyond the vision communities such as in social network alignment ( Zhang & Tong , 2016 ; Heimann et al. , 2018 ; Xiong & Yan , 2020 ) assuming fixed network structure for individual node matching in two networks . We argue that freezing graph topology for matching can hinder the capacity of graph matching solvers . For a pre-defined graph topology , the linked nodes sometimes result in less meaningful interaction , especially under the message-passing mechanism in graph neural networks ( Kipf & Welling , 2017 ) . We give a schematic demonstration in Fig . 1 . Though some earlier attempts ( Cho & Lee , 2012 ; Cho et al. , 2013 ) seek to adjust the graph topology under traditional non-deep learning setting , such procedures can not be readily integrated into end-to-end deep learning frameworks due to undifferentiable nature . Building upon the hypothesis that there exists some latent topology better than heuristically created one for GM , our aim is to learn it ( or its distribution ) for GM . Indeed , jointly solving matching and graph topology learning can be intimidating due to the combinatorial nature , which calls for more advanced approaches . In this paper , we propose an end-to-end framework to jointly learn the latent graph topology and perform GM , termed as deep latent graph matching ( DLGM ) . We leverage the power of graph generative model to automatically produce graph topology from given features and their geometric relations , under specific locality prior . Different from generative learning on singleton graphs ( Kipf & Welling , 2016 ; Bojchevski et al. , 2018 ) , our graph generative learning is performed in a pairwise fashion , leading to a novel matching-guided generative paradigm . The source code will be made publicly available . Contributions : 1 ) We explore a new direction for more flexible GM by actively learning latent topology , in contrast to previous works using fixed topology as input ; 2 ) Under this setting , we propose a deterministic optimization approach to learn graph topology for matching ; 3 ) We further present a generative way to produce latent topology under a probabilistic interpretation by ExpectationMaximization . This framework can also adapt to other problems where graph topology is the latent structure to infer ; 4 ) Our method achieves state-ofthe-art performance on public benchmarks . 2 RELATED WORKS . In this section , we first discuss existing works for graph topology and matching updating whose motivation is a bit similar to ours while the technique is largely different . Then we discuss relevant works in learning graph matching and generative graph models from the technical perspective . Topology updating and matching . There are a few works for joint graph topology updating and matching , in the context of network alignment . Specifically , given two initial networks for matching , Du et al . ( 2019 ) show how to alternatively perform link prediction within each network and node matching across networks based on the observation that these two tasks can benefit to each other . In their extension ( Du et al. , 2020 ) , a skip-gram embedding framework is further established under the same problem setting . In fact , these works involve a random-walk based node embedding updating and classification based link prediction modules and the whole algorithm runs in a one-shot optimization fashion . There is neither explicit training dataset nor trained matching model ( except for the link classifier ) , which bears less flavor of machine learning . In contrast , our method involves training an explicit model for topology recovery and matching solving . Specifically , our deterministic technique ( see Sec . 3.4.1 ) solves graph topology and matching in one-shot , while the proposed generative method alternatively estimates the topology and matching ( see Sec . 3.4.2 ) . Our approach allows to fully leverage multiple training samples in many applications like computer vision to boost the performance on test set . Moreover , the combinatorial nature of the matching problem is not addressed in ( Du et al. , 2019 ; 2020 ) , and they adopt a greedy selection strategy instead . While we develop a principled combinatorial learning approach to this challenge . Also their methods rely on a considerable amount of seed matchings , yet this paper directly learns the latent topology from scratch which is more challenging and seldom studied . Learning of graph matching . Early non-deep learning-based methods seek to learn effective metric ( e.g . weighted Euclid distance ) for node and edge features or affinity kernel ( e.g . Gaussian kernel ) in a parametric fashion ( Caetano et al. , 2009 ; Cho et al. , 2013 ) . Recent deep graph matching methods have shown how to extracte more dedicated feature representation . The work ( Zanfir & Sminchisescu , 2018 ) adopts VGG16 ( Simonyan & Zisserman , 2014 ) as the backbone for feature extraction on images . Other efforts have been witnessed in developing more advanced pipelines , where graph embedding ( Wang et al. , 2019a ; Yu et al. , 2020 ; Fey et al. , 2020 ) and geometric learning ( Zhang & Lee , 2019 ; Fey et al. , 2020 ) are involved . Rolı́nek et al . ( 2020 ) study the way of incorporating traditional non-differentiable combinatorial solvers , by introducing a differentiatiable blackbox GM solver ( Pogancic et al. , 2020 ) . Recent works in tackling combinatorial problem with deep learning ( Huang et al. , 2019 ; Kool & Welling , 2018 ) also inspire developing combinatorial deep solvers , for GM problems formulated by both Koopmans-Beckmann ’ s QAP ( Nowak et al. , 2018 ; Wang et al. , 2019a ) and Lawler ’ s QAP ( Wang et al. , 2019b ) . Specifically , Wang et al . ( 2019a ) devise a permutation loss for supervised learning , with an improvement in Yu et al . ( 2020 ) via Hungarian attention . Wang et al . ( 2019b ) solve the most general Lawler ’ s QAP with graph embedding technique . Generative graph model . Early generative models for graph can date back to ( Erdos & Renyi , 1959 ) , in which edges are generated with fixed probability . Recently , Kipf & Welling ( 2016 ) present a graph generative model by re-parameterizing the edge probability from Gaussian noise . Johnson ( 2017 ) propose to generate graph in an incremental fashion , and in each iteration a portion of the graph is produced . Gómez-Bombarelli et al . ( 2018 ) utilized recurrent neural network to generate graph from a sequence of molecule representation . Adversarial graph generation is considered in ( Pan et al. , 2018 ; Wang et al. , 2018 ; Bojchevski et al. , 2018 ) . Specifically , Wang et al . ( 2018 ) ; Bojchevski et al . ( 2018 ) seek to unify graph generative model and generative adversarial networks . In parallel , reinforcement learning has been adopted to generate discrete graphs ( De Cao & Kipf , 2018 ) . 3 LEARNING LATENT TOPOLOGY FOR GM . In this section , we describe details of the proposed framework with two specific algorithms derived from deterministic and generative perspectives , respectively . Both algorithms are motivated by the hypothesis that there exists some latent topology more suitable for matching rather than a fixed one . Note the proposed deterministic algorithm performs a standard forward-backward pass to jointly learn the topology and matching , while our generative algorithm consists of an alternative optimization procedure between estimating latent topology and learning matching under an Expectation-Maximization ( EM ) interpretation . In general , the generative algorithm assumes that a latent topology is sampled from a latent distribution , where the expected matching accuracy sufficing this distribution is maximized . Therefore , we expect to learn a topology generator sufficing such distribution . We reformulate GM into Bayesian fashion for consistent discussion in Sec . 3.1 , detail deterministic/generative latent module in Sec . 3.2 and discuss the loss functions from a probabilistic perspective in Sec . 3.3 . We finally elaborate on the holistic framework and the optimization procedure for both algorithms ( deterministic and generative ) in Sec . 3.4 .

Summary: The authors address the problem of discrete keypoint matching. For an input pair of images, the task is to match the unannotated (but given as part of the input) keypoints. The main contribution is identifying the bottleneck of the current SOTA algorithm: a fixed connectivity construction given by Delauney triangulation. By replacing this with an end-to-end learnable algorithm, they outperform SOTA with a decent margin.