Graph Neural Networks For Multi-Image Matching

1 INTRODUCTION . Feature matching is an essential part of Structure from Motion and many geometric computer vision applications . The goal in multi-image feature matching is to take 2D feature positions from three or more images and find which ones correspond to the same point in the 3D scene . Methods such as SIFT feature matching ( Lowe , 2004 ) combined with RANSAC ( Fischler & Bolles , 1981 ) have been the standard for decades . However RANSAC-based approaches are limited to matching pairs of images , which can lead to global inconsistencies in the matching . Other works , such as Wang et al . ( 2018 ) , have shown improvement in performance by optimizing cycle consistency , i.e . enforcing the pairwise feature matches to be globally consistent . However , these multi-view consistency algorithms struggle in distributed and noisy settings . Having image features suited for this task would help improve performance , and deep learning has revolutionized how image features are computed ( Yi et al. , 2016 ) . In this paper , we want to leverage the power of deep representations in order to compute feature descriptors that are robust across multiple views . Unfortunately , there are obstacles to applying multi-view constraints directly to deep learning . Multi-view constraints are formulated in terms of sparse features , which traditional convolutional neural nets are not designed to handle . Thus we will need a new architecture to handle such constraints . More fundamentally , deep neural networks need large amounts of labeled data to train . Consequently unsupervised training is a more practical approach . In the absence of direct supervision , the additional signal of geometric constraints can help disambiguate visually similar features and reject outliers . Thus incorporating such constraints is important in training a network to solve this task . In this work , we address these concerns using Graph Neural Networks ( GNNs ) . The proposed method works directly on the graph of correspondences between the image features , which is agnostic to how the correspondences were computed , thus allowing the algorithm to work in a broad class of environments . To the best of our knowledge this work is the first to apply deep learning to the multi-view feature matching problem . We use an unsupervised loss , the cycle consistency loss , to train the network . We use geometric consistency losses to aid training , though no geometric information is used at inference time . Although our network is simple , it shows promising results compared to baselines which optimize for cycle-consistency without learned embeddings , using a matrix factorization loss ( Zhou et al. , 2015b ; Leonardos et al. , 2017 ) . Furthermore , since inference requires only a single forward pass over the neural network , our approach is faster to achieve comparable accuracy than methods which must solve an optimization problem every time . We perform experiments on the Rome16K dataset ( Li et al. , 2010 ) to test the effectiveness of our method compared to optimization based methods . Our contributions in this work are : • We use a novel architecture to address the multi-image feature matching problem using GNNs with graph embeddings . • We introduce an unsupervised multi-view cycle consistency loss that does not require labeled correspondences to train . • We demonstrate the effectiveness of geometric consistency losses in improving training . 2 RELATED WORK . 2.1 FEATURE MATCHING . Image feature matching has a rich history of research in computer vision . Much work has be done using hand-crafted feature descriptors such as SIFT ( Lowe , 2004 ) , SURF ( Bay et al. , 2006 ) , BRIEF ( Calonder et al. , 2012 ) , or ORB ( Mur-Artal et al. , 2015 ) . RANSAC Fischler & Bolles ( 1981 ) is the most widely used robust estimation technique to filter out outliers from the matches . The combination of RANSAC and hand-crafted feature descriptors has constituted the bulk of the matching literature for the last 40 years . More recently Suh et al . ( 2015 ) and Hu et al . ( 2016 ) have shown graph matching of the features can be added for more robust matches between images . 2.2 MULTI-IMAGE MATCHING . Multi-image matching has traditionally been done using optimization based methods minimizing a cycle consistency based loss ( see Section 3.3 ) . Pachauri et al . ( 2013 ) and Arrigoni et al . ( 2017 ) use the eigenvectors of the matching matrix to obtain a low dimensional embedding . However , the assumption of low Gaussian noise is not realistic . Zhou et al . ( 2015b ) and ? use more sophisticated optimization techniques on the matching matrix and thus produce more robust solutions . Leonardos et al . ( 2017 ) implement a distributed optimization scheme to solve for cycle consistency . Swoboda et al . ( 2019 ) implement a convex relaxation of the low dimensional embedding problem . Shi et al . ( 2016 ) use tensor power iterations to solve the matching problem , also taking into account the intraimage matching graph . As an alternative to optimization based techniques , Tron et al . ( 2017 ) used density based clustering techniques to compute multi-image correspondence . Fathian et al . ( 2019 ) use a similar technique but formulates it as a generalized Rayleigh quotient problem to achieve better results . Moving away from feature matching , Zach et al . ( 2010 ) uses cycle-consistency-like constraints on pose graphs quite effectively . To the best of our knowledge , we are the first to use graph neural networks for multi-image matching . 2.3 DEEP LEARNING FOR MATCHING . Previous attempts to improve image matching techniques using machine learning have focused on learning the descriptors given ground truth correspondence from curated datasets such as Zagoruyko & Komodakis ( 2015 ) ; Yi et al . ( 2016 ) ; and Brachmann et al . ( 2017 ) . Rocco et al . ( 2018 ) use a dense matching tensor with a weakly supervised loss to learn good features for matching . Zhou et al . ( 2019 ) has shown that these learned representations have limited ability for improving SfM tasks . There are other methods to build correspondences such as Choy et al . ( 2016 ) , but they only handle two-view constraints and require dense correspondences . Most similar to our work , Zhang & Lee ( 2019 ) use graph neural networks to do intra-image feature processing before doing two image inter-image feature similarity . However , their work only trains for pairs of image , while ours is explicitly trained for multi-image matching . Also similarly , Yi et al . ( 2018 ) attempts to improve correspondences by learning match probabilities for two-view RANSAC for greater robustness and speed . Like us Zhu et al . ( 2017 ) use cycle consistency in their loss ; however their method is for image generation and is restricted to pairwise cycle consistency . Our method can be applied to 3 or more images . Hartmann et al . ( 2017 ) learns multi-image matching but requires heavy supervision from 3D object reconstructions , which can be difficult or expensive to obtain . Zhou et al . ( 2015a ) , unlike our method , uses dense correspondence , but uses cycle consistency to find semantic matches accross multiple views . Suwajanakorn et al . ( 2018 ) find 3 dimensional latent keypoints , trained using ground truth rotation and translation . However , their method is restricted to a limited number of object categories , which is different from the SfM setting we are considering here . 2.4 GRAPH NEURAL NETWORKS . Graph neural networks , true to their name , are deep neural networks operating in graph domains . Multi-image matching is novel application of them , as they are more typically used for applications such as citation networks or recommendation systems . They have received more attention recently ( Bronstein et al. , 2017 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Scarselli et al. , 2009 ; Gama et al. , 2018b ; a ; Battaglia et al. , 2018 ) . The first methods to learn neural networks over graphs were the so-called Spectral methods . They used the eigenvectors of the graph Laplacian to compute convolutions , which can learn graph specific convolution kernels ( as in Bruna et al . ( 2013 ) ) , but require an a-priori known graph structure . Newer non-spectral methods do not require a-priori knowledge , as seen in Bronstein et al . ( 2017 ) ; Kipf & Welling ( 2017 ) ; Scarselli et al . ( 2009 ) ; and Gama et al . ( 2018a ) . Most of these methods use polynomials of the graph Laplacian to compute neighborhood averages . Gama et al . ( 2018b ; a ) formalize this notion and generalize it beyond the use of the graph Laplacian . To improve performance , more sophisticated aggregation techniques and global information passing can be used as discussed in Battaglia et al . ( 2018 ) . The closest work on Graph Neural Networks to ours is Kipf & Welling ( 2016 ) , which uses GNNs to reconstruct adjacency matrices , though not in a geometric context . 3 METHOD . Our goal is to learn optimal features that capture multiple image views by filtering out noisy feature matches . The input to our algorithm is a set of features and noisy correspondences , and the output is a new set of features where the pairwise similarities of these features correspond to the true matches . An outline of our approach can be seen in figure 1 . We do this by training the new set of feature embeddings to be cycle consistent . We formulate this problem in terms of the correspondence graph of the features . Graphs G = ( V , E ) have a set of vertices V and of directed edges E ⊆ V × V . For a vertex v ∈ V we use N h ( v ) to denote the h-hop neighbors of v , with the superscript left out for 1-hop neighbors . Similarly E ( v ) is used to denote the edges associated with v. To denote the vertices connected to an edge e ∈ E we write e ( v1 , v2 ) . 3.1 CORRESPONDENCE GRAPH . We assume there is an initial set of feature matches represented as a graph G = ( V , E ) , with an associated adjacency matrix A . The graph is constructed from putative correspondences of image features across images , typically constructed using feature descriptor distance ( e.g . SIFT feature distance ) . While there are many interesting methods for computing these putative correspondences ( Suh et al. , 2015 ; Yi et al. , 2018 ) , we do not explore them in this work . Typically putative correspondences are matched probabilistically , meaning a feature in one image matches to many features in another . The ambiguity in the matches could come from repeated structures in the scene , insufficiently informative low-level feature descriptors , or just an error in the matching algorithm . Filtering out these noisy matches is our primary learning goal . Each vertex of the graph v ∈ V is an image feature , corresponding to some ground truth 3D point p ( v ) . Each edge e = ( v1 , v2 ) ∈ E is a potential correspondence . Associated with each vertex v is an embedding fv ∈ R m , which can include the visual feature descriptor , position , scale , orientation , etc . Similarly , each edge e has an associated feature fe ∈ R p ( in this work , initially just the weight of the feature association ) . We use these features as the initialization for our learning algorithm . In the absence of noise or outliers , this graph would have a connected component for each visible point in the world , all mutually disjoint . Without noise , vertices v would only match with other vertices v′ that correspond to the same 3D point in the scene . Since features in this case represent unique locations in the scene , no points in the same image would have edges e between them . Mathematically , this can be expressed as e = ( v1 , v2 ) ∈ E =⇒ P ( v1 ) = P ( v2 ) . In the noisy case we expect this structure to be corrupted , i.e . there are some edges e = ( v1 , v2 ) ∈ E such that P ( v1 ) 6= P ( v2 ) . Thus we need to prune the erroneous edges . However , standard CNNs can not operate on this general graph structure . Thus we can not use standard convolutional nets to learn features for this task . Instead we use graph networks to learn feature representations on this space , which we describe in the next section .

Summary: This paper proposes a multi-image matching method using a GNN with cyclic and geometric losses. The authors use vertex embeddings to exploit cycle consistency constraints in an efficient manner, then the GNN is used to learn the appropriate embeddings in an unsupervised fashion. The geometric consistency loss is used to aid training in addition. Experimental evaluation shows its performance compared to MatchALS (Zhou et al., 2015) and PGDDS (Leonardos et al., 2016).